Quality Assurance / SDET Engineer, GenAI/AI Platform

About Us: We love going to work and think you should too. Our team is dedicated to trust, customer obsession, agility, and striving to be better everyday. These values serve as the foundation of our culture, guiding our actions and driving us towards excellence. We foster a culture of performance and recognition, allowing us to transform growth as we enable our employees to do the best work of their careers. This position is located in Pune. You'll be working in a major tech center of Pune, India. Across the globe, our Centers of Energy serve as hubs where we accelerate productivity and collaboration, inspire creativity, and cultivate a culture of connection and celebration. Our teams coordinate their time in Centers of Energy to reflect how they work best. To learn more about life at LogicMonitor, check out our Careers Page . What You'll Do: LogicMonitor® is the AI-first hybrid observability platform powering the next generation of digital infrastructure. LogicMonitor delivers complete visibility and actionable intelligence across on-premises, cloud, and edge environments. By anticipating issues before they strike, optimizing resources in real time, and enabling faster, smarter decisions, LogicMonitor helps IT and business leaders protect margins, accelerate innovation, and deliver exceptional digital experiences without compromise. Our customers love LogicMonitor's ability to bring cloud and traditional IT together into one view, as seen in minimal churn rates, expansion business, and exciting new customer references. In fact, LogicMonitor has received the highest Net Promoter Score of any IT Infrastructure Management provider. LogicMonitor also boasts high employee satisfaction. We have been certified as a Great Place To Work®, and named one of BuiltIn's Best Places to Work for the seventh year in a row! LogicMonitor is looking for a highly skilled QA/ SDET with 2 to 3 years of experience to build and scale automated test frameworks for Generative AI features across our observability platform.In this role, you will ensure the quality, reliability, safety, and performance of LLM-based workflows , including AI assistants, Retrieval-Augmented Generation (RAG) pipelines, AI-generated incident summaries, auto-remediation agents, tool calling, and AI-driven insights. You will work closely with engineering, product, and applied AI teams to validate AI experiences in production. Here's a closer look at this key role: 1. Test Strategy for GenAI Features Define end-to-end test strategies for GenAI-driven product features , including: AI assistant chat flows (multi-turn conversations) AI-generated RCA summaries and incident timelines RAG-based responses using knowledge base, tickets, and observability signals Agent execution flows (tool calling, action orchestration) Establish quality standards for AI output across: Factuality, relevance, completeness, groundedness Hallucination risk mitigation User trust and explainability 2. Automation Framework End-to-End Testing Build scalable automation test frameworks for API and UI experiences. Automate validation of: AI endpoints (REST) Orchestration workflows Streaming behaviors Structured response schemas (JSON, Pydantic models, etc.) Develop regression test packs that run in CI/CD pipelines and validate: Deterministic system behavior around non-deterministic model outputs 3. AI Evaluation (LLM Testing) Create and maintain LLM evaluation test suites , including: Golden datasets (prompt → expected response patterns) Rubric-based scoring (LLM judge + deterministic validation checks) Failure taxonomy (hallucinations, irrelevant retrieval, refusal bugs, etc.) Build automated pipelines for: Drift testing Prompt regression testing Retrieval quality regression testing 4. Reliability, Performance Scale Testing Design and implement performance and load tests for: High concurrency chat experiences Streaming response latency Tool execution latency RA ... (truncated, view full listing at source)