Principal AI Quality Developer

Job Requisition ID # 26WD95110 Position Overview We are hiring a principal quality engineering leader to design, build, and operationalize testing and evaluation for applied AI and ML systems across Autodesk products, with a focus on the Fusion Industry Cloud. This role is responsible for validating the quality, safety, reliability, and performance of AI features and agentic workflows shipped by the applied AI engineering team. You will create repeatable evaluation approaches for predictive and generative AI systems, build automated test harnesses integrated into CI/CD, and partner closely with Applied AI engineers, Product, Security, and Platform teams to ensure AI capabilities meet defined acceptance thresholds and customer expectations. Autodesk is a global leader in 3D design, engineering, and entertainment software, empowering innovators everywhere to imagine, design, and make a better world. Our software is used by professionals and hobbyists alike, across industries from architecture and manufacturing to media and entertainment. We are committed to diversity, flexibility, and continuous learning, offering a collaborative environment where you can grow your skills and make a real impact. You will lead the quality strategy for applied AI experiences that directly impact product experience and customer outcomes. This role offers the opportunity to define how we test and trust AI features at scale, partner with senior engineering leaders, and raise the bar for reliability and responsible AI in a high-impact product environment. Responsibilities Define and drive the quality strategy for applied AI and agentic features within Autodesk products and services with a focus on Fusion Industry Cloud Test planning, acceptance criteria, and release readiness for AI-powered capabilities Design and build automated evaluation frameworks for LLM and ML outputs, including curated gold sets, synthetic test sets, adversarial test cases, and regression suites that run in CI/CD Create repeatable evaluation metrics and scoring approaches for AI behavior, including relevance, faithfulness, hallucination rate, toxicity, bias, robustness, latency, and cost, and partner with Engineering and Product to define pass or fail thresholds Validate retrieval-augmented generation workflows, tool use, and agent behavior across edge cases, including prompt sensitivity, context recall, and retrieval accuracy Lead AI safety and abuse testing in partnership with Security, including prompt injection and jailbreak scenarios, data leakage risks, and misuse cases relevant to product integrations Establish production monitoring and post-deployment validation for AI features, including drift detection, failure taxonomy, and alerting tied to customer impact and SLAs Perform deep issue triage and root-cause analysis for AI-related defects across data, prompts, retrieval, model versions, and integration logic, and drive fixes with Applied AI engineers Mentor engineers and QA practitioners on AI testing approaches, improve team testing standards, and contribute to hiring and interview loops for quality roles Collaborate with cross-functional partners to ensure responsible AI practices are embedded into development workflows, including documentation of test methodology, evaluation results, and release sign-off criteria Minimum Qualifications 7 years of experience in quality engineering, test automation, or software engineering, with demonstrated ownership of test strategy and automation for complex systems, including services, platforms, or product integrations Strong programming skills in Python and experience with test frameworks such as pytest or unittest and building automated test harnesses integrated with CI/CD Hands-on experience testing AI or ML systems, including model output evaluation, error analysis, regression testing, and monitoring of quality over time Practical experience validating LLM-based systems, including prompt testing, retrieva ... (truncated, view full listing at source)