Applied Data Scientist, Evaluation & Model Behavior

Applied Data Scientist, Evaluation & Model Behavior THINK DIFFERENT. BUILD THE FUTURE. 🚀 OUR MISSION Build everyday AGI. Trustworthy, consumer-grade agents that redefine human–AI collaboration for millions. Software shouldn’t wait for commands; it should partner with you, amplifying what you can do every single day. WHY AGI, INC. We’re a stealth team of elite founders and AI researchers, with backgrounds spanning Stanford, OpenAI, and DeepMind. We’re industry leaders in mobile and computer-use agents, bringing these capabilities to consumer scale. Grounded in years of agent research, our AI is designed with trustworthiness and reliability as core pillars, not afterthoughts. We are supported by tier-1 investors who funded the first generation of AI giants; now they’re backing us to build the next: everyday AGI. (Watch the demo https://drive.google.com/file/d/1ZydjdMeMh3x-QItUPQFbJUbhBW-4-XHa/view?usp=sharing) If you see possibility where others see limits, read on. ABOUT THE ROLE As an Applied Scientist focused on Evaluation & Model Behavior, you will design and implement the systems used to measure and improve the performance of Computer Use Agents. This is not a support role. You will be responsible for the technical definition of model quality, including the design of evaluation metrics, the curation of training datasets, and the engineering of system prompts. You'll work directly with the engineering team to translate product requirements into technical specifications and quantifiable benchmarks. You'll focus on rigor, clarity, and impact, ensuring every metric, dataset, and prompt moves us toward more reliable, trustworthy agents. WHAT YOU'LL DO Model Behavior Design: Translate product requirements into technical specifications for model behavior. Engineer system prompts and few-shot examples to address specific capability gaps and behavioral failures. Evaluation Design: Define metrics for reasoning, tool usage, and safety, and validate these metrics against human judgment to ensure statistical rigor. Data Strategy: Design algorithms to filter, score, and select training data. Write Python scripts to sanitize inputs and manage the training data lifecycle from raw logs to high-quality datasets. Failure Analysis: Investigate regressions in model benchmarks. Diagnose root causes, distinguishing between data quality issues, prompt instruction failures, or underlying model capability gaps and implement fixes. Ground Truth Management: Define rubrics and guidelines for human annotation. Maintain reference datasets ("Golden Sets") to establish a consistent baseline for model performance evaluation. MINIMUM QUALIFICATIONS - Master's degree or PhD in Computer Science, Data Science, Statistics, or a related technical field, or equivalent practical experience - 3+ years of experience in Data Science, Machine Learning, or Applied Science - Proficiency in Python, with experience writing production-quality code for data pipelines or evaluation harnesses - Experience with experimental design, A/B testing, or statistical analysis PREFERRED QUALIFICATIONS - Experience with Large Language Models (LLMs), including prompt engineering, fine-tuning, or RLHF workflows - Experience building automated evaluation systems or implementing model-based evaluation frameworks - Ability to translate product requirements into measurable technical metrics - Experience managing human-in-the-loop data pipelines or annotation quality control WHY THIS ROLE MATTERS You can't improve what you can't measure. You can't ship what you can't trust. You will define the technical definition of quality for our agents — the metrics that predict real-world success, the datasets that encode user intent, and the prompts that shape model behavior. Your work will directly determine how quickly we can iterate and how confidently we can ship. OUR CULTURE 🏢 All in, in person — work moves faster face-to-face 🚀 Ship by default — speed and polish c ... (truncated, view full listing at source)