Principal Evals Engineer

Discover the Role As Principal Evals Engineer, you will define how AI performance is measured across the organisation. You will design and build the shared evaluation architecture used across multiple AI products and engineering teams. This includes creating robust evaluation harnesses, managing golden datasets, automating grading, detecting behavioural regressions and ensuring that engineering teams can make release decisions based on evidence rather than subjective judgement. This is a senior individual contributor role. You will remain highly hands on, building frameworks, running experiments and setting the technical standard for AI evaluation across the organisation. Discover the Responsibilities Define the overall AI evaluation strategy and reference architecture across the engineering organisation. Build shared evaluation frameworks, harnesses and reusable libraries. Develop and maintain golden datasets and dataset versioning strategies. Design automated evaluation and grading systems. Implement regression detection for non deterministic AI behaviour. Evaluate LLM output quality, hallucination, grounding, faithfulness and citation accuracy. Build evaluation frameworks for RAG and retrieval systems. Evaluate agentic systems, including tool usage, multi step reasoning and failure recovery. Design and calibrate LLM as a judge systems against human evaluation. Establish production monitoring, online evaluation and drift detection. Develop adversarial testing methodologies covering prompt injection, jailbreaks and data leakage. Integrate AI evaluation into CI/CD pipelines and release processes. Help engineering teams understand and apply rigorous evaluation standards independently. Contribute to building the broader evaluation engineering function as the organisation scales. Discover the Required We are looking for a senior engineer with a strong track record of building AI or testing infrastructure at scale. You should have experience owning evaluation frameworks rather than simply working inside an existing testing platform. Discover the Requirements Staff or Principal level engineering experience. Strong hands on Python engineering skills. Deep experience with LLM or Generative AI evaluation. Experience evaluating RAG, retrieval or agentic systems. Strong understanding of hallucination, grounding and output quality measurement. Experience building regression detection for non deterministic AI systems. Practical experience with LLM as a judge techniques. Strong understanding of rubric design and calibration against human labels. Statistical literacy including sampling, confidence intervals, significance and inter rater agreement. Strong CI/CD and software quality engineering experience. Experience building reusable frameworks or shared engineering libraries. Hands on experience working with production AI systems. Discover the Desired Experience with one or more of the following would be highly valuable: Ragas. DeepEval. Promptfoo. Braintrust. LangSmith. Langfuse. LangChain or LangGraph. Vector databases such as pgvector or Qdrant. AI red teaming or adversarial testing. Voice or conversational AI evaluation. Human evaluation programme design. Data pipeline or performance testing. Arabic language AI evaluation, including golden datasets, dialect coverage and right to left validation. Experience working within regulated, government or highly secure environments. Technology Environment The primary language is Python, with additional use of TypeScript and Java. The wider environment includes custom evaluation harnesses, LLM as a judge patterns, golden datasets, Langfuse, Ragas, DeepEval, Promptfoo, LangChain, LangGraph, Microsoft Agent Framework, pgvector, Qdrant, Playwright, Cypress, Great Expectations, Grafana, Docker, Kubernetes and Azure. The team is pragmatic about tooling and is more interested in strong engineering judgement than experience with one specific framework.

Desired Candidate Profile

Staff or Principal level engineering experience. Strong hands on Python engineering skills. Deep experience with LLM or Generative AI evaluation. Experience evaluating RAG, retrieval or agentic systems. Strong understanding of hallucination, grounding and output quality measurement. Experience building regression detection for non deterministic AI systems. Practical experience with LLM as a judge techniques. Strong understanding of rubric design and calibration against human labels. Statistical literacy including sampling, confidence intervals, significance and inter rater agreement. Strong CI/CD and software quality engineering experience. Experience building reusable frameworks or shared engineering libraries. Hands on experience working with production AI systems.