Quality Assurance Engineer

Job Description Roles & Responsibilities Key Responsibilities 1. AI & LLM Validation Non-Deterministic Testing : Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy against Gold Standard datasets RAG Evaluation : Implement automated metrics (e.g., RAGA S, faithfulness, answer relevance) to verify that Retrieval-Augmented Generation pipelines accurately cite technical and regulatory documentation Prompt Regression : Design regression suites to monitor prompt drift, ensuring model updates do not degrade the quality of AI-generated engineering documents 2. Integration & System Verification Enterprise Integration : Build robust tests to validate data consistency between AI agents and critical systems (e.g., SAP S/4HAN A, Ariba), ensuring the integrity of Bill of Materials (BOM) and financial data Performance Benchmarking : Design tests to validate latency and throughput for forecasting models and risk-scoring engines using tools like Locust, JMeter, or K 6 API & Security Validation : Automate testing of secure API gateways, verifying Role-Based Access Control (RBAC ) and PII redaction logic before data reaches AI models . 3. Governance & Traceability V-Model Alignment : Map automated test cases to System Requirements to create digital evidence for formal Verification and Validation (V&V ) reports Stage Gate Compliance : Prepare Test Readiness packages for formal reviews, providing quantitative evidence that systems are stable enough to move from MVP to Production Defect Lifecycle Management : Manage the feedback loop between Requirements Quality Assistants and development teams, tracing AI logic defects back to specific model versions Desired Candidate Profile Core Automation : Expert proficiency in Python (Pytest ) and standard libraries (Selenium/Playwrigh t, Requests) AI Evaluation : Hands-on experience with LLM evaluation frameworks (e.g., DeepEval, TruLen s) and Ground Truth dataset management Performance Engineering : Proficiency in crafting Performance Test Plans and implementations (Locust, K6, etc.) Data Validation : Expertise in SQ L and data quality tools (e.g., Great Expectation s) for Data Lakehouses and Vector Databases CI/CD & DevOps : Strong experience integrating quality gates into GitLab CI/C D pipelines Engineering Practices : Deep understanding of modern QE practices, including Shift Lef t, Test Pyramid, and Mono-repo architectures Professional Qualification Experience : 5+ years in QA Automation, with 2+ years focused on complex data-driven applications, ML models, or AI agents Domain Expertise : Background in Defense, Aerospac e, or highly regulated industries is a strong plus. Familiarity with IV& V processes is highly desirable Analytical Mindset : Ability to define pass/fail criteria for probabilistic systems and communicate Confidence Levels to engineering leadership This is a Fixed Term Contract role. Company Industry IT - Software Services Department / Functional Area QualityTestingQAQCInspector Keywords Quality Assurance Engineer Get real-time job updates only on our App