Lead AI QA Engineer

Ovii's Interpretation of the Role

Lead AI QA Engineer responsible for designing evaluation harnesses, automating AI/LLM test suites, and integrating quality checks into CI pipelines for Kobie's India Tech Hub.

Role Snapshot

• Lead AI testing initiatives
• Build evaluation harnesses for LLM agents
• Automate AI/LLM test suites
• Integrate tests into CI/CD
• Drive AI QA standards
• Collaborate across data, engineering, and product

Must-Have Requirements

• Python
• PyTest
• SQL
• Git
• Docker
• REST APIs
• CI tool (GitHub Actions, Jenkins, etc.)
• Data security & responsible AI knowledge
• QA / SDET
• AI / LLM testing

Nice-to-Have Signals

• Dataiku
• LangSmith / Langfuse
• Snowflake
• Red‑teaming / adversarial testing
• Performance testing tools (Locust, JMeter, k6)
• ISTQB or AI testing certification
• Familiarity with multi‑agent patterns
• Experience in loyalty / martech / adtech domains
• red‑teaming
• performance testing
• loyalty
• martech
• adtech

Work Setup

• Location: Bengaluru, India
• Work mode: HYBRID
• Remote scope: UNSPECIFIED
• Employment type: Full-Time

Eligibility Gates

• Visa sponsorship: unknown

Not Specified in JD

• Salary range
• Visa sponsorship
• Remote eligibility
• Certifications

What You'll Likely Work On

• Design and build evaluation harnesses for AI agents, including golden datasets and LLM‑as‑judge graders
• Develop frameworks to verify generated AI output, enforce guardrails, and mitigate hallucinations
• Wire evaluations into CI pipelines (GitHub Actions, Jenkins) and monitor quality, safety, and cost SLOs
• Instrument observability (LangSmith, Langfuse, MLflow, OpenTelemetry) and triage production drift
• Own end‑to‑end quality release criteria, run pre‑prod and shadow tests, and drive regression fixes
• Partner with data engineers and platform teams on Snowflake‑backed retrieval testing and observability
• Shape internal AI QA standards and share knowledge across India and U.S. teams

Good Fit If You Have

• Strong written and verbal communication with technical and non‑technical stakeholders
• Proven ability to work independently across time zones
• Experience in regulated or PCI‑compliant environments
• Passion for AI/LLM testing and responsible AI practices