FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in designing and building evaluation harnesses for agentic AI systems, with strong proficiency in Python, SQL, and automated testing frameworks. Capable of integrating evaluations into CI/CD pipelines while ensuring data security and compliance in regulated environments.
Highest-signal resume keywords
Python ProgrammingAutomated Testing FrameworksCI/CD IntegrationData Security PracticesAI Evaluation Frameworks
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
PythonSQLPyTestETLAutomated Test SuitesLLM TestingDataikuGitDockerREST APIs
Soft Skills
Strong Communication SkillsTeam CollaborationIndependent WorkProject Management
Tools & Technologies
Dataiku EvaluationsGitHub ActionsJenkinsLangSmithLangfuseMLflowOpenTelemetrySnowflakeCortex AnalystCortex Search Services
Certifications & Qualifications
ISTQBAI Testing Certification
Industry Keywords
Agentic AI SystemsData SecurityPCI ComplianceRegulated EnvironmentsB2B Domain
Tech Stack
Tools & technologiesAWSAzureCloudDockerETLGoogle Cloud PlatformJenkinsJMeterPythonSnowflakeSQL
About the role
Key responsibilities & impact- Design and build Python evaluation harnesses for agentic systems, including golden datasets, LLM-as-judge graders, multi-turn regression suites, and trace-based assertions
- Develop frameworks to verify generated AI output
- Author automated test suites for prompts, tools, structured outputs, retrieval pipelines, and end-to-end agent workflows
- Validate guardrails around tool execution, authentication scoping, input/output validation, PII protection, prompt-injection protection, and hallucination mitigation
- Integrate evaluations into CI using Dataiku Evaluations, GitHub Actions, or Jenkins against quality, safety, and cost SLOs
- Instrument traces with LangSmith, Langfuse, MLflow, or OpenTelemetry and triage production drift into the evaluation harness
- Define release criteria, run pre-production and shadow tests, and partner with engineering to root-cause and fix regressions
- Partner with data engineers on Snowflake-backed retrieval testing using Cortex Analyst and Cortex Search Services
- Collaborate with platform teams on observability, security, and cost
- Shape internal QA standards for AI and data engineering
- Contribute to design reviews and share knowledge across India and U.S. teams
- Collaborate with developers, AI engineers, data engineers, DBAs, and product partners across environments
- Within the first 90 days, stand up a production-grade evaluation harness wired into CI, automate trace-based assertions against staging traffic, establish a quality scorecard, and recommend the next testing investment
- Protect Kobie’s internal Dataiku agent platform, which automates analyst workflows and surfaces loyalty-program insights from Snowflake
Requirements
What you’ll need- 6+ years of experience designing test strategies and evaluation harnesses for production-grade, agentic AI systems
- Experience with ETL
- 3+ years of professional QA / SDET experience, with production experience automating tests for backend services or data pipelines
- 1+ years of hands-on experience testing LLM or AI features in production, including prompt regression, tool/function-call validation, structured outputs, and RAG correctness
- Working knowledge of evaluation frameworks such as RAGAS, DeepEval, LangSmith, Langfuse, or comparable LLM-as-judge tooling
- Strong Python and PyTest skills
- Solid SQL skills
- Comfort with at least one cloud platform: AWS, Azure, or GCP
- Fluency with Git, Docker, REST APIs, and at least one CI tool: GitHub Actions, Jenkins, GitLab CI, or CircleCI
- Solid understanding of data security and responsible AI practices, particularly in PCI-compliant or regulated environments
- Ability to work independently and within a team, managing priorities across concurrent projects and time zones
- Strong written and verbal communication skills, with technical and non-technical stakeholders
- A bachelor’s degree is not required; equivalent practical experience counts
- Bonus: Dataiku DSS or Dataiku Evaluations experience
- Bonus: Dataiku LLM Mesh, Knowledge Banks, Prompt Studio, or Visual / Code Agents experience
- Bonus: Snowflake, Snowpark, or Snowflake Cortex experience
- Bonus: Red-teaming, prompt-injection testing, or adversarial test generation for LLMs
- Bonus: Familiarity with multi-agent patterns
- Bonus: Locust, JMeter, or k6 experience
- Bonus: ISTQB, AI Testing, or comparable QA certification
- Bonus: Experience in loyalty, martech, adtech, or a comparable data-rich B2B domain
Benefits
Comp & perks- Highly competitive benefits
- Comprehensive health coverage for teammates and dependents
- Well-being perks
- Public holidays
- Flexible time off
- Work-life balance
- Collaborative, flexibility, and career growth-focused culture
- Global project exposure and collaboration with U.S. teams
- Professional exposure to top industry leaders
- Charitable partnerships and volunteer opportunities
- Physical office in Bengaluru planned for collaboration and connection