FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in test strategy ownership, automation frameworks, and evaluation techniques for AI agents. Proficient in Python and Java, with strong analytical and communication skills to identify testing gaps and improve release quality.
Highest-signal resume keywords
Test Strategy OwnershipUI Automation Using SeleniumAPI Automation FrameworksPython ProgrammingExperience Testing GenAI Agents
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Software TestingSDET ExperienceEnd-to-End Regression SuitesAutomation ImprovementEvaluation TechniquesDebugging Agent FailuresTest Code ReviewAnalytical Skills
Soft Skills
Strong Communication SkillsMentoring SDETs
Tools & Technologies
SeleniumPlaywrightLangChainLangGraphLangSmithCI/CD Integration
Industry Keywords
AI AgentsChatbotsIVRsEvaluation DatasetsRelease Quality Criteria
Tech Stack
Tools & technologiesJavaPythonSelenium
About the role
Key responsibilities & impact- Own test strategy and release quality criteria for the product area
- Build and maintain UI automation, API automation and end-to-end regression suites
- Design evaluation strategy for AI agents using LLM-as-judge, rubrics and other evaluation techniques
- Build eval datasets and automated evaluation suites in Python integrated with CI/CD
- Test AI agents for accuracy, consistency, tool usage, safety and data handling
- Debug agent failures using LangSmith and similar AI observability tools
- Analyse release-over-release quality and identify regressions
- Review product requirements and identify gaps in requirements and test coverage
- Review test code, set testing standards and mentor SDETs
- Communicate findings and recommendations to engineering, product and leadership
- Work closely with GenAI engineering, CX engineering and product teams
Requirements
What you’ll need- 6–9 years in software testing, including hands-on SDET experience
- Good working knowledge of Python and Java
- Experience owning test strategy and improving automation and release quality
- Experience with UI automation using Selenium or Playwright
- Experience building API automation and end-to-end frameworks with CI/CD
- Ability to analyse product requirements and identify testing gaps
- Prior experience testing chatbots, IVRs and GenAI agents
- Experience designing evals, rubrics and LLM-as-judge evaluations
- Working knowledge of LangChain, LangGraph and LangSmith or similar tools
- Experience building eval harnesses or tooling for AI agents
- Strong analytical and communication skills
Benefits
Comp & perks- Remote work arrangement
- Certified Great Place to Work engineering center