QA Automation Engineer – AI

Anaplan

. Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python .

Posted 10/13/2026full-timeRemote • North Carolina • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in developing and executing automated test coverage for GenAI systems, utilizing Python and various testing methodologies. Proficient in curating test datasets and implementing testing strategies for AI applications, ensuring high-quality evaluations and performance metrics.

Highest-signal resume keywords

Python Coding ExperienceTest Automation FrameworksAPI Testing ToolsGenAI TechnologiesCI/CD Pipeline Familiarity

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

QA EngineeringTest AutomationFunctional TestingIntegration TestingRegression TestingPerformance TestingSecurity TestingPrompt EngineeringScriptingAdversarial Testing

Tools & Technologies

PostmanREST AssuredPytestAutomated Testing FrameworksInternal Tools

Industry Keywords

GenAILLMsConversational AIAgentic SystemsModel HallucinationsBiasesUser Acceptance TestingAutomated CoverageEvaluation FrameworksNon-Deterministic Systems

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python
Implement testing strategies for conversational AI, agentic systems, and LLM-powered workflows
Run and maintain automated prompt-testing suites, including regression tests for unintended model-behaviour changes
Support evaluation frameworks measuring GenAI accuracy, relevance, safety, consistency, and latency
Curate high-quality test datasets and golden examples representing diverse user scenarios and edge cases
Assist with adversarial testing for model hallucinations and biases, and support production alerting integration
Use internal tools and testing frameworks to help developers validate GenAI code and assist with user acceptance testing
Own automated coverage for the workstream and document testing procedures and known issues

Requirements

What you’ll need

Solid QA or test engineering experience, with exposure to or a keen interest in AI/ML systems
Hands-on Python coding experience
Good understanding of GenAI technologies, including LLMs, prompt engineering, and AI application patterns
Solid experience with test automation frameworks and scripting, including Python, Pytest, API, and UI testing
Practical knowledge of functional, integration, regression, performance, and security testing methodologies
Ability to execute test cases and evaluate criteria for non-deterministic systems
Experience with API testing tools such as Postman and REST Assured, plus backend testing
Familiarity with CI/CD pipelines and automated testing integration

Benefits

Comp & perks

Inclusive culture focused on diversity, equity, inclusion and belonging
Reasonable accommodation for individuals with disabilities
Equitable benefits and all privileges of employment