Senior Software Development Engineer in Test – AI

Anaplan

. Design and architect comprehensive test plans, evaluation frameworks, and quality metrics for LLM-powered applications and agentic workflows .

Posted 10/13/2026full-timeRemote • North Carolina • United StatesSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and implementing comprehensive test plans and automated testing frameworks for GenAI applications, with a strong focus on quality assurance and performance metrics. Proficient in Python and familiar with various testing methodologies and tools to ensure high-quality AI solutions.

Highest-signal resume keywords

GenAI TechnologiesPython Coding ExperienceTest Automation FrameworksAPI Testing ToolsCI/CD Pipelines

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Test AutomationSoftware Testing MethodologiesPrompt EngineeringAdversarial TestingTest Case DesignPerformance TestingRegression TestingFunctional TestingIntegration TestingAnalytical Problem-Solving

Soft Skills

Attention to Detail

Tools & Technologies

SeleniumPytestPostmanREST Assured

Industry Keywords

LLM-Powered ApplicationsQuality MetricsEvaluation FrameworksQuality Assurance Best PracticesReal-Time Monitoring

Tech Stack

Tools & technologies

JavaScriptPythonSelenium

About the role

Key responsibilities & impact

Design and architect comprehensive test plans, evaluation frameworks, and quality metrics for LLM-powered applications and agentic workflows
Develop robust end-to-end automated test suites in Python across evaluations, API checks, UI checks, and performance tests
Create automated prompt testing systems and regression suites to detect unintended changes in model behavior
Establish evaluation criteria for GenAI quality across accuracy, relevance, safety, consistency, and latency
Perform adversarial testing (red-teaming) to identify and mitigate hallucinations, biases, and security vulnerabilities
Develop internal tools and testing frameworks for validating GenAI implementations
Deploy real-time monitoring and alerting systems to detect quality degradation in production AI features
Own the quality and testing strategy, maintain high-fidelity test datasets, and define QA best practices

Requirements

What you’ll need

Deep understanding of GenAI technologies, including LLMs, prompt engineering, and AI application patterns
Strong hands-on Python coding experience
Expert experience with test automation frameworks and scripting, including Python, JavaScript, Selenium, and Pytest
Strong background in software testing methodologies: functional, integration, regression, performance, and security testing
Proven ability to design test cases and evaluation criteria for non-deterministic systems
Strong analytical and problem-solving skills with attention to detail
Experience with API testing tools, including Postman and REST Assured, and backend testing
Strong experience with CI/CD pipelines and automated testing integration

Benefits

Comp & perks

Inclusive culture focused on diversity, equity, inclusion and belonging
Reasonable accommodation for individuals with disabilities
Equitable benefits and all privileges of employment