Software Development Test Engineer, AI & Automation

Keeper Security, Inc.

. Serve as an embedded quality engineering partner within Keeper’s AI team .

Posted 10/8/2026full-timeRemote • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in automated testing frameworks, AI/ML validation strategies, and CI/CD integration, with a strong focus on quality assurance in AI-driven applications. Proficient in programming languages such as Python and Java, alongside a solid understanding of shift-left testing and continuous quality principles.

Highest-signal resume keywords

Automated Testing Framework DevelopmentAI/ML Validation StrategiesCI/CD IntegrationTest Management with TestRailProgramming in Python, Java, or JavaScript/TypeScript

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Automated TestingTest Strategy DefinitionAI/ML TestingRegression TestingIntegration TestingAPI TestingTest AutomationDebugging and TroubleshootingEvaluation Criteria DefinitionTest Coverage Analysis

Soft Skills

Analytical SkillsProblem-Solving SkillsWritten CommunicationVerbal CommunicationCollaboration

Tools & Technologies

PostmanREST AssuredGitHub ActionsJenkinsGitLab CITestRailDockerKubernetesAI-Assisted ToolsCI/CD Pipelines

Industry Keywords

AI Testing MethodologiesShift-Left TestingContinuous QualityModel EvaluationPerformance TestingSecurity TestingCybersecurity SoftwareAgentic SystemsRetrieval-Augmented GenerationSynthetic Test Data

Tech Stack

Tools & technologies

CloudCyber SecurityDockerJavaJavaScriptJenkinsKubernetesPythonRustTypeScript

About the role

Key responsibilities & impact

Serve as an embedded quality engineering partner within Keeper’s AI team
Work alongside engineers and Product throughout design and development
Define test strategy, acceptance criteria and automation requirements
Design and maintain automated testing frameworks for AI-powered applications, backend services, APIs and supporting software systems
Develop deterministic and probabilistic validation strategies for AI/ML outputs
Define evaluation criteria, tolerances and quality thresholds for AI model and LLM-driven functionality
Build automated regression, integration, API and end-to-end coverage alongside feature development
Develop test harnesses and evaluation workflows for prompts, models, agents, tool-calling workflows and AI-integrated applications
Validate AI behavior across models, prompts, inputs and system configurations
Integrate automated validation and quality gates into CI/CD pipelines
Build and maintain test datasets, mocks, fixtures and synthetic data
Analyze model and application failures and produce clear, reproducible defect reports
Evaluate AI outputs for reliability, consistency, safety and alignment with expected product behavior
Improve application testability, observability and diagnostic capabilities
Track test cases, execution results and coverage using TestRail or similar platforms
Reduce flaky tests and improve speed, stability and maintainability of automated test suites
Contribute to AI testing methodologies and automation best practices
Use AI-assisted tools to improve automation development, test generation, debugging, analysis and documentation

Requirements

What you’ll need

5+ years of experience in SDET, software development, test automation or another code-intensive quality engineering role
Hands-on experience testing AI/ML, LLM-powered or model-integrated software systems
Strong understanding of deterministic software testing and non-deterministic AI/ML validation
Experience defining evaluation criteria, expected behaviors, tolerances or scoring approaches for AI-driven functionality
Strong programming skills in Python, Java, JavaScript/TypeScript, Rust or another general-purpose programming language
Significant experience designing, developing and maintaining automated test frameworks
Experience building automated functional, integration, API and end-to-end test coverage
Experience testing REST APIs and distributed application workflows using Postman, REST Assured or similar platforms
Experience integrating automated tests into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI or similar technologies
Experience working within an Agile software development team with close collaboration between QA, Engineering and Product
Strong understanding of shift-left testing, continuous quality and testability principles
Experience with TestRail or similar test management platforms
Strong debugging and troubleshooting skills using logs, APIs, application telemetry and development tools
Experience with Git-based development workflows, code review and modern software engineering practices
Familiarity with AI-assisted development and testing tools such as Claude, ChatGPT, GitHub Copilot or similar technologies
Strong analytical and problem-solving skills with the ability to evaluate variable AI behavior without relying solely on exact-match outcomes
Strong written and verbal communication skills and ability to collaborate closely with AI engineers, Software Engineers, Product Managers and QA professionals
Bachelor’s degree in Computer Science, Engineering or a related technical field, or equivalent practical experience
Qualified candidates must be a U.S. Person, defined as a U.S. citizen or lawful permanent resident
Preferred: experience testing LLMs, agentic systems, retrieval-augmented generation, tool calling or prompt-driven workflows
Preferred: experience building automated model evaluation, benchmarking or regression frameworks
Preferred: familiarity with AI evaluation metrics and techniques for measuring quality across non-deterministic outputs
Preferred: experience with Rust in production or test automation environments
Preferred: experience with performance, reliability, security or adversarial testing of AI-enabled systems
Preferred: experience with AI-assisted generation of automated tests or synthetic test data
Preferred: familiarity with Docker, Kubernetes or cloud-based testing environments
Preferred: experience testing cybersecurity, identity, PAM or other security-sensitive software
Preferred: experience building shared automation frameworks or internal testing tools used across engineering teams

Benefits

Comp & perks

Medical, Dental & Vision (inclusive of domestic partnerships)
Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life
Voluntary Short/Long Term Disability Insurance
401K (Roth/Traditional)
A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)