Software Development Engineer in Test – AI

Newfold Digital

. Design, validate, and optimize testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents .

Posted 11/2/2026full-timeRemote • IndiaMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in AI Testing, including Prompt Engineering, Context Engineering, and AI Evaluation Frameworks. Proficient in developing automated validation pipelines and CI/CD integration using Python and relevant tools.

Highest-signal resume keywords

Python ProgrammingAI/LLM TestingCI/CD PipelinesDocker & KubernetesRobot Framework

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Software TestingQuality EngineeringAPI TestingData ValidationPerformance TestingSecurity TestingDebuggingAnalytical SkillsTroubleshootingMLOps Workflows

Soft Skills

CollaborationAgile/Scrum ExperienceProblem-Solving

Tools & Technologies

Robot FrameworkPlaywrightSeleniumCypressGitHub ActionsJenkinsGitLab CI/CDAzure DevOpsAWSGCP

Industry Keywords

AI GovernanceCompliance RequirementsDistributed Systems TestingSynthetic MonitoringObservabilityMulti-modal AI TestingAI Red TeamingAgent Reliability TestingTest Data EngineeringHuman-in-the-loop Evaluation

Tech Stack

Tools & technologies

AWSAzureCloudCypressDistributed SystemsDockerGoogle Cloud PlatformGraphQLJenkinsKubernetesMicroservicesPythonSeleniumSQL

About the role

Key responsibilities & impact

Design, validate, and optimize testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents
Develop AI evaluation frameworks for hallucination detection, groundedness, toxicity and bias validation, response quality, context relevance, prompt regression, determinism, consistency, safety, and guardrail validation
Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics
Create synthetic and benchmark datasets for AI testing scenarios
Test multi-agent orchestration workflows and agent memory/context retention
Validate MCP integrations and tool interoperability
Work on Context Engineering, Prompt Engineering, and Prompt Optimization strategies
Evaluate latency, token usage, model cost optimization, throughput, and scalability
Develop and maintain scalable UI/API automation frameworks using Python and tools such as Robot Framework, Playwright, Selenium, and Cypress
Automate testing for REST APIs, GraphQL APIs, event-driven systems, streaming systems, webhooks, microservices, and MFE architectures
Build reusable automation libraries, utilities, and AI-assisted testing workflows
Integrate automation into CI/CD pipelines and quality gates
Validate RAG architectures, including chunking, embedding quality, retrieval accuracy, re-ranking, and vector search relevance
Build or validate AI agents using LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, and OpenAI SDKs
Validate tool calling, function calling, agent memory, agent planning/execution, multimodal AI systems, and structured outputs
Maintain quality workflows with GitHub Actions, Jenkins, GitLab CI/CD, and Azure DevOps
Work with Docker and Kubernetes environments
Perform cloud-based testing and deployments across AWS, Azure, OCI, and GCP
Validate distributed systems reliability and resiliency
Monitor AI system behaviors using observability, logging, and tracing tools
Contribute to shift-left and quality engineering best practices
Collaborate with Engineering, AI/ML, Platform, Product, and DevOps teams

Requirements

What you’ll need

Bachelor’s/Master’s degree in Computer Science, Engineering, AI/ML, or related field preferred
Equivalent hands-on experience and demonstrated projects are highly valued
Strong programming experience in Python
Strong understanding of Software Testing & Quality Engineering principles
Hands-on experience with AI/LLM testing, Prompt Engineering, Context Engineering, AI Evals, RAG systems, and AI Agents
Experience with CI/CD pipelines, Docker & Kubernetes, cloud platforms (AWS/Azure/OCI/GCP), API testing & automation, microservices testing, and UI automation
Experience with SQL and data validation
Strong debugging, analytical, and troubleshooting skills
Experience working in Agile/Scrum environments
Robot Framework experience preferred
Playwright experience preferred
Knowledge of LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, and MCP
Hands-on experience with Cursor, GitHub Copilot, Claude, Gemini, and OpenAI APIs
Experience with vector databases and embeddings
Exposure to MLOps workflows and ML lifecycle testing
Knowledge of performance testing and security testing
Experience validating AI governance/compliance requirements
Experience with contract testing, chaos testing, or resilience engineering
Familiarity with accessibility testing and cross-browser/device testing
Exposure to synthetic monitoring and production observability
AI Red Teaming, LLM Security Testing, Agent Reliability & Safety Testing, AI Cost Optimization, Distributed Systems Testing, Test Data Engineering, Feature Flag Testing, AI Observability & Telemetry, Reinforcement Learning feedback workflows, Human-in-the-loop evaluation systems, Knowledge Graph integrations, and Multi-modal AI testing are high-impact bonus skills
Ability to experiment with AI tools and build side projects, prototypes, PoCs, or open-source contributions
Ability to rapidly learn emerging AI frameworks, tools, and architectures

Benefits

Comp & perks

Work-life balance
Diversity and inclusion programs
Learning and development opportunities
20+ affinity groups for networking and connection
Competitive HMO benefits – 175k MBL with one free dependent upon one year of service
Punctuality Bonus
Generous Vacation policy
Mentoring and career growth opportunities