Senior AI Developer

ClickUp

. Design, build, and own ClickUp's AI evaluation system, including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features .

Posted 11/6/2026full-timeRemote • United StatesSenior💰 $200,000 - $250,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building and evaluating AI systems, particularly LLMs, with a focus on production quality, statistical analysis, and backend engineering. Proficient in integrating AI features into platforms while ensuring compliance with privacy and security standards.

Highest-signal resume keywords

LLM Evaluation SystemsBackend EngineeringStatistical RigorCloud-Native TechnologiesAI Privacy and Security

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

PythonGoTypeScriptA/B TestingModel BenchmarkingDistributed SystemsAPIs DevelopmentData AnonymizationExperiment DesignAI Model Integration

Soft Skills

CollaborationCommunication

Tools & Technologies

KubernetesDockerAWSGCPAzureLangGraphAirflowRay

Industry Keywords

AI EvaluationQuality TrackingModel SelectionLatency ManagementRegulatory Adherence

Tech Stack

Tools & technologies

AirflowAWSAzureCloudDistributed SystemsDockerGoogle Cloud PlatformKubernetesPythonRayTypeScriptGo

About the role

Key responsibilities & impact

Design, build, and own ClickUp's AI evaluation system, including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features
Build and evolve the core agent harness and the AI that performs well on evaluations, partnering with product and engineering to define quality standards
Build self-improvement loops where agents modify other agents or prompts, rerun evaluations, and iterate automatically
Benchmark newly released models and own model selection, routing, and fallback strategies across providers while balancing quality, latency, and cost
Architect, design, and implement scalable AI platform services for deployment, orchestration, and lifecycle management of LLMs and other AI models
Build and maintain robust APIs and backend systems integrating AI-powered features into ClickUp's core platform
Develop infrastructure for model serving, monitoring, logging, and automated evaluation to ensure reliable, high-performing production AI services
Drive best practices in AI privacy, security, and compliance, including data anonymization, secure data handling, and regulatory adherence
Collaborate with product, frontend, and data teams to deliver reliable, user-centric AI experiences

Requirements

What you’ll need

5+ years of software engineering experience, including 2+ years building production LLM or ML systems
Hands-on experience building LLM evaluation systems in production, including golden datasets, LLM-as-judge, judge calibration, offline and online evals, tied to real product quality
Strong experiment design and statistical rigor, including A/B testing, offline vs. online metrics, and separating real regressions from noisy or variable results
Proven track record of applying LLMs and AI models to real-world, customer-facing product features
Proven experience integrating and managing multiple LLMs, including benchmarking, routing, and cost/latency/quality tradeoffs
Deep expertise in backend engineering, distributed systems, and cloud-native technologies such as Kubernetes, Docker, and AWS/GCP/Azure
Strong programming skills in Python, Go, TypeScript, or similar backend and AI platform languages
Proficiency in orchestration frameworks and workflow engines such as LangGraph, Airflow, Ray, or similar
Familiarity with search and retrieval technologies and their integration into AI-driven applications
Excellent collaboration and communication skills, with a track record of working effectively in cross-functional teams
Passion for staying at the forefront of AI and applying new technologies to solve real-world problems at scale
Visa sponsorship is not guaranteed and depends on business needs for the specific role

Benefits

Comp & perks

Equity
401k
Health, Dental, and Vision insurance
Spending accounts
Life & Disability
Paid parental leave
Flexible paid time off
Enhanced employee assistance program
Employee wellness stipend
Professional development stipend