Senior AI Developer

ClickUp

. Design, build, and own ClickUp's AI evaluation system, including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features .

Posted 11/5/2026full-timeRemote • United StatesSenior💰 $200,000 - $250,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building and evaluating AI systems, particularly LLMs, with a focus on quality, performance, and compliance. Proficient in backend engineering and integrating AI features into production environments while collaborating effectively across teams.

Highest-signal resume keywords

LLM Evaluation SystemsBackend EngineeringStatistical RigorCloud-Native TechnologiesAI Privacy and Compliance

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

PythonGoTypeScriptA/B TestingModel BenchmarkingDistributed SystemsData AnonymizationModel ServingIntegration of AI ModelsExperiment Design

Soft Skills

CollaborationCommunication

Tools & Technologies

KubernetesDockerAWSGCPAzureLangGraphAirflowRayBraintrustLangSmith

Industry Keywords

AI EvaluationMachine LearningProduction SystemsQuality TrackingRegulatory Adherence

Tech Stack

Tools & technologies

AirflowAWSAzureCloudDistributed SystemsDockerGoogle Cloud PlatformKubernetesPythonRayTypeScriptGo

About the role

Key responsibilities & impact

Design, build, and own ClickUp's AI evaluation system, including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features
Build and evolve the core agent harness and the AI that performs well on evaluations, partnering with product and engineering to define quality standards
Build self-improvement loops where agents modify other agents or prompts, rerun evaluations, and iterate automatically
Benchmark new models and own model selection, routing, and fallback strategies across providers while balancing quality, latency, and cost
Architect, design, and implement scalable AI platform services for deployment, orchestration, and lifecycle management of LLMs and other AI models
Build and maintain robust APIs and backend systems integrating AI-powered features into ClickUp's core platform
Develop infrastructure for model serving, monitoring, logging, and automated evaluation for reliable, high-performance production AI services
Drive best practices in AI privacy, security, and compliance, including data anonymization, secure data handling, and regulatory adherence
Collaborate cross-functionally with product, frontend, and data teams to deliver reliable, user-centric AI experiences

Requirements

What you’ll need

5+ years of software engineering experience, including 2+ years building production LLM or ML systems
Hands-on experience building LLM evaluation systems in production, including golden datasets, LLM-as-judge, judge calibration, offline and online evals, tied to real product quality
Strong experiment design and statistical rigor, including A/B testing, offline vs. online metrics, and separating real regressions from noisy or variable results
Proven track record of applying LLMs and AI models to real-world, customer-facing product features
Proven experience integrating and managing multiple LLMs, including benchmarking, routing, and cost/latency/quality tradeoffs
Deep expertise in backend engineering, distributed systems, and cloud-native technologies such as Kubernetes, Docker, and AWS/GCP/Azure
Strong programming skills in Python, Go, TypeScript, or similar backend and AI platform languages
Proficiency in orchestration frameworks and workflow engines such as LangGraph, Airflow, Ray, or similar
Familiarity with search and retrieval technologies and their integration into AI-driven applications
Excellent collaboration and communication skills, with a track record of working effectively in cross-functional teams
Passion for staying at the forefront of AI and applying new technologies to solve real-world problems at scale
Nice to have: background in data science, applied science, or machine learning engineering
Nice to have: experience with eval tooling such as Braintrust, LangSmith, Weave, Arize, or in-house frameworks
Nice to have: experience with model routing, AI gateways, or inference optimization
Must be located in or legally authorized to work in the United States; visa sponsorship is not guaranteed and depends on business needs

Benefits

Comp & perks

Equity
401k
Health, Dental, and Vision insurance
Spending accounts
Life & Disability
Paid parental leave
Flexible paid time off
Enhanced employee assistance program
Employee wellness stipend
Professional development stipend