Lead Data Engineer – AI/Machine Learning

Core Specialty Insurance Holdings, Inc.

. Design, build, and optimize data pipelines, ingestion frameworks, and platform components supporting analytics, reporting, and AI/ML use cases .

Posted 9/22/2026full-timeUnited StatesSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates extensive experience in data engineering, particularly in building and optimizing data pipelines and AI/ML frameworks. Proficient in autonomous project ownership, collaboration with cross-functional teams, and implementing MLOps strategies.

Highest-signal resume keywords

Data Pipeline DesignAI/ML Framework DevelopmentMLOps Strategy ImplementationSnowflake and Databricks ExperiencePython Programming Skills

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data Pipeline DesignData OptimizationDistributed Data ProcessingAI/ML DeploymentAPI Design and IntegrationModel Evaluation FrameworksVersion ControlTesting PracticesData Vault 2.0Agentic Workflows

Soft Skills

Autonomous Project OwnershipInnovative Problem SolvingCollaboration with StakeholdersTechnical GuidanceDocumentation Skills

Tools & Technologies

SnowflakeDatabricksAzure Synapse AnalyticsAWSAzureGCPLLM APIsVector DatabasesLangChainLangGraph

Industry Keywords

Data EngineeringAI/MLData GovernanceCompliance StandardsProperty & Casualty InsuranceMLOpsData WarehouseData LakehouseEmbedding ModelsHybrid Search

Tech Stack

Tools & technologies

AWSAzureGoogle Cloud PlatformPythonVault

About the role

Key responsibilities & impact

Design, build, and optimize data pipelines, ingestion frameworks, and platform components supporting analytics, reporting, and AI/ML use cases
Own complex engineering initiatives autonomously from technical design through implementation and rollout
Identify and resolve performance, scalability, and reliability issues across the data platform
Proactively identify gaps and propose improvements to data engineering solutions
Write clean, well-tested, well-documented code and infrastructure-as-code
Define AI/ML frameworks and evaluate tools, platforms, and standards for building and deploying AI/ML solutions
Build working prototypes that provide immediate value to engineering teams
Shape and help implement MLOps strategy, including model deployment, monitoring, versioning, and lifecycle management
Collaborate with Data Governance to align AI/ML frameworks with data governance, security, and compliance standards
Design and advocate for scalable data infrastructure patterns for AI/ML use cases
Partner with Data Science, Data Engineering, and business stakeholders to assess readiness gaps and build a roadmap
Advise the VP, Head of Data on emerging AI/ML technologies, practices, and industry trends
Document AI/ML standards, frameworks, and decisions
Provide senior technical guidance on architecture, design patterns, best practices, AI/ML readiness, and MLOps frameworks
Partner with Enterprise Architecture on architectural blueprints for AI readiness
Report directly to the VP, Head of Data
Perform other duties as assigned

Requirements

What you’ll need

Minimum 7+ years of experience in data engineering, with experience working on large-scale, mature data platforms
3+ years of experience developing ML or AI deliverables, including deployment to production
Bachelor's degree in related field or demonstrated equivalent experience in a related field required
Working knowledge of Agentic Workflows for engineering and architecture
Demonstrated experience taking autonomous technical ownership of complex projects from design through delivery, with minimal oversight
Experience contributing to or shaping AI/ML enablement efforts, such as defining frameworks, evaluating MLOps tooling, or building infrastructure that supports model training and deployment
Experience partnering with Data Governance, Data Science, or Compliance teams to align technical practices with governance and regulatory requirements
Track record of proposing and driving innovative technical solutions rather than simply executing predefined plans
Experience designing or implementing Agentic workflows for data engineering preferred
Experience working with Property & Casualty insurance carriers preferred
Experience with Data Vault 2.0 or Ensemble data modeling techniques preferred
Authorized to work for any employer in the U.S.; no current or future work authorization sponsorship
Strong data engineering fundamentals, including data pipeline design, optimization, and distributed data processing
Hands-on experience with Snowflake, Databricks, and/or Azure Synapse Analytics
Strong knowledge of AWS, Azure, or GCP and modern data warehouse/lakehouse architectures
Strong Python programming skills
Software engineering practices including testing, version control, and code review
API design and integration
Practical experience with LLM APIs and open-weight models
Prompt engineering and prompt evaluation
Understanding of context windows, tokenization, embeddings, hallucination, latency, and cost tradeoffs
RAG and data retrieval
Vector databases and embedding models
Chunking strategies, hybrid search, and reranking
Agentic systems and orchestration frameworks such as LangChain, LangGraph, or LlamaIndex
Tool-use/function-calling design, multi-step reasoning chains, agent memory, and state management
Understanding when to fine-tune versus prompt versus use RAG
Familiarity with parameter-efficient methods such as LoRA
Model evaluation frameworks, A/B testing for model outputs, and observability
Deployment patterns including latency/cost optimization, caching, streaming responses, and fallback handling
Versioning prompts and models
Bias/safety evaluation, guardrails, and appropriate handling of PII

Benefits

Comp & perks

Competitive salary
Opportunities for professional development and advancement
Medical insurance
Dental insurance
Vision insurance
Life insurance
Short-term disability
Long-term disability
Company match of 100% of a 6% contribution 401(k) plan
Employee Assistance Plan
Health Savings Account
Flexible Spending Account
Health Reimbursement Account
Wellness program