Senior Data Engineer

Allstate

. Design, develop, and enhance scalable data pipelines and data processing systems .

Posted 10/30/2026full-timeRemote • Illinois • United StatesSenior💰 $80,000 - $136,625 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates strong proficiency in developing scalable data pipelines using PySpark and Python, with hands-on experience in modern data engineering frameworks and tools. Capable of optimizing data workflows for performance and cost efficiency while ensuring data quality and compliance.

Highest-signal resume keywords

PySpark DevelopmentData Pipeline OptimizationSQL and NoSQL DatabasesMicrosoft Fabric NotebooksContainerization with Docker and Kubernetes

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data Pipeline DevelopmentData ModelingData Architecture PrinciplesBatch and Real-Time Streaming SolutionsSQL Query OptimizationCI/CD ImplementationMonitoring and Observability SolutionsCloud Data Services (Azure, AWS)Apache SparkKafka

Soft Skills

Strong CommunicationAnalytical SkillsTroubleshootingProblem-SolvingCollaboration

Tools & Technologies

Microsoft FabricDatadogDockerKubernetesJenkinsGitHub ActionsAzure DevOpsFlinkDbtGenerative AI Tools

Industry Keywords

Data QualityData IntegrityData LineageRegulatory ComplianceMetadata ManagementOperational ResiliencePerformance MetricsIterative EngineeringProduct-Centric EnvironmentsBackground Investigation

Tech Stack

Tools & technologies

ApacheAWSAzureCloudDockerJenkinsKafkaKubernetesNoSQLPySparkPythonSparkSQL

About the role

Key responsibilities & impact

Design, develop, and enhance scalable data pipelines and data processing systems
Build reusable ingestion, transformation, and storage patterns
Integrate data from APIs, databases, event streams, and third-party systems
Implement data models and schemas creating unified views of business operations
Apply version control, testing, CI/CD, and automated deployments
Ensure data quality, integrity, and reliability through validation, monitoring, and observability
Optimize workflows for performance, cost efficiency, and operational resilience
Document data flows, lineage, and technical components
Collaborate with product managers, engineers, and analysts to understand data requirements
Participate in iteration planning, daily standups, retrospectives, and product ceremonies
Contribute to metadata management, data lineage, privacy, security, and regulatory compliance
Evaluate new technologies, frameworks, and patterns
Share knowledge and mentor less experienced engineers
Own the end-to-end lifecycle of data products from design and implementation through production support, data quality, and performance metrics

Requirements

What you’ll need

Strong proficiency in developing scalable data pipelines using PySpark and Python
Hands-on experience with modern data engineering frameworks and tools, including Apache Spark, Kafka, Flink, dbt, or equivalent technologies
Strong knowledge of SQL, NoSQL databases, data modeling, and data architecture principles
Demonstrated ability to optimize SQL queries, data pipelines, and storage for performance, scalability, and cost efficiency
Experience designing and developing batch and/or real-time streaming data solutions
Hands-on experience with Microsoft Fabric Notebooks for developing, debugging, testing, and operationalizing data engineering workflows
Experience with containerization and orchestration technologies, such as Docker and Kubernetes
Experience implementing monitoring and observability solutions; Datadog experience preferred
Ability to work effectively in collaborative, iterative, and product-centric engineering environments
Strong communication, analytical, troubleshooting, and problem-solving skills
Experience with cloud data services (Azure, AWS) preferred
Hands-on experience with Microsoft Fabric components, including Lakehouse, Warehouse, Data Pipelines, Dataflows Gen2, and Semantic Models preferred
Familiarity with CI/CD pipelines, such as Jenkins, GitHub Actions, and Azure DevOps preferred
Familiarity with generative and agentic AI tools preferred
Must submit to a background investigation
Allstate generally does not sponsor individuals for employment-based visas for this position
Dedicated, private workspace free from distractions with appropriate desk and seating
Reliable internet with minimum speeds of 50 MB download and 5 MB upload

Benefits

Comp & perks

Comprehensive technology setup including a laptop, monitors, headset, keyboard, and mouse
Monthly connectivity reimbursement to help offset internet costs
Remote work arrangement