Senior Data Engineer II

Allstate

. Design, build, and maintain scalable batch and real-time data pipelines using Apache Spark, SQL, Python, and cloud-native data technologies .

Posted 9/17/2026full-timeRemote • Illinois • United StatesSenior💰 $90,700 - $153,925 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and implementing scalable data pipelines and architectures, particularly using Apache Spark, Python, and SQL. Proficient in data governance, quality frameworks, and CI/CD practices to ensure efficient data engineering processes.

Highest-signal resume keywords

Apache SparkPythonSQLData GovernanceCI/CD

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data Pipeline DevelopmentETL/ELT OptimizationDimensional ModelingLakehouse ArchitectureData Integration TechniquesPerformance TuningData Quality FrameworksInfrastructure as CodeAutomated Testing PracticesAI-Assisted Development

Soft Skills

Technical LeadershipMentorshipCollaborationContinuous ImprovementCommunication

Tools & Technologies

Microsoft FabricOneLakeDelta FormatsParquet FormatsKafkaEvent StreamsAgentic AICloud Analytics PlatformsData VaultKimball Methodology

Certifications & Qualifications

Cloud Engineering Certifications

Industry Keywords

Data EngineeringEnterprise Data PlatformsMedallion ArchitectureAgile Delivery ProcessesData Reliability

Tech Stack

Tools & technologies

ApacheCloudETLKafkaPythonSparkSQLVault

About the role

Key responsibilities & impact

Design, build, and maintain scalable batch and real-time data pipelines using Apache Spark, SQL, Python, and cloud-native data technologies
Architect and implement Medallion Lakehouse data patterns for enterprise analytics, reporting, operational intelligence, and AI use cases
Develop and optimize ETL/ELT pipelines from diverse enterprise source systems
Design and implement dimensional models including fact and dimension tables
Build and manage data workloads within Microsoft Fabric, OneLake, and modern Lakehouse platforms
Establish data quality, observability, and governance frameworks with automated validation, reconciliation, monitoring, and alerting
Optimize Spark workloads for performance, scalability, reliability, and cost efficiency
Manage schema evolution, data lineage, metadata, and source-system changes
Develop reusable frameworks, accelerators, and engineering standards
Leverage Agentic AI-assisted development for code generation, testing, documentation, troubleshooting, and productivity
Implement CI/CD and DevSecOps practices for automated testing, deployment, release management, and rollback
Lead technical design reviews and mentor junior and mid-level data engineers
Collaborate with architects, stakeholders, analytics engineers, and data scientists to translate business requirements into scalable solutions
Partner with platform, security, privacy, and governance teams to ensure compliance
Drive continuous improvement across data engineering practices, architecture standards, and Agile delivery processes
Serve as a technical leader and trusted advisor
Champion modern data architecture, reusable standards, and automation-first approaches
Drive adoption of AI-assisted engineering practices
Provide technical leadership, mentorship, and guidance without direct people management responsibilities

Requirements

What you’ll need

7+ years of experience as a Data Engineer building and supporting enterprise-scale data platforms and pipelines
Strong hands-on experience with Apache Spark
Advanced proficiency in Python, SQL, and modern data engineering development practices
Experience designing and implementing Lakehouse architectures and Medallion Architecture patterns
Strong understanding of dimensional modeling, including star schemas, snowflake schemas, fact tables, dimension tables, and semantic data models
Experience with Microsoft Fabric, OneLake, Delta/Parquet formats, or comparable cloud analytics platforms
Expertise in data integration, transformation, and optimization techniques for large-scale analytical workloads
Experience implementing CI/CD pipelines, source control, Infrastructure as Code (IaC), and automated testing practices
Familiarity with AI-powered engineering tools and Agentic AI-assisted software development practices
Strong understanding of distributed computing, performance tuning, scalability, fault tolerance, and data reliability
Experience leading technical initiatives and mentoring engineering team members
Preferred: experience with Kafka, Event Streams, or similar technologies
Preferred: experience designing enterprise data products within Microsoft Fabric
Preferred: familiarity with Data Vault, Kimball, or other enterprise data modeling methodologies
Preferred: experience supporting machine learning, generative AI, or advanced analytics workloads
Preferred: knowledge of data governance, metadata management, master data management, and data quality frameworks
Preferred: cloud or data engineering certifications
Required background investigation
Must have reliable internet with minimum speeds of 50 MB download and 5 MB upload when working from home
Must have a dedicated, private workspace free from distractions with appropriate desk and seating
Allstate generally does not sponsor individuals for employment-based visas for this position

Benefits

Comp & perks

Comprehensive technology setup including a laptop, monitors, headset, keyboard, and mouse
Monthly connectivity reimbursement for employees eligible to work from home
Remote work arrangement
Dedicated private workspace and reliable internet requirements support for home working