Apply

Ready to go for it?

AI Apply speeds things up—apply directly if you prefer.

FREE ACCESS
5,000–10,000 jobs/day
Scoutfield Logo

See all jobs on Scoutfield

Search thousands of fresh jobs every day.

Discover
  • Fresh listings
  • Fast filters
  • No subscription required
Create a free account and start exploring right away.
Circana

Senior Director, Software Engineering

Circana

. Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow .

Posted 10/30/2026full-timeRemote • Illinois • United StatesSenior💰 $120,000 - $165,000 per yearWebsite

Core Competencies

Role fit
Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and optimizing scalable data workflows using Python, PySpark, and Apache Spark, while ensuring data quality and governance. Proven ability to lead teams, implement CI/CD pipelines, and leverage cloud technologies for efficient data processing.

Highest-signal resume keywords
Python ProgrammingApache Spark ExpertiseData Engineering LeadershipCI/CD ImplementationCloud Technologies Knowledge

ATS Keywords

Tailor your resume
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills
Data Workflow DesignPySpark DevelopmentSQL ProgrammingData Quality EnforcementPerformance TuningData PartitioningAirflow DAG ManagementReal-Time Data ProcessingBatch Data ProcessingMachine Learning Support
Soft Skills
Strong CommunicationCollaboration SkillsTechnical Mentorship
Tools & Technologies
Apache AirflowGitDockerKubernetesTerraformAzureAWSGoogle Cloud
Industry Keywords
Data EngineeringAgileScrumData WarehouseData LakeAI-Assisted EngineeringGenerative AI Tools

Tech Stack

Tools & technologies
AirflowApacheAWSAzureCloudDockerKubernetesPySparkPythonSparkSQLTerraform

About the role

Key responsibilities & impact
  • Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow
  • Implement real-time and batch data processing using Spark
  • Enforce data quality, governance, and security best practices throughout the data lifecycle
  • Monitor and automate data systems to ensure availability, reliability, and performance
  • Manage cloud infrastructure and optimize data-processing costs
  • Implement CI/CD pipelines for reliable data-workflow deployments
  • Build and optimize large-scale data-processing pipelines using Apache Spark and PySpark
  • Implement data partitioning, caching, and performance tuning for Spark workloads
  • Work with structured and unstructured data to support analytics and machine-learning initiatives
  • Design and maintain Airflow DAGs to automate complex data workflows
  • Monitor, troubleshoot, and optimize job execution and dependencies
  • Lead a team of data engineers and provide technical guidance and mentorship
  • Promote coding standards, version control, and documentation best practices
  • Engage directly with clients to understand business needs and communicate technical concepts
  • Collaborate with engineering, product, and architecture peers to drive scalable, maintainable products
  • Partner with Program Management to align delivery timelines and mitigate risks

Requirements

What you’ll need
  • 8+ years of experience in data engineering
  • Expertise in Python, PySpark, Apache Spark, and Apache Airflow
  • Strong programming skills in Python and SQL
  • Ability to write efficient and maintainable code
  • Deep understanding of Spark internals, including RDDs, DataFrames, DAG execution, and partitioning
  • Experience with Airflow DAGs, scheduling, and dependency management
  • Knowledge of Git, Docker, Kubernetes, Terraform, and DevOps best practices for CI/CD workflows
  • Experience optimizing large-scale data processing
  • Experience leading teams and working in Agile/Scrum environments
  • Experience with common data warehouse and/or data lake platforms
  • Knowledge of cloud technologies, including Azure, AWS, or Google Cloud
  • Proven experience using generative AI tools for software development, data engineering, system design, testing, and troubleshooting
  • Strong understanding of AI-assisted engineering practices, including prompt engineering, code generation, automated testing, and AI-driven productivity workflows
  • Ability to identify opportunities to apply AI solutions to improve productivity, code quality, and operational efficiency
  • Strong communication and collaboration skills
  • Experience with ML Engineering is a plus

Benefits

Comp & perks
  • Flexible and accommodating work environment
  • Paid time off
  • Medical insurance
  • Dental insurance
  • Vision insurance
  • 401(k)
  • Bonus pay eligibility
  • Accessibility accommodations for applicants with disabilities
  • Background checks conducted in accordance with local legislation
  • Current employer will not be contacted without permission