FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Lead Software Engineer – Data Engineer
Caterpillar Inc.. Lead and mentor a team of data engineers and platform developers .
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates extensive expertise in Data Engineering and Data Platform roles, with a strong focus on AWS services, Snowflake architecture, and building scalable data ingestion pipelines. Proven ability to lead teams, drive Agile execution, and integrate AI capabilities into data workflows.
Highest-signal resume keywords
AWS Data EcosystemSnowflake ArchitectureData Ingestion PipelinesCI/CD ToolsLeadership in Engineering Teams
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Python ProgrammingSQL ProgrammingData ModelingETL/ELT FrameworksReal-Time StreamingDistributed SystemsSoftware Engineering PrinciplesAgile MethodologiesData GovernanceMetadata Management
Soft Skills
LeadershipCommunicationStakeholder Management
Tools & Technologies
AWS S3AWS GlueAWS LambdaAWS EMRAWS RedshiftJenkinsAzure DevOpsGitHubKafkaSpark Streaming
Certifications & Qualifications
Bachelor’s Degree in Computer ScienceMaster’s Degree in Engineering
Industry Keywords
Data EngineeringData PlatformsReal-Time AnalyticsAI IntegrationManufacturing Environments
Tech Stack
Tools & technologiesAmazon RedshiftAWSAzureDistributed SystemsETLJenkinsKafkaPythonSnowflakeSparkSQL
About the role
Key responsibilities & impact- Lead and mentor a team of data engineers and platform developers
- Drive Agile execution and ensure predictable, high-quality delivery
- Establish engineering best practices, code quality, and CI/CD standards
- Architect scalable and secure data platforms on AWS
- Design robust data ingestion frameworks for batch and near real-time pipelines
- Define best practices in data modeling, governance, and metadata management
- Lead design and development of scalable ingestion pipelines for structured and unstructured data
- Build and optimize Snowflake-based data platforms for performance and cost
- Enable ingestion of diverse sources including databases, APIs, files, and streaming data
- Leverage AWS services including S3, Glue, Lambda, EMR, and Redshift for end-to-end pipelines
- Implement CI/CD pipelines using Azure DevOps and Jenkins
- Ensure system scalability, resiliency, and operational readiness
- Enforce modular design, code quality, testing, and version control
- Drive automation and continuous improvement
- Promote reusable frameworks for ingestion and transformation
- Partner with product managers, SMEs, and business stakeholders
- Translate business needs into scalable data solutions
- Collaborate with Data Science and AI teams to operationalize ML models and analytics workflows
- Promote AI integration into data engineering pipelines
- Support real-time analytics and edge AI use cases in manufacturing environments
- Use AI extensively in building and testing data ingestion and data pipelines
Requirements
What you’ll need- 10+ years of experience in Data Engineering / Data Platform roles
- Strong experience in AWS data ecosystem (S3, Glue, Lambda, EMR, Redshift)
- Deep expertise in Snowflake (architecture, optimization, data modeling)
- Strong programming skills in Python and SQL
- Extensive experience with data ingestion pipelines and ETL/ELT frameworks
- Exposure to real-time streaming (Kafka, Spark Streaming)
- Experience with CI/CD tools (GitHub, Jenkins, AWS CloudFormation etc.)
- Solid understanding of distributed systems and scalable architectures
- Strong foundation in software engineering principles (Git, testing, design patterns)
- Experienced in working with Agile teams
- Collaborate with Data Science and AI teams to operationalize ML models and analytics workflows
- Promote integration of AI capabilities into data engineering pipelines (e.g., GenAI, MCP, ATA)
- Support real-time analytics and edge AI use cases in manufacturing environments
- Use AI extensively in building and testing Data Ingestion and Data pipeline
- Bachelor’s or Master’s degree in Computer Science / Engineering
- Proven track record in leading engineering teams and delivering at scale
- Strong leadership, communication, and stakeholder management skills
Benefits
Comp & perks- Relocation is available for this position