FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Senior Data Engineer
Seneca Holdings. Design, build, and maintain scalable data pipelines within a Databricks E2 environment hosted on AWS .
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in building and maintaining scalable data pipelines within a Databricks E2 environment on AWS, utilizing advanced SQL, Python, and Spark for data processing and analytics. Proficient in implementing medallion architecture and ensuring compliance with security and governance standards.
Highest-signal resume keywords
Databricks DevelopmentAWS Data Pipeline ManagementAdvanced SQL SkillsApache Spark and PySparkGitLab and CI/CD
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Data Pipeline DesignSQL AggregationsPython ProgrammingR ProgrammingData Cleansing and TransformationDelta Lake ManagementReal-Time Data IngestionData Processing Logic DevelopmentSource-to-Target MappingUnit Testing
Soft Skills
CollaborationAgile Project Management
Tools & Technologies
DatabricksAWS DMSGitLabCI/CD PipelinesPandasApache Spark
Industry Keywords
FISMA ComplianceMedallion ArchitectureData Lakehouse DesignChange Data CaptureData Quality Analysis
Tech Stack
Tools & technologiesApacheAWSPandasPySparkPythonSparkSQL
About the role
Key responsibilities & impact- Design, build, and maintain scalable data pipelines within a Databricks E2 environment hosted on AWS
- Analyze and collect data from relational databases, APIs, external data providers, and real-time streaming sources
- Design and implement pipelines to cleanse, transform, and aggregate data for reporting and analytics
- Build and maintain Databricks medallion architecture using Bronze, Silver, and Gold layers
- Develop source-to-target mapping documentation and conduct unit testing
- Write advanced SQL aggregations and analyze data for anomalies, quality issues, and inconsistencies
- Implement real-time and near-real-time ingestion using AWS DMS and other AWS-native services
- Develop data-processing logic using Python and/or R with Spark, PySpark, and Pandas
- Manage source code, versioning, and deployments using GitLab and CI/CD pipelines
- Collaborate with cross-functional teams in an Agile project environment
- Ensure alignment with FISMA High and multi-tenant security, governance, and compliance requirements
- Leverage AI automation tools for pipeline development, testing, validation, and delivery
Requirements
What you’ll need- Proven, hands-on experience as a Data Engineer or Databricks Developer building production-grade data pipelines
- Strong working knowledge of Databricks on AWS, including E2 architecture, cluster configuration, job orchestration, and workspace management
- Hands-on experience with Apache Spark, PySpark, and Pandas for large-scale, distributed data processing
- Solid programming skills in Python; working knowledge of R is a plus
- Experience with Databricks Auto Loader for scalable, incremental file ingestion
- Practical experience with AWS Database Migration Service (DMS) for change data capture and real-time data replication
- Strong experience with Delta Lake and Delta tables, including schema evolution, time travel, OPTIMIZE, Z-ORDER, and VACUUM
- Experience with Amazon RDS and other relational database sources used in data extraction and integration workflows
- Advanced SQL skills, including complex joins, window functions, aggregations, and performance tuning
- Solid understanding of medallion architecture and modern data lakehouse design principles
- Experience with GitLab and CI/CD pipelines for automated testing, build, and deployment of data engineering code
- Experience working in Agile/Scrum project environments
Benefits
Comp & perks- Competitive pay
- Medical, dental, vision, life, and disability benefits
- Voluntary critical illness, hospital, and accident benefits
- Health savings and flexible spending accounts
- Retirement 401K plan
- Paid leave programs
- Flexible work-life balance
- Professional development opportunities
- Performance and recognition programs
- Collaborative work environment
- Financial and non-financial benefits for Seneca Nation members