FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Senior Databricks Data Engineer
State Street. Design, build, and test end-to-end data pipelines using streaming, events, and batch ingestion on Azure and Cosmos DB .
Tech Stack
Tools & technologiesApacheAzureCloudDockerETLKubernetesScalaShell ScriptingSparkSQLUnix
About the role
Key responsibilities & impact- Design, build, and test end-to-end data pipelines using streaming, events, and batch ingestion on Azure and Cosmos DB
- Develop and implement ETL processes for data ingestion, transformation, and loading into data lakes
- Work extensively on Databricks and data warehousing concepts
- Design and develop custom high-throughput, configurable frameworks and libraries
- Drive change through collaboration, influence, and demonstrations of POCs
- Handle all software development lifecycle activities including design, coding, integration testing, deployment, and documentation
- Collaborate within an agile project team
- Follow best practices and coding standards
- Monitor and troubleshoot data pipelines
- Implement data quality checks and validations
- Contribute to platform development, enhancement, and maintenance as part of an agile scrum team
- Grow personal skillset
Requirements
What you’ll need- Bachelor’s Degree level qualification in a computer or IT related subject
- 10+ years of overall Bigdata data pipeline experience
- 5+ years of Databricks hands on experience
- 5+ years of experience on cloud-based development including Azure Services, Azure Devops, Kubernetes, Docker
- Experience in micro-services architecture and understanding of Cloud Computing; Azure preferred
- Proficient working knowledge of Databricks, Delta Lake, and Apache Spark
- Hands-on development experience in Databricks SQL and Scala
- Strong understanding of Databricks platform including clusters, jobs, and other resources
- Experience troubleshooting DevOps pipelines using ADO and Harness
- Experience using version control such as Git
- Experience handling semi-structured data including Avro, JSON, and XML
- Basic knowledge of shell scripting, UNIX, TOAD, and SQL Developer
- Experience performing data analysis and data exploration
- Experience working in the financial industry
- Experience with agile development methodology
- Effective written and oral professional communication
- Ability to multitask; energetic, fast learner, and problem solver
Benefits
Comp & perks- Retirement savings plan (401K) with company match
- Insurance coverage including basic life, medical, dental, vision, long-term disability, and other optional additional coverages
- Paid time off including vacation, sick leave, short term disability, and family care responsibilities
- Access to Employee Assistance Program
- Incentive compensation including eligibility for annual performance-based awards
- Eligibility for certain tax advantaged savings plans
- Inclusive development opportunities
- Flexible work-life support
- Paid volunteer days
- Employee networks