Senior Databricks Data Engineer – Vice President

State Street

. Design, build, and test end-to-end data pipelines, including streaming, events, and batch data ingestion, on cloud-based infrastructure using Azure and Cosmos DB .

Posted 10/16/2026full-timeUnited StatesSenior💰 $120,000 - $202,500 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and implementing end-to-end data pipelines using Azure and Databricks, with a strong focus on ETL processes, data quality, and troubleshooting. Proficient in cloud-based development and agile methodologies, ensuring effective collaboration and communication within project teams.

Highest-signal resume keywords

Azure ServicesDatabricks SQLApache SparkETL ProcessesData Pipeline Troubleshooting

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data Pipeline DevelopmentETL ProcessesDatabricksScalaCloud ComputingMicro-Services ArchitectureData AnalysisData Quality ChecksShell ScriptingSQL

Soft Skills

Effective CommunicationCollaborationInfluencing

Tools & Technologies

Azure DevOpsKubernetesDockerGitSnowflake

Industry Keywords

Financial IndustryAgile DevelopmentData WarehousingSemi-Structured Data

Tech Stack

Tools & technologies

ApacheAzureCloudDockerETLKubernetesScalaShell ScriptingSparkSQLUnix

About the role

Key responsibilities & impact

Design, build, and test end-to-end data pipelines, including streaming, events, and batch data ingestion, on cloud-based infrastructure using Azure and Cosmos DB
Develop and implement ETL processes for data ingestion, transformation, and loading data into data lakes
Work extensively on Databricks and data warehousing concepts
Design and develop custom high-throughput and configurable frameworks and libraries
Drive change through collaboration, influence, and demonstration of POCs
Handle all aspects of the software development lifecycle, including design, coding, integration testing, deployment, and documentation
Work collaboratively within an agile project team
Follow best practices and coding standards
Monitor and troubleshoot data pipelines to identify and resolve issues
Implement data quality checks and validations to ensure data accuracy
Perform data analysis and data exploration
Grow personal skillset

Requirements

What you’ll need

Bachelor’s Degree level qualification in a computer or IT related subject
10+ years of overall Bigdata data pipeline experience
5+ years of Databricks hands on experience
5+ years of experience on cloud-based development including Azure Services, Azure Devops, Kubernetes, Docker
Experience in micro-services architecture and understanding of Cloud Computing; Azure preferred
Proficient working knowledge of Databricks, Delta lake and Apache Spark
Hands-on development experience in Databricks SQL and Scala
Strong understanding of the Databricks platform, including clusters, jobs and other resources
Strong hands-on experience troubleshooting DevOps pipelines (ADO and Harness)
Experience using version control such as Git
Experience handling semi-structured data including Avro, JSON and XML
Basic knowledge of shell scripting, UNIX, TOAD, and SQL Developer
Effective professional written and oral communication
Experience working in the financial industry
Experience with agile development methodology
Experience on Snowflake is a plus

Benefits

Comp & perks

Retirement savings plan (401K) with company match
Insurance coverage including basic life, medical, dental, vision, long-term disability, and other optional additional coverages
Paid time off including vacation, sick leave, short term disability, and family care responsibilities
Employee Assistance Program
Incentive compensation including eligibility for annual performance-based awards
Eligibility for certain tax advantaged savings plans
Inclusive development opportunities
Flexible work-life support
Paid volunteer days
Employee networks