Senior Databricks Data Engineer

State Street

. Design and develop custom high-throughput and configurable data pipelines using Databricks .

Posted 9/16/2026full-timeQuincy • New Jersey • United StatesSenior💰 $90,000 - $157,500 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and developing scalable ETL pipelines and data ingestion solutions using Databricks SQL and Scala/PySpark on cloud-based infrastructure. Proven ability to optimize data pipelines and troubleshoot DevOps processes while collaborating effectively with cross-functional teams.

Highest-signal resume keywords

Databricks SQLScala/PySparkCloud-Based DevelopmentData EngineeringAgile Development Methodology

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data Pipeline DevelopmentETL Pipeline DesignData Integration SolutionsPerformance OptimizationRoot Cause AnalysisData Lakehouse SolutionsMicroservices ArchitectureAPI DevelopmentData CurationBatch Processing

Soft Skills

Critical ThinkingCommunicationProblem-SolvingTeam CollaborationMultitasking

Tools & Technologies

DatabricksAzureAWSDevOpsSQL

Industry Keywords

Financial IndustryData EngineeringCloud InfrastructureAgile Methodology

Tech Stack

Tools & technologies

AWSAzureCloudETLMicroservicesPySparkScalaSQL

About the role

Key responsibilities & impact

Design and develop custom high-throughput and configurable data pipelines using Databricks
Drive change through collaboration, influence, and demonstration of proofs of concept
Design, build, and test end-to-end data ingestion pipelines for streaming, events, and batch processing
Develop data integration and data curation solutions
Design, develop, and deploy scalable ETL pipelines on cloud-based infrastructure using Databricks SQL and Scala/PySpark
Optimize and tune Databricks job performance, including query optimization, resource management, and scaling strategies
Address data and environment issues
Perform impact analysis, root cause analysis, and corrective, adaptive, and perfective maintenance
Manage all aspects of the software development lifecycle, including design, coding, integration testing, deployment, and documentation
Work onsite four days per week and remotely one day per week
Work standard business hours aligned to project and stakeholder needs, 8am to 5pm

Requirements

What you’ll need

Bachelor’s Degree level qualification in a computer or IT related subject
7+ years of overall IT experience
5+ years of Data Engineering experience handling large datasets
3+ years of Databricks hands-on experience using SQL & Scala/PySpark
3+ years of experience on cloud-based development using Azure/AWS
Strong hands-on experience with Databricks SQL & Scala/PySpark for data engineering pipelines
Strong hands-on experience troubleshooting DevOps pipelines and Azure services
Expertise building end-to-end Data Lakehouse solutions using Databricks
Experience with Microservices and API/Event Driven Architecture/Development
Strong critical thinking, communication, and problem-solving skills
Professional written and oral communication
Experience working in the financial industry
Experience collaborating with team members, users, and stakeholders across geographical locations
Ability to multitask, meet aggressive timelines, and maintain strong work ethics
Experience with agile development methodology
Team player with a positive attitude, enthusiasm, initiative, and self-motivation

Benefits

Comp & perks

Retirement savings plan (401K) with company match
Basic life insurance
Medical insurance
Dental insurance
Vision insurance
Long-term disability insurance
Optional additional insurance coverages
Paid vacation leave
Paid sick leave
Short-term disability
Family care responsibilities leave
Employee Assistance Program
Incentive compensation, including eligibility for annual performance-based awards
Eligibility for certain tax-advantaged savings plans
Inclusive development opportunities
Flexible work-life support
Paid volunteer days
Employee networks