Staff Security Data Engineer

Adobe

. Own end-to-end designs shaping the Security Data Platform architecture on Databricks and AWS using Spark, Delta Lake, and Unity Catalog .

Posted 9/28/2026full-timeSan Jose • California • United StatesLead💰 $208,300 - $301,600 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates extensive expertise in designing and implementing complex distributed systems using Spark, Databricks, and AWS, with a strong focus on data quality, observability, and operational readiness. Proven ability to lead cross-functional teams, mentor engineers, and drive measurable improvements in platform performance and cost efficiency.

Highest-signal resume keywords

Spark ExpertiseDatabricks ProficiencyAWS ExperienceProduction Terraform SkillsAirflow Orchestration

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Python ProgrammingAdvanced SQLData ModelingQuery OptimizationDistributed Systems DesignCI/CD PracticesData Quality AssurancePerformance TuningIncident ResponseAutomated Testing

Soft Skills

Effective Technical CommunicationMulti-Functional InfluenceMentoring

Tools & Technologies

TerraformAirflowMWAADelta LakeUnity CatalogS3IAMVPC NetworkingCribl StreamVector

Industry Keywords

Security TelemetrySIEMSOAROCSFData ClassificationData RetentionCloud InfrastructureLakehouse ArchitectureOperational AutomationTelemetry Routing

Tech Stack

Tools & technologies

AirflowAWSCloudDistributed SystemsPythonSparkSQLTerraformUnity

About the role

Key responsibilities & impact

Own end-to-end designs shaping the Security Data Platform architecture on Databricks and AWS using Spark, Delta Lake, and Unity Catalog
Translate ambiguous needs into design proposals, implementation plans, and production capabilities
Build and evolve distributed batch and streaming pipelines for EDR telemetry, network logs, identity events, cloud audit logs, and other security data sources
Write and review production Python and SQL code, develop platform features, refactor unreliable code, and establish engineering patterns
Lead code and design reviews and own test strategy and automation choices
Build and maintain tested Terraform modules for AWS and Databricks infrastructure, including S3, IAM, VPC networking, and MWAA
Automate delivery through version control and CI/CD
Design layered lakehouse data models and normalization patterns, including OCSF mappings where appropriate
Establish data contracts, quality checks, cataloging, classification, lineage, access controls, and retention policies
Engineer and operate Airflow and MWAA orchestration, including dependencies, retries, backfills, alerting, and automated recovery
Define service level objectives and instrument data freshness, completeness, pipeline health, and query performance
Diagnose and resolve complex production issues across Spark, storage, orchestration, and cloud infrastructure
Improve incident response, runbooks, recovery testing, and operational readiness
Lead measurable cost and performance improvements through query and Spark tuning, storage lifecycle policies, partitioning, and compute right-sizing
Lead cross-team technical discussions and communicate decisions, risks, and business impact
Partner with detection and security analytics engineers on SIEM modernization and automated detection deployment
Mentor engineers through pairing, technical reviews, documentation, and knowledge sharing
Evaluate and introduce platform innovations, including AI-assisted engineering and operational automation

Requirements

What you’ll need

Typically 10 or more years in data or software engineering, with demonstrated Staff-level scope and impact
Experience personally delivering complex distributed systems and leading full-system design and implementation across team boundaries
Deep production expertise in Spark traditional and real-time workloads, Databricks, Delta Lake, and lakehouse architecture
Strong Python and advanced SQL skills for data modeling, profiling, and query optimization
Strong AWS experience across storage, compute, networking, and IAM
Production Terraform experience designing, testing, and maintaining reusable infrastructure modules
Practical understanding of distributed systems, including partitioning, replication, consistency, back pressure, scalability, and fault tolerance
Experience with Airflow, MWAA, or equivalent orchestration
Experience with production observability, data quality, automated testing, CI/CD, and reliable deployment and recovery practices
Demonstrated ability to improve platform cost, performance, and reliability with measurable results
Effective technical communication and multi-functional influence
Preferred: experience with security telemetry, OCSF or comparable security schemas, and data requirements for SIEM, SOAR, threat detection, or investigations
Preferred: experience with Unity Catalog governance, data classification and retention, regional data controls, or multi-region platform design
Preferred: experience with Cribl Stream, Vector, or similar telemetry routing and enrichment tools

Benefits

Comp & perks

Annual Incentive Plan (AIP)
Certain roles may be eligible for a new hire equity award
Comprehensive benefits programs
Accommodation support for applicants with disabilities or special needs