Senior Data Engineer – Databricks

AVANTTi

. Develop and implement data engineering solutions using Databricks .

Posted 11/7/2026full-timeRemote • BrazilSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in Data Engineering with a strong focus on developing and optimizing data pipelines using Databricks, Apache Spark, and Delta Lake. Proficient in ensuring data quality, integrity, and observability while collaborating with cross-functional teams to deliver innovative data solutions.

Highest-signal resume keywords

Data EngineeringDatabricksApache SparkPipeline DevelopmentData Quality

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

DatabricksApache SparkDelta LakeDelta Live TablesYAML PipelinesSQLPythonPySparkAmazon RedshiftGoogle BigQuery

Soft Skills

AnalyticalProblem-SolvingCommunicationCollaborationProactive

Tools & Technologies

Apache AirflowDbtDatabricks JobsDatabricks Auto LoaderGreat Expectations

Industry Keywords

Batch ProcessingMicro-batch ProcessingStreaming ProcessingData QualityObservability

Tech Stack

Tools & technologies

AirflowAmazon RedshiftApacheBigQueryJavaPySparkPythonSnowflakeSparkSQLUnity

About the role

Key responsibilities & impact

Develop and implement data engineering solutions using Databricks
Build, maintain, and optimize scalable pipelines for Batch, Micro-batch, and Streaming processing
Implement solutions with Apache Spark, Delta Lake, Delta Live Tables (Lakeflow Declarative Pipelines), Unity Catalog, and Apache Iceberg
Develop YAML-driven pipelines
Implement incremental ingestion and automated processing with Databricks Auto Loader
Develop and manage workflows and orchestrations with Databricks Jobs/Workflows, Apache Airflow, or dbt
Ensure data quality, integrity, traceability, and observability
Implement pipeline auditing, monitoring, and validation
Participate in technical design and architecture sessions
Translate complex requirements into structured technical activities, ensuring quality and timely delivery
Collaborate with Solution Architects and cross-functional teams to implement customer data platforms
Participate in the Databricks Center of Excellence by creating accelerators, reusable components, and technical improvements
Track trends and new capabilities in the Databricks ecosystem, applying continuous innovation

Requirements

What you’ll need

Strong experience in Data Engineering, with at least 2 years of hands-on experience developing and designing pipelines using Databricks
Proficiency in pipeline development and data processing in distributed environments
Hands-on experience with Databricks, Delta Lake, Delta Live Tables (Lakeflow Declarative Pipelines), Unity Catalog, Apache Iceberg / Uniform, and Databricks Auto Loader
Experience building scalable, configuration-driven YAML pipelines
Knowledge of Batch, Micro-batch, and Streaming ingestion and processing patterns
Experience with Databricks Jobs/Workflows, Apache Airflow, or dbt
Knowledge of data quality, auditing, and observability, including Databricks, Great Expectations, Collibra, or dbt
Proficiency in Apache Spark, Presto, and Amazon EMR
Experience with Amazon Redshift, Snowflake, and Google BigQuery
Strong knowledge of SQL, Stored Procedures, and object-oriented programming
Experience with Python, PySpark, or Java
Familiarity with DevOps and CI/CD, including Databricks Asset Bundles, automated validation, and testing
Knowledge of version control, automation, and deployment of data solutions
Analytical, curious, and focused on solving complex problems
Technical autonomy and the ability to lead activities with minimal supervision
Comfortable working in cross-functional teams
Ability to break down complex technical challenges into structured deliverables
Proactive in identifying opportunities for improvement and innovation
Strong communication skills and the ability to share knowledge with the team
Commitment to quality, efficiency, and meeting delivery deadlines