Senior Data Engineer – Databricks, AI-Enabled Data Platforms

Keep IT Simple

. Design, develop, and maintain scalable data pipelines using Databricks, Spark, and cloud-native technologies .

Posted 10/9/2026full-timeSão Paulo • BrazilSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and implementing scalable data pipelines using Databricks and Apache Spark, with a strong focus on cloud-native architectures across Azure and AWS. Proficient in data governance, MLOps, and building optimized ETL/ELT frameworks to support AI and machine learning workloads.

Highest-signal resume keywords

DatabricksApache SparkDelta LakeCloud Data PlatformsMLOps

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data EngineeringData WarehousingETL/ELT PipelinesPythonSQLPowerShellData ModelingPerformance OptimizationData Quality FrameworksFeature Engineering

Soft Skills

Technical LeadershipStrategic ThinkingProblem SolvingStakeholder ManagementCommunication Skills

Tools & Technologies

Azure Data FactoryDatabricks WorkflowsKafka/Event StreamingGitHubAzure DevOpsTerraformUnity CatalogCI/CD PipelinesInfrastructure as CodeMonitoring Solutions

Certifications & Qualifications

Databricks Certified Data Engineer ProfessionalAzure Data Engineer AssociateAWS Data Analytics Certification

Industry Keywords

Cloud SecurityData GovernanceAI SolutionsRegulated IndustriesData Modernization

Tech Stack

Tools & technologies

ApacheAWSAzureCloudETLKafkaOraclePostgresPySparkPythonSparkSQLTerraformUnity

About the role

Key responsibilities & impact

Design, develop, and maintain scalable data pipelines using Databricks, Spark, and cloud-native technologies
Build and optimize batch and real-time data ingestion frameworks
Develop reusable engineering patterns, frameworks, and accelerators for enterprise data solutions
Implement data quality, observability, lineage, and governance capabilities across the data ecosystem
Support large-scale data modernization and cloud migration initiatives
Design and implement Delta Lake architectures and Medallion data models
Build optimized ETL/ELT pipelines using Databricks notebooks, workflows, and Delta Live Tables
Establish performance tuning, cost optimization, and workload management best practices
Leverage Unity Catalog for data governance, security, and metadata management
Implement CI/CD pipelines and Infrastructure-as-Code for Databricks deployments
Design cloud-native data architectures across Azure and AWS environments
Integrate enterprise data sources including SQL Server, Oracle, APIs, SaaS platforms, and streaming platforms
Build resilient and secure data services supporting high availability and disaster recovery requirements
Partner with infrastructure and security teams to implement cloud security controls and compliance standards
Build and optimize data products supporting AI, machine learning, and generative AI workloads
Develop feature engineering pipelines and curated datasets for model training and inference
Support vector databases, semantic search, RAG, and LLM-powered applications
Collaborate with Data Scientists and AI Engineers to operationalize machine learning solutions
Implement MLOps and DataOps practices to support scalable AI delivery
Serve as a technical lead for complex data engineering initiatives
Define architecture standards, engineering best practices, and development guidelines
Mentor junior and mid-level engineers
Conduct architecture reviews and provide technical recommendations
Drive adoption of automation, observability, and platform engineering principles

Requirements

What you’ll need

Bachelor's degree in Computer Science, Information Technology, Engineering, or related discipline
8+ years of experience in data engineering, data warehousing, or data platform development
5+ years of hands-on experience with Databricks and Apache Spark
Experience designing and implementing enterprise-scale cloud data platforms
Experience leading technical initiatives and mentoring engineering teams
Experience with Databricks, Apache Spark (PySpark, Spark SQL), Delta Lake, Delta Live Tables, Unity Catalog, data warehousing, and data modeling
Experience with Python, SQL, and PowerShell or scripting automation
Experience with Microsoft Azure, AWS, and cloud-native storage and compute services
Experience with SQL Server, Oracle, PostgreSQL, and databases
Experience with Azure Data Factory, Databricks Workflows, Kafka/Event Streaming, and REST APIs
Experience with GitHub, Azure DevOps, CI/CD pipelines, Terraform, and Infrastructure as Code
Knowledge of monitoring and alerting solutions, data quality frameworks, performance optimization, logging, and troubleshooting
Knowledge of machine learning data pipelines, feature stores, MLOps, generative AI concepts, vector databases, RAG, AI governance, and responsible AI principles
Preferred: Databricks Certified Data Engineer Professional, Azure Data Engineer Associate certification, AWS Data Analytics certification
Preferred experience supporting regulated industries, implementing enterprise data governance programs, and integrating AI solutions into production environments
Strategic thinking, problem solving, technical leadership, architecture design, stakeholder management, communication and presentation skills, continuous improvement, innovation and AI adoption, and risk and compliance awareness

Benefits

Comp & perks

Modelo de contratação: PJ (Pessoa Jurídica)
Forma de atuação híbrida, com possibilidade de trabalho remoto parcial