FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Senior Data Engineer – Databricks, AI Platforms
Keep IT Simple. Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL .
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in designing and developing scalable data pipelines and data products using Databricks, Spark, Python, and SQL, while ensuring compliance with data governance and security standards. Proficient in implementing Lakehouse architectures and supporting AI and machine learning initiatives through robust data solutions.
Highest-signal resume keywords
DatabricksApache SparkPythonSQLData Governance
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Data EngineeringETL DevelopmentData Pipeline OptimizationCloud-Based Data LakesData WarehousingFeature EngineeringMLOpsData ModelingGenerative AILarge Language Models
Soft Skills
Technical LeadershipMentoring
Tools & Technologies
Databricks Delta LakeUnity CatalogMLflowKafkaEvent Hub
Certifications & Qualifications
Databricks Certified Data Engineer Professional
Industry Keywords
DataOpsCI/CDData ObservabilityData SecurityCloud Data Platforms
Tech Stack
Tools & technologiesApacheAWSAzureCloudETLKafkaPythonSparkSQLUnity
About the role
Key responsibilities & impact- Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL
- Build and optimize batch, streaming, and real-time data integration solutions from enterprise and third-party data sources
- Implement Databricks Lakehouse architectures using Delta Lake and Medallion design patterns
- Develop and maintain data products supporting analytics, predictive modeling, machine learning, and Generative AI applications
- Collaborate with Data Scientists and AI Engineers to prepare, transform, and govern data for AI and ML use cases
- Design and implement feature engineering pipelines and support ML lifecycle processes
- Develop data solutions supporting RAG, vector search, semantic search, and LLM-based applications
- Optimize Spark jobs, SQL workloads, and data processing frameworks for performance, scalability, and cost efficiency
- Implement data quality, observability, lineage, governance, and monitoring capabilities
- Ensure compliance with data privacy, security, and responsible AI standards
- Contribute to CI/CD, Infrastructure-as-Code, and DataOps practices across the data platform
- Mentor junior engineers and promote engineering best practices
- Partner with data scientists, AI engineers, architects, and business stakeholders to deliver trusted, governed data products
- Enable enterprise AI and Generative AI initiatives through robust, secure, and governed data platforms
Requirements
What you’ll need- Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, or a related field
- 7+ years of experience in data engineering, ETL development, or large-scale data platform engineering
- 3+ years of hands-on experience with Databricks and Apache Spark
- Strong proficiency in Python, SQL, and distributed data processing frameworks
- Experience building cloud-based data lakes, data warehouses, and Lakehouse architectures
- Experience supporting AI, machine learning, or advanced analytics initiatives
- Strong understanding of data modeling, data governance, and enterprise data management practices
- Experience developing and optimizing large-scale data pipelines in AWS, Azure, or Google Cloud
- Experience with Databricks Delta Lake, Unity Catalog, Delta Live Tables, MLflow, Mosaic AI, and Databricks Workflows
- Hands-on experience supporting Generative AI, Large Language Models (LLMs), RAG architectures, vector databases, or AI-powered applications
- Familiarity with LangChain, Semantic Kernel, OpenAI APIs, Hugging Face, or similar technologies
- Experience with feature stores, model training pipelines, and MLOps
- Experience with Kafka, Event Hub, or other streaming technologies
- Databricks Certified Data Engineer Professional or equivalent cloud certification (preferred)
- Databricks Platform Engineering
- Apache Spark Development
- AI & Machine Learning Data Engineering
- Generative AI Data Solutions
- Lakehouse Architecture
- Data Modeling & Data Warehousing
- Data Governance & Security
- Data Observability & Reliability Engineering
- Cloud Data Platforms (AWS/Azure)
- DataOps, CI/CD & Automation
- Technical Leadership & Mentoring
Benefits
Comp & perks- Contratação como PJ (Pessoa Jurídica)
- Modelo de trabalho híbrido, com 3 dias presenciais por semana no escritório de Pinheiros/SP