Senior Data Engineer

Sumo Logic

. Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake .

Posted 11/12/2026full-timeRemote • IndiaSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and optimizing scalable data pipelines using Databricks, Apache Spark, and Python, while ensuring data governance and integration with enterprise applications. Proficient in implementing medallion architecture and maintaining data workflows on AWS infrastructure.

Highest-signal resume keywords

Databricks ExpertisePython/PySpark ProficiencyData Pipeline DevelopmentAWS Cloud Infrastructure KnowledgeData Architecture and Modeling

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

DatabricksApache SparkPythonSQLDelta LakeData Ingestion PipelinesREST APIs IntegrationMedallion ArchitectureGit/GitHubCI/CD Practices

Soft Skills

Problem-Solving SkillsCommunication SkillsCollaboration Skills

Tools & Technologies

Databricks WorkflowsAuto LoaderLakeflow Spark Declarative PipelinesAmazon RedshiftAmazon EC2Unity Catalog

Industry Keywords

Data EngineeringData GovernanceIncremental ProcessingAnalytics-Ready DatasetsCloud Infrastructure

Tech Stack

Tools & technologies

Amazon RedshiftApacheAWSCloudEC2PySparkPythonSparkSQLUnity

About the role

Key responsibilities & impact

Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake
Design and implement Bronze, Silver, and Gold medallion architecture layers
Build and maintain integrations with enterprise applications, REST APIs, databases, and other source systems
Develop reusable ingestion frameworks for onboarding new data sources
Build batch and near-real-time ingestion pipelines with incremental loads, CDC, schema evolution, and error recovery
Configure and manage Databricks Workflows and implement pipelines using Auto Loader and Lakeflow Spark Declarative Pipelines
Optimize Spark and Delta Lake workloads, query performance, partitioning, joins, data-skew handling, and compute costs
Implement Unity Catalog governance, access controls, lineage, auditing, masking, and row-level security
Maintain data workflows on Amazon Redshift and Amazon EC2
Collaborate with Engineering and IT/Cloud teams on connectivity, IAM permissions, S3 access, deployments, and operational issues
Follow version control, code review, automated testing, CI/CD, and technical documentation practices
Partner with Data Analytics and business stakeholders to deliver trusted, documented, analytics-ready datasets
Provide technical guidance and improve automation, engineering standards, platform reliability, and resource utilization
Report directly to the Senior Manager - Business Insights & Analytics and collaborate with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders

Requirements

What you’ll need

5+ years of experience in data engineering or a related technical role, including developing and supporting production data pipelines
Strong hands-on expertise in Databricks, including developing, troubleshooting, and optimizing production workloads
Strong proficiency in Python/PySpark, SQL, and Apache Spark for scalable data processing and transformation
Practical experience with Delta Lake, including table management, incremental processing, and performance optimization
Experience with data architecture and modeling, including medallion architecture and the development of analytics-ready datasets
Experience with Databricks Workflows or equivalent orchestration solutions
Proven experience building data ingestion pipelines and integrating REST APIs, databases, and enterprise applications
Understanding of reliable data processing patterns, including retries, idempotency, schema evolution, backfills, and failure recovery
Working knowledge of AWS services and cloud infrastructure, particularly IAM roles and policies, S3 storage and access management, and secure connectivity between source systems and data platforms
Experience with Git/GitHub, code reviews, automated testing, and CI/CD practices
Strong problem-solving skills and the ability to independently design, build, deploy, and maintain production data solutions
Good communication and collaboration skills to work effectively with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders
Experience with Databricks Lakeflow Spark Declarative Pipelines, Auto Loader, and advanced incremental ingestion patterns is desirable
Experience implementing Unity Catalog governance is desirable
Experience with Databricks Genie, Genie Agents, and Genie Code is desirable
Experience in AI/ML, Data Science, LLMs, contextualization, and Amazon Bedrock is desirable

Benefits

Comp & perks

Remote work arrangement
AI-powered Talent Matching tool used in recruitment
Privacy and data protection compliance information provided