Senior Data Engineer – AWS, PySpark, Kafka

Leega

. Design and evolve event-driven architecture on AWS, using Kafka as the backbone between transactional systems and the analytics platform .

Posted 10/22/2026contractRemote • BrazilSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and evolving event-driven architectures on AWS, utilizing Kafka and PySpark for data engineering. Proficient in implementing Change Data Capture (CDC), managing data schemas, and ensuring compliance with industry regulations.

Highest-signal resume keywords

Event-Driven Architecture DesignApache Kafka ExpertiseAWS Ecosystem ProficiencyPySpark DevelopmentInfrastructure As Code

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Kafka Producers And ConsumersChange Data Capture (CDC)Advanced SQLSchema Registry ManagementData Lake ArchitectureEvent PersistencePerformance OptimizationAutomated TestingData Engineering Best PracticesNumerical Correctness

Soft Skills

Clear Written CommunicationIndependent Issue InvestigationMentoring And Technical Reference

Tools & Technologies

AWS GlueAWS EMRAWS RedshiftTerraformCloudFormationKafka ConnectDebeziumIcebergDeltaGit

Industry Keywords

Financial Markets ExperienceRegulatory Data ComplianceLGPD ComplianceEvent ContractsData Quality Tools

Tech Stack

Tools & technologies

AirflowAmazon RedshiftApacheAWSKafkaPySparkPythonSparkSQLTerraform

About the role

Key responsibilities & impact

Design and evolve event-driven architecture on AWS, using Kafka as the backbone between transactional systems and the analytics platform
Develop Kafka producers and consumers with PySpark (Spark Structured Streaming), managing offsets, checkpointing, idempotency, late data, and delivery semantics
Define and advocate for the design of topics, keys, partitioning, retention, compaction, and event contracts
Govern schemas in Schema Registry, defining compatibility and evolution without breaking consumers
Implement CDC for transactional databases via Kafka Connect or Debezium
Integrate sources such as custodians, distributors, and market data providers’ APIs
Build event persistence in the S3 Data Lake using Iceberg or Delta, integrating Glue, EMR, and Redshift
Optimize performance and costs in Kafka and Spark, including lag, rebalancing, brokers, micro-batches, skew, and spill
Instrument observability and automated quality controls for event flows
Ensure lineage, traceability, access controls, encryption, sensitivity-based segregation, and compliance with Brazil’s LGPD and industry regulations
Automate provisioning and deployment using infrastructure as code and CI/CD
Document architectural decisions (ADRs), participate in code reviews and pair programming, and help define standards
Collaborate with Wealth Management, Sales, Risk, Compliance, and Operations
Serve as a technical reference for mid-level and junior engineers

Requirements

What you’ll need

6+ years of experience in data engineering, including at least 2 years with architectural responsibility for event-driven platforms
Proven track record of launching and operating a Kafka streaming platform in production, including incident response and on-call support
Deep knowledge of Apache Kafka: topics, partitions, keys, consumer groups, offsets, retention, compaction, rebalancing, producer and consumer configuration, and lag troubleshooting
Hands-on experience with producers and consumers using PySpark (Spark Structured Streaming) integrated with Kafka, including checkpointing, triggers, watermarks, and state management
Strong experience with the AWS ecosystem: Amazon MSK, S3, Glue, EMR, Lambda, IAM, and Redshift or equivalent technologies
Python applied to data engineering, with modular, testable, and version-controlled code, type annotations, and automated tests
Advanced SQL, including window functions, CTEs, and execution plan analysis
Experience with Schema Registry and Avro, Protobuf, or JSON Schema formats, including schema compatibility and evolution
Experience with Kafka Connect or Debezium for Change Data Capture (CDC)
Infrastructure as code: Terraform, CloudFormation, or CDK
Git, code review, and CI/CD applied to data engineering
Experience with open table formats such as Iceberg or Delta and Lakehouse architecture on S3
Strong focus on numerical correctness and delivery guarantees
Clear written communication for documenting event architecture
Ability to independently investigate issues end to end
Ability to serve as a technical reference through code reviews, pair programming, and mentoring
Ability to translate business needs into sustainable event designs
Honesty when estimating scope, risk, and timelines
Nice to have: financial markets experience; regulatory data involving CVM, BACEN, BSM, or ANBIMA; Confluent; Kafka Streams; ksqlDB; Apache Flink; Kinesis; Airflow; Step Functions; dbt; data quality, catalog, or observability tools; and AWS or Confluent certifications

Benefits

Comp & perks

Ongoing professional development
Dynamic and collaborative environment
100% remote work
Opportunity to work on financial markets and investment projects
Work with AWS, Kafka, PySpark, and data engineering technologies
Inclusive company: applications from people with disabilities are welcome