FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in designing and developing ETL/ELT pipelines using PySpark and SQL within Databricks on AWS, while ensuring data quality and governance through effective monitoring tools. Proficient in optimizing processes and implementing best practices for data security and privacy.
Highest-signal resume keywords
ETL/ELT Pipeline DevelopmentPySpark ProficiencyAWS Infrastructure ExperienceData Quality MonitoringGit-Based Version Control
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Apache SparkPythonSQLData Pipeline DevelopmentCode OptimizationData MigrationStructured StreamingData Formats (Text, Delta, Parquet, JSON, CSV, XML)DevOpsCI/CD
Soft Skills
TeamworkRelationship ManagementClient ManagementInfluencing SkillsWillingness to Embrace Modern Technology
Tools & Technologies
DatabricksAWS S3Atlassian StackWeb API FrameworksMonitoring Tools
Certifications & Qualifications
Bachelor's Degree
Industry Keywords
Data GovernanceData QualityData SecurityData RetentionAutomation
Tech Stack
Tools & technologiesApacheAWSETLPySparkPythonSparkSQL
About the role
Key responsibilities & impact- Design, develop, test, deploy and monitor data pipelines in Databricks on AWS from a wide variety of data sources.
- Design, develop, test, deploy and monitor scalable code with PySpark and SQL in Databricks.
- Identify opportunities to improve internal process through code optimisation and automation.
- Build data quality dashboards, lineage flows and/or monitoring tools to utilize the data pipeline, providing active monitoring and actionable insight into overall data quality and data governance.
- Assist in migrating data from legacy systems onto newly developed solutions.
- Follow and lead best practices on all data security, retention, and privacy policies.
Requirements
What you’ll need- Bachelor’s degree.
- 3+ years’ experience of building ETL/ELT pipelines.
- Proven competency in solution design, development, implementation, reporting and analysis.
- Proficiency in Apache-Spark, Python and SQL languages.
- Proficiency in working with Text, Delta, Parquet, JSON, CSV, and XML data formats.
- Working knowledge of Spark structured streaming.
- AWS infrastructure experience, specifically working with S3.
- Solid understanding of git-based version control, DevOps, and CI/CD. Experience of working on Atlassian stack a plus.
- Knowledge of common web API frameworks and web services.
- Strong teamwork, relationship, and client management skills, and the ability to influence peers and senior management to accomplish team goals.
- Willingness to embrace modern technology, best practice, and ways of work.