Data Engineer

• Design, develop, and maintain robust ETL pipelines and processes on Databricks.
• Troubleshoot and resolve Databricks and SSIS pipeline/package errors and performance issues.
• Optimize data flow performance and minimize data latency.
• Implement data quality checks, validations, and cleansing procedures.
• Develop and maintain Databricks pipelines and datasets using Python, Spark, and SQL.
• Design and implement batch and streaming data lake architectures for analytics and AI/ML workloads.
• Optimize Databricks jobs for performance and cost-effectiveness.
• Integrate Databricks with other data sources and systems.
• Migrate legacy SSIS packages and SQL Server procedures to Databricks pipelines.
• Participate in data warehouse design and implementation.
• Collaborate with business users on department reporting requirements.
• Maintain complex SSRS reports, dashboards, and visualizations.
• Troubleshoot SSRS report performance and data consistency issues.
• Collaborate with business users, data analysts, and IT teams.
• Document development work and procedures.
• Contribute to AI-assisted tooling, strategies, and best practices.
• Keep current with data integration, reporting, and data engineering advancements.