Data Engineer

• Compare Apache Spark and DuckDB on existing transformations to select suitable transformations for migration
• Implement DuckDB versions of selected transformations with automated equivalence tests against Spark versions
• Develop and test a way to run DuckDB transformations on Databricks, including compute type, dependencies, and data access
• Modify pipeline orchestration so each step declares whether it runs on Spark or DuckDB
• Create a command to run DuckDB transformations locally with sample data or cloud storage data
• Ensure one code version for each transformation runs locally and on Databricks without changes
• Build a science module that groups products into likely complements and substitutes and assembles product hierarchies from product descriptions
• Develop a prototype AI debugging agent, its read-only toolset, and an evaluation framework for identifying root causes of forecast differences
• Benchmark deep learning forecasting models against production models and analyze when they improve promotional demand forecasting performance

Explore Data Engineer Intern, Data Science Intern at Cognira. Review requirements and application details.