AIM Qualifications and Assessment Group

Data Engineer

• Design and develop enterprise-scale data solutions using Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Azure Databricks
• Build and optimize scalable ETL/ELT pipelines for structured, semi-structured, and unstructured data
• Develop Lakehouse, Data Warehouse, and Medallion Architecture solutions
• Design and implement batch and real-time data processing frameworks
• Lead data modernization, migration, and integration initiatives from legacy platforms
• Develop data foundations for AI, Machine Learning, and Generative AI solutions
• Prepare, transform, and optimize datasets for AI applications, model training, and fine-tuning
• Design and implement RAG architectures and vector search solutions
• Support Azure OpenAI, Microsoft Copilot, and enterprise LLM implementations
• Collaborate with Data Scientists and AI Engineers to operationalize AI and machine learning solutions
• Build distributed data processing solutions using PySpark and Spark SQL
• Develop automated integrations connecting APIs, SaaS applications, databases, and cloud services
• Design reusable data engineering frameworks, components, and standards
• Optimize workloads for performance, scalability, reliability, and cost efficiency
• Implement data quality, validation, monitoring, and error-handling processes
• Implement data governance and cataloging using Microsoft Purview
• Establish data lineage, classification, cataloging, and access controls
• Ensure alignment with security, privacy, compliance, SOC 2, NIST, and CMMC requirements
• Design and support semantic data models for Power BI and enterprise analytics
• Enable self-service analytics and support KPI frameworks, executive dashboards, reporting, and predictive analytics
• Provide technical guidance and mentorship to junior and mid-level data engineers
• Participate in solution architecture, technical design reviews, implementation planning, and presales activities
• Work with clients and stakeholders to translate business needs into scalable technical solutions
• Develop technical documentation, engineering standards, and reusable implementation patterns
• Troubleshoot data pipeline, platform, and integration issues
• Monitor and optimize data workloads and cloud resources
• Support enhancements, upgrades, and ongoing maintenance of data platforms
• Identify improvements in data quality, automation, performance, and operational efficiency
• Stay current with Microsoft Fabric, Azure, AI, Generative AI, and data engineering technologies

Senior Data Engineer – AI, LLM Solutions