Scicom Infrastructure Services

Data Engineer

• Bridge business, contracting, data-governance, and engineering teams for a large-scale Databricks implementation
• Lead requirements-gathering sessions with procurement, contracting, program, analytics, governance, and technical stakeholders
• Define, construct, document, and maintain data contracts between data producers and consumers
• Translate business requirements into user stories, acceptance criteria, process flows, data mappings, interface specifications, and technical requirements
• Conduct source-system analysis, data profiling, gap assessments, and source-to-target mapping
• Identify differences between existing procurement data and required OCDS structures
• Facilitate agreement among data owners, producers, consumers, architects, and governance teams
• Maintain traceability through data models, engineering implementation, testing, and acceptance
• Evaluate requested changes for downstream effects on data products, reports, integrations, and analytical use cases
• Support backlog refinement, sprint planning, demonstrations, testing, and stakeholder acceptance
• Apply OCDS to procurement and public-contracting datasets across planning, tender, award, contract, and implementation stages
• Develop and maintain mappings for OCDS releases, records, packages, identifiers, organizations, parties, items, milestones, documents, transactions, amendments, and related elements
• Interpret and apply OCDS schemas, codelists, validation rules, and implementation guidance
• Determine when OCDS extensions are necessary and document assumptions, business rules, mappings, extensions, and exceptions
• Construct contracting records from multiple transactional releases and establish rules for amendments, updates, cancellations, corrections, and historical changes
• Validate transformed data against OCDS JSON schemas and resolve missing, invalid, inconsistent, or nonconforming procurement data
• Support publication, exchange, analysis, or internal use of standardized contracting data
• Design, develop, test, and maintain Databricks data pipelines using Apache Spark, PySpark, Python, and SQL
• Build ingestion and transformation pipelines for structured and semi-structured procurement data from JSON, CSV, XML, Parquet, relational databases, APIs, and files
• Implement Bronze, Silver, and Gold data layers using medallion architecture
• Build normalized, dimensional, analytical, and OCDS-aligned data models
• Use Delta Lake for schema enforcement, schema evolution, versioning, auditability, and reliable processing
• Develop reusable frameworks for mapping source procurement data into OCDS-compatible outputs
• Implement batch and incremental ingestion patterns
• Use Databricks Workflows, notebooks, jobs, Auto Loader, Delta Live Tables, or Lakeflow as appropriate
• Develop REST API integrations for source ingestion and downstream data delivery
• Implement automated data-quality, reconciliation, completeness, and conformity checks
• Support Unity Catalog implementation for metadata, ownership, lineage, access control, and data discovery
• Optimize Spark jobs, SQL queries, clusters, partitioning, file sizes, and data layouts
• Participate in code reviews, automated testing, CI/CD, deployment, monitoring, and production support
• Investigate pipeline failures, data discrepancies, and performance issues
• Define measurable data-quality rules for accuracy, completeness, validity, timeliness, consistency, and uniqueness
• Develop validation controls for OCDS fields, identifiers, dates, amounts, currencies, organizations, classifications, and contracting relationships
• Create dashboards or reports showing data-contract compliance and data-quality results
• Establish processes for detecting and managing schema drift
• Document data lineage from procurement systems through Databricks transformations and downstream products
• Work with governance teams to assign data owners, stewards, classifications, retention requirements, and access policies
• Ensure sensitive procurement and supplier information is handled according to security and privacy requirements
• Support auditability through documented rules, version-controlled mappings, validation results, and change histories

Explore Databricks Data Engineer – Data Contracts, OCDS at Scicom Infrastructure Services. Review requirements and application details.

Databricks Data Engineer – Data Contracts, OCDS

Operations

• Monitor the health, availability, performance, and capacity of Databricks workspaces, jobs, pipelines, clusters, SQL warehouses, and related cloud services
• Provide day-to-day operational support for production Databricks environments
• Monitor and troubleshoot ETL, ELT, streaming, API, and file-based data pipelines
• Restart, rerun, recover, reprocess, backfill, reconcile, and remediate failed workflows and data loads
• Review driver and executor logs, Spark UI metrics, cluster event logs, audit logs, and application telemetry to identify root causes
• Support Databricks Workflows, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, and batch-processing workloads
• Maintain cluster, serverless compute, instance pool, cluster policy, SQL warehouse, and environment configurations
• Coordinate platform upgrades, runtime and library updates, deployments, releases, changes, rollbacks, and post-deployment monitoring
• Serve as first- or second-level support for production incidents; triage, coordinate response, provide status updates, escalate, and conduct root-cause analyses
• Maintain incident, problem, service-request, change, deployment, and operational records
• Develop and maintain dashboards, alerts, operational reports, health indicators, SLIs, and SLOs
• Monitor job success, latency, processing duration, utilization, freshness, quality, and compute consumption
• Validate data contracts, schemas, record counts, control totals, data quality, lineage, audit evidence, and governance requirements
• Support identity, access, entitlement, Unity Catalog, secrets, security, privacy, backup, recovery, retention, archival, and disaster-recovery activities
• Analyze resource consumption, job duration, query performance, storage, and workload patterns; recommend cost and performance optimizations
• Create and maintain operational procedures, troubleshooting guides, escalation matrices, runbooks, inventories, configuration documentation, and recovery requirements
• Identify automation and continuous-improvement opportunities and participate in operational-readiness reviews

Explore Data Operations Specialist at Scicom Infrastructure Services. Review requirements and application details.