Principal Software Engineer

HubSpot

. Define the architecture and evolution of HubSpot’s core observability platform covering distributed tracing, metrics, and structured logging .

Posted 9/18/2026full-timeRemote • United StatesLead💰 $313,800 - $502,080 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in architecting and evolving observability platforms, with a focus on telemetry pipeline design, instrumentation standards, and AI observability. Proven ability to lead technical strategy, mentor engineering teams, and drive OpenTelemetry adoption while ensuring high-quality production code delivery.

Highest-signal resume keywords

Telemetry Pipeline DesignOpenTelemetry ProficiencyDistributed Tracing ExpertiseReal-Time Telemetry TriggersSLO/SLI Design

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Telemetry Pipeline DesignInstrumentation API DesignProduction Code ShippingDynamic SamplingData FidelityHigh-Cardinality Data ManagementAutomated Operational WorkflowsAI ObservabilityIncident Response PatternsDebugging Workflows

Soft Skills

Mentoring Senior EngineersTechnical Strategy InfluenceCollaboration with Engineering Teams

Tools & Technologies

OpenTelemetryDistributed Tracing EnginesMetric BackendsLog Ingestion Infrastructure

Industry Keywords

ObservabilityTelemetryAI SystemsML SystemsMicroservices

Tech Stack

Tools & technologies

Distributed SystemsMicroservices

About the role

Key responsibilities & impact

Define the architecture and evolution of HubSpot’s core observability platform covering distributed tracing, metrics, and structured logging
Set instrumentation standards across a large, polyglot engineering organization
Lead the technical strategy for tracing and understanding AI agents and ML-powered systems in production
Define telemetry primitives, standards, and debugging workflows for AI and ML systems
Architect high-cardinality, high-throughput telemetry pipelines and storage systems
Balance sampling, fidelity, retention, cost, query latency, and developer ergonomics
Ship production code and lead design reviews and high-impact initiatives from prototype to production
Design instrumentation APIs and libraries and drive OpenTelemetry adoption
Build tooling that turns raw telemetry into actionable operational signals
Define SLO/SLI, alerting, and proactive incident-response patterns
Partner with infrastructure, platform, and product engineering teams to address signal gaps
Influence technical strategy and translate observability considerations into product and operational decisions
Mentor senior engineers and tech leads
Capture learnings from major incidents and large-scale migrations

Requirements

What you’ll need

Proven experience building observability or telemetry tooling for internal engineering teams
Experience architecting developer platforms serving thousands of engineers across a large organization
Deep expertise in telemetry pipeline design, including high-cardinality data, dynamic sampling, query latency, retention economics, and data fidelity
Strong technical fluency with OpenTelemetry, distributed tracing engines, metric backends, and large-scale log ingestion infrastructure
Proven track record linking telemetry signals to automated operational workflows
Experience designing real-time telemetry triggers for automated remediation, dynamic runbooks, or AI-assisted root-cause diagnosis across microservices environments
Experience with large-scale distributed systems, developer platform design, and AI observability
Ability to ship production code and lead high-impact initiatives end-to-end
Ability to design instrumentation APIs and libraries and drive OpenTelemetry adoption
Experience defining SLO/SLI design and alerting practices
Ability to partner with infrastructure, platform, and product engineering teams
Ability to mentor senior engineers and tech leads

Benefits

Comp & perks

Equity plan participation with restricted stock units (RSUs) for eligible roles
Annual bonus targets under HubSpot’s bonus plan for eligible roles
On-target commission for employees in eligible roles
Overtime pay for eligible roles
Flexible work arrangements
In-person onboarding at a regional HubSpot office for Engineering team members
In-person Product Group Summit and other gatherings for broader Product team members
Accommodation support for travel limitations or other reasons
Disability accommodations and assistance