Senior Observability Systems Engineer

Backblaze

. Set technical direction and standards for observability across the organization, including instrumentation conventions, alerting philosophy, and dashboard design patterns .

Posted 10/15/2026full-timeRemote • United StatesSenior💰 $178,000 - $200,000 per yearWebsite

Tech Stack

Tools & technologies

ConsulElasticSearchGrafanaKubernetesLinuxLogstashPrometheusPythonGo

About the role

Key responsibilities & impact

Set technical direction and standards for observability across the organization, including instrumentation conventions, alerting philosophy, and dashboard design patterns
Mentor engineers on the Observability team and act as a technical resource for SRE, Storage Engineering, and Network Engineering
Lead design reviews and represent the team in cross-functional architecture discussions
Evaluate build-versus-buy decisions for observability and AIOps tooling, including data sovereignty and compliance considerations
Produce and maintain design documentation, runbooks, and architectural decision records
Own and evolve the metrics pipeline, including VictoriaMetrics, Prometheus, and Consul-based service discovery
Own and expand the OpenSearch, Logstash, and Filebeat logging stack across Backblaze data centers
Architect and expand distributed tracing with Jaeger and OpenTelemetry
Define and maintain Grafana dashboard standards
Drive expansion of internal application telemetry with application teams
Define capacity and retention strategy for the telemetry stack
Own infrastructure-as-code for observability deployments across multiple Kubernetes clusters
Write and review automated playbooks for Debian-based bare metal servers
Partner with SRE on Consul-based service discovery and scrape target automation
Lead Ceph storage observability, including OSD health, performance metrics, and alerting integration
Collaborate with Network Engineering on BGP, routing, and network-level observability

Requirements

What you’ll need

8+ years in a systems engineering, SRE, DevOps, or observability-focused role, including demonstrated technical leadership on a platform or infrastructure team
Deep hands-on experience with Prometheus, VictoriaMetrics, or equivalent metrics platforms, including PromQL, Alertmanager, and operating these systems at scale
Experience designing and operating log aggregation pipelines such as Elasticsearch/OpenSearch, Logstash, Filebeat, or similar
Working knowledge of distributed tracing (Jaeger, OpenTelemetry, or equivalent), including instrumentation strategy, not just operation
Proficiency in Python and/or Go for automation and tooling
Experience with Kubernetes, including deploying workloads and understanding pod networking and storage
Experience with infrastructure-as-code tooling and configuration management
Strong understanding of Linux internals, with Debian/Ubuntu preferred
Demonstrated experience mentoring engineers and raising the technical bar on a team
Excellent written and verbal communication skills
Experience leading the evaluation or deployment of AIOps tooling, including alert correlation, deduplication, or automated root cause analysis
Experience building, deploying, or operating AI agents or agentic AI systems
Familiarity with LLM APIs (Anthropic, OpenAI) or self-hosted LLM inference in an infrastructure context
Familiarity with Ceph storage, including OSD management, pool configuration, and CRUSH maps
Experience with incident management platforms
Familiarity with BGP, network observability, or working in multi-DC environments
Experience with secrets management in production environments
Experience operating in a regulated or compliance-driven environment
Prior experience in a storage-focused company or hyperscale infrastructure environment

Benefits

Comp & perks

10% target annual bonus
RSUs
Equal Opportunity Employer commitment
Diversity, equity, and inclusion values
Workforce belonging commitment