All work

Cloud observability / Case study

StackPulse

See where a request spent its time.

Type
Open-source engineering
Release
v1.1.0
Stack
Python 3.12 · FastAPI · OpenTelemetry · Docker Compose
Automated tests
83

System architecture

  1. W3C Trace DAG & Critical-Path Analyzer (stackpulse/tracing/analyzer.py)
  2. Service Topology & Multi-Window SLO Engine (stackpulse/slo/engine.py)
  3. Chaos Workload Simulator & Incident Correlator
Three layers from the project’s architecture blueprint

01 / Problem

Why it exists

During microservice latency spikes or retry storms, engineers waste critical minutes staring at flat span lists without knowing which downstream service owns the true critical path or whether error budgets are burning at a page-worthy rate.

02 / Design

How it works

Merge overlapping child intervals before computing exclusive self-duration, and pair 1h/6h fast-burn windows with 24h/72h slow-burn windows.

  1. W3C Trace DAG & Critical-Path Analyzer (stackpulse/tracing/analyzer.py)

    Assembles flat spans into a directed tree, computes interval-union exclusive time, walks the critical path, and flags N+1 query patterns.

  2. Service Topology & Multi-Window SLO Engine (stackpulse/slo/engine.py)

    Tracks caller-to-callee latency percentiles and evaluates 1h/6h and 24h/72h error-budget burn rates.

  3. Chaos Workload Simulator & Incident Correlator

    Generates deterministic baseline, DB pool saturation, N+1 regression, and 503 retry-storm workloads and exports SRE postmortems.

Interval-Union Exclusive Duration & Multi-Window Burn-Rate Alerting
Context
Naive subtraction of child span durations fails when a parent service issues concurrent RPC calls, and single-window error-rate alerts either page too late or reset too slowly.
Decision
Merge overlapping child intervals before computing exclusive self-duration, and pair 1h/6h fast-burn windows with 24h/72h slow-burn windows.
Trade-offs
Requires sorting child span intervals per parent node (O(k log k)), which executes in ~51 µs per trace.
Verification
Verified across 83 pytest tests in tests/test_stackpulse.py, tests/test_otlp_and_alerts.py, and tests/test_server_hardening.py, and benchmarked across 500 traces (3,625 spans).
Security and operational constraints
  • Bounded in-memory ring-buffer store prevents unbounded memory exhaustion under high span ingestion load.
  • Self-hostable Docker Compose architecture keeps telemetry data inside organizational boundaries.
  • Input validation on W3C trace/span identifiers and attribute payloads.
  • Self-exclusive span duration uses interval-union merging so concurrent child spans never cause negative or over-subtracted parent self-time.
  • PAGE_CRITICAL alerts fire only when both short (1h) and long (6h) windows exceed the 14.4x burn-rate threshold, preventing flapping.
  • All chaos scenarios are deterministic and reproducible via CLI seed.

03 / Evidence

How to verify it

Repository-reported / Trace DAG Critical-Path Speed

19,525 traces/s · 51.2 µs/trace

Reconstructs W3C span trees, identifies critical-path bottlenecks, and flags N+1 query patterns

Reproduce from the command line
git clone https://github.com/codesphere-u/stackpulse.git
cd stackpulse && pytest -v
python -m stackpulse.cli simulate --scenario all
View source on GitHub
Next case study · Local AI Auralis Local Search private documents on your own machine.