Software Development Engineer Intern
Amazon Web Services (AWS) · May 2025 – Aug 2025
The Amazon EKS Metrics Service collects and publishes cluster metrics, but a CloudWatch API outage could leave permanent gaps in monitoring data. I designed and built a four-component backfill system in Go that preserves failed work and replays it in order, turning outages from permanent data loss into a bounded processing delay.
Architecture
Failure-to-recovery flow (the steps, not the four components)
- 01
Detect
A failed CloudWatch call is caught, real or simulated
- 02
Save
The complete request is stored in a structured layout
- 03
Reprocess
Failed reads are retried in the background
- 04
Republish
Failed writes are retried in the background
- 05
Back up
Durable storage with a lifecycle for long outages
- 06
Replay
Chronological replay through the live code path
Key decisions
DECISION 01
Save the full request, not just a timestamp
Preserving everything makes accurate replay possible instead of a lossy approximation.
→ Zero data loss on recovery
DECISION 02
Replay through the same code path as live data, in order
Recovered data takes the normal route, so dashboards fill in gaps instead of showing data out of sequence.
→ Outages become a bounded delay
DECISION 03
Inject failures from configuration, in test environments only
Failure handling could previously be exercised only during a real outage.
→ Recovery verified end to end; cannot trigger in a live environment
The problem
The service depends on CloudWatch APIs to read and publish metrics. When one of them went down, the existing retry logic and circuit breakers eventually gave up and the affected data was abandoned for good, leaving gaps in dashboards and making historical trends unreliable.
There was also no safe way to test any of this: failure handling could only be exercised during a real outage, when the stakes were highest.
Approach
The system follows a "save first, process later" principle. The moment a failure is detected, the complete request is preserved in a structured layout instead of just a timestamp, which is what makes accurate replay possible rather than a lossy approximation.
Four components share the work: failure simulation and detection, reprocessing of failed reads, republishing of failed writes, and durable backup with a storage lifecycle for extended outages. Recovery runs in the background on staggered schedules, and saved data is always replayed in chronological order through the same code path as live data, so recovered dashboards fill in gaps instead of showing data out of sequence.
Making failure testable
A configuration-driven failure simulation framework lets the team inject failures at chosen rates and verify recovery on demand, instead of waiting for a real outage. It is only enabled in test environments, so the same capability used for testing cannot trigger in a live one.
Validation
Verified end to end by simulating an outage, confirming the data was fully preserved, then removing the failure and confirming automatic recovery, cross-checked against before-and-after monitoring dashboards. Covered by component-level, end-to-end and concurrency tests (including Go's race detector) for the parts designed to run unattended in the background.