Incident Monitoring & Response
Problem Management & Post-Incident Analysis (RCA)
Quality Assurance, Testing & Operations
Development Practices & Release Management
Security & Financial Risk Management
Leadership Accountability
Monitoring only works if the team still trusts it — and response speed depends on knowledge that isn't locked inside one person's head.
Real signals get buried in that noise. MTTR stretches from 15 minutes to 45+ as engineers second-guess even valid alerts — a payment-latency alert once got ignored for 40 minutes until a customer complained.
What It Is
Alerts toggling on/off repeatedly (disk: 89%→87%→91%) are treated as benign noise. Teams normalise and stop investigating. Real degradation gets buried in false positives.
Why It Matters
- Real issues are buried in false positives
- MTTR extends from 15 min to 45+ min due to second-guessing
- Engineers ignore critical alerts — alert fatigue + flapping = disaster
Real Scenario
Payment latency alert fires whenever p99 >500ms. Brief spikes cause constant fire-and-clear. After 3 weeks, engineers stop looking. Genuine degradation fires the same alert, ignored 40 minutes until a customer complaint arrives.
Observable Symptoms
- Alert volume >200/day; <50% actionable
- Same alert fires >10× in a 2-hour window
- Team closes alerts via dashboard with zero investigation
- Post-mortem: "Alert was firing all day; we thought it was noise"
Prevention & Correction
- Alert tuning: hysteresis (fire on >500ms AND stays >500ms for 2 consecutive checks)
- Alert deduplication: same alert within 5 min = 1 notification
- Baseline suppression during known high-variance periods
- Monthly audit: for each alert <30% precision, root-cause and fix
Success Metrics
- Alert precision >70%
- P1/P2 MTTD <10 minutes
- Team confidence: "If the alert fired, we trust it"
15 minutes is a contractual SLA, not an aspiration. A detection gap of over 10 minutes — for example, monitoring that isn't integrated with the alerting system — is enough to blow through it.
What It Is
P1/P2 incidents resolve after 30, 60, or 90+ minutes. Root causes: monitoring not integrated with alert systems, lack of up-to-date runbooks, unclear triage procedures.
Why It Matters
- 15-minute SLA is contractual — breach = loss of customer trust
- Each additional minute compounds TPV loss (peak: ~600M VND/min; normal: ~370M VND/min)
- Customers escalate to social media — brand damage is outsized
Real Scenario
API Gateway exhausts worker processes. Alerts in a legacy monitoring system not integrated with GOC. On-call finds out via customer ticket at T+25 minutes.
Observable Symptoms
- MTTR for P1s >30 minutes with no clear downward trend
- Detection gap >10 minutes (incident started 10:00 AM, alert fired 10:10 AM)
- Triage phase >5 minutes; on-call unclear what to check first
- Runbooks outdated or referring to deprecated services
Prevention & Correction
- Alert integration: all monitoring metrics must route to GOC
- Runbook automation: critical paths scripted; runbook links to scripts, not prose
- MTTD target: <5 minutes from incident start to alert
- Post-mortem: break down detection/triage/resolution time; if any phase >5 min, document the failure
Success Metrics
- P1 MTTR <15 min (first alert to service recovery)
- P1 MTTD <5 min
- 100% of P1-critical paths have automated runbooks
- Triage decision tree covers >95% of common incidents
That's a bus factor of 1. When they're out sick or leave, incidents take 2–3× longer to resolve — and the hero burns out from being the bottleneck.
What It Is
One engineer is the only person who can resolve a class of incidents. They maintain critical knowledge in their head, don't write runbooks, and others don't practice incident handling for these systems.
Why It Matters
- Bus factor = 1 — if engineer leaves, quits, or is sick, team is helpless
- Incidents take 2–3× longer when the hero is unavailable
- Hero burnout; they become a bottleneck; team can't grow
Real Scenario
ETL pipeline has occasional hung-worker issues. Only one engineer understands queue drain order. While on leave, a worker hang causes a 3-hour pipeline delay, resolved only after calling them on their day off.
Observable Symptoms
- Same engineer paged >2× per week for a class of incidents
- Post-mortem: "(Name) debugged and restarted the service"
- Runbooks say: "Contact (Name) for database issues"
- When hero is on vacation, resolution time jumps 3–5×
Prevention & Correction
- Write a runbook immediately after every incident the hero handles
- Runbook minimum: decision tree + 5 debugging steps + script links + escalation path
- Pair training: hero + backup handle next 3 incidents together; backup drives
- Quarterly simulation drills: test runbooks without the hero present
Success Metrics
- Bus factor >2 for all critical incident types
- Runbook hit-rate >80%
- MTTR similar with hero vs. backup (within 10%)
- Zero incidents with notes "waiting for (Name) to debug"
With 200+ alerts/day and under 30% actionable, real signals get lost in the noise — and the team spends 2+ hours a day managing alerts instead of building.
What It Is
Engineers receive >50 alerts/hour; <30% actionable. After weeks of noise, developers develop alert blindness — auto-acknowledge and dismiss without reading. Critical real alerts are missed.
Why It Matters
- Critical real issues missed (similar alert title = assumed noise)
- MTTR extends because real alerts ignored for 30+ minutes
- Operational overhead: engineers spend 2+ hours/day managing alerts instead of building
Real Scenario
API with 50 endpoints: 100 base alerts + Redis (20) + DB (30) + infra (20) = 170 alerts. During peak deployment, all fire simultaneously. A genuine payment latency spike is dismissed as noise, missed for 35 minutes.
Observable Symptoms
- Daily alert count >200; team closes without action
- Alert-to-incident ratio: <20% of alerts result in an actual ticket
- Engineers disable or mute alerts ("too many false positives")
- Post-mortem: "We didn't see the alert; Teams was too noisy"
Prevention & Correction
- Alert quality: measure precision. If <60%, reduce volume before adding new alerts
- Alert consolidation: all latency alerts for a service = 1 alert with breakdown
- Severity levels: P1/P2 = phone call; P3/P4 = Teams channel only
- Monthly noisy-alert review: for each alert <30% precision, root-cause and fix
Success Metrics
- Alert precision (true positives / total fired) >70%
- P1 alerts have 100% precision
- On-call satisfaction: >80% say "I trust the alerts"
- MTTD for P1s: <5 min
1. Why does tolerating "flapping" alerts hurt Zalopay more than most companies?
Muted trust in alerts directly threatens the 15-minute SLA, which is tied to real money moving every minute.
2. What's the real risk of relying on one "hero" engineer?
A single point of knowledge is a single point of failure — the team's actual response capacity is only as strong as its weakest bus factor.
All three items below share one root myth — "the RCA is filed, so the incident is closed." At Zalopay, an RCA only counts once the fix is verified and owned.
If the fix was never actually deployed and verified, the incident comes back — and every recurrence is worse: more degraded data, higher load, less customer trust. One RCA for a Transfer API connection-pool exhaustion said "increase max connections" on June 5; the config change was never actually deployed, and the same incident recurred on September 2.
What It Is
The same root cause appears in >1 P1/P2 incident within a 90-day rolling window. Either the RCA process failed to identify the true root cause or failed to implement and verify the fix.
Why It Matters
- Wastes incident response time (same issue = re-triage, re-diagnose, re-fix)
- Each recurrence is exponentially worse (data degraded, load grown, trust lower)
- Signals dysfunction in problem-solving rigour
- 2nd occurrence has higher severity, longer MTTR, and customer escalation
Real Scenario
June 5: Connection pool exhaustion in Transfer API. On-call restarts service (symptom fix). RCA says 'increase max connections.' September 2: same service, same error — config change was never actually deployed. RCA closed without verifying the fix.
Observable Symptoms
- RCAs show duplicate root cause
- Post-mortems show: "Fix was deployed but incomplete"
- Incident notes: "Same as incident #1234 on June 5th"
- "This issue happened to us last month" (customer feedback)
Prevention & Correction
- RCA taxonomy: tag every RCA with root cause category for searchability
- Weekly query across RCA database; flag duplicate root causes
- Preventive action verification: action deployed before RCA is closed
- Escalation: recurring incident within 90 days = severity raised one level
Success Metrics
- 0 recurring root causes within 90 days
- 100% of RCAs have documented preventive action
- 100% of preventive actions deployed before RCA closes
- 30-day post-deployment verification: no recurrence confirmed
The underlying defect (leak, misconfiguration, logic bug) is still there — the system becomes a ticking time bomb, and eventually the restart won't save it.
What It Is
Incident is 'resolved' by restarting a service, clearing a cache, or rebooting a server. The underlying defect (code bug, resource leak, misconfiguration) is never addressed. Same incident recurs.
Why It Matters
- Creates fragile "ticking time bomb" systems
- Next occurrence is a surprise; team hasn't prepared a real fix
- Customer loses confidence ("you just rebooted again?")
- Eventually the reboot doesn't work → extended P1 outage
Real Scenario
Redis cache fills and stops accepting writes. Incident response: flush cache. Resolved — for 10 days. Nobody investigates why. The application's cache-miss handler has an O(n²) loop generating 10× expected write volume.
Observable Symptoms
- RCA notes: "Restarted service. Issue resolved." — no root cause documented
- Action items: "Investigate connection pool settings" — vague, no owner
- Same service restarted >2× in a month
- Monitoring cycles: healthy → slow → errors → restart → healthy → repeat
Prevention & Correction
- RCA minimum: (1) symptom, (2) immediate action, (3) root cause, (4) evidence, (5) preventive action, (6) verification
- RCA template: enforce structure so 'restart and move on' is impossible
- Peer review: every RCA reviewed by senior engineer; symptom-fix RCAs rejected
- Trending: track 'days to repeat' for each incident class
Success Metrics
- 0% of RCAs with "restart/cache flush" as root cause
- 100% of RCAs have documented evidence of root cause
- Incident recurrence rate <10% per incident class per quarter
- 30-day post-RCA verification: >95% show "confirmed fixed"
By the time it's written days later, logs have already rotated away — one RCA filed 4 days after the incident ended up with just "Database migration may have caused slowdown. Team to review," and no action items were ever created.
What It Is
RCA is treated as a compliance checkbox, not a learning opportunity. Deferred days after the incident — team memory fades, logs rotate away. Final document lists symptoms instead of causes.
Why It Matters
- No systemic learning → same mistakes repeat
- Data loss: logs from incident are gone by the time RCA is done
- Action items don't get assigned; improvement never happens
Real Scenario
DB migration causes 10-minute payment timeout. On-call restarts connection pool. Report filed 4 days later. By then, logs have rotated. RCA contains: 'Database migration may have caused slowdown. Team to review.' No action items created.
Observable Symptoms
- RCA written >48 hours after incident
- RCA document is 1 paragraph: "Service went down; we fixed it"
- Root cause section: "Possible cache issue" (vague)
- Action items: "Team to discuss" (no owner, no deadline)
Prevention & Correction
- RCA timing: incident kickoff within 4 hours of P1/P2 resolution
- Mandatory fields: symptom, immediate action, root cause, evidence, preventive action, verification
- RCA database: searchable, taggable, centrally tracked
- RCA follow-up at 30 days: was preventive action deployed? Has issue recurred?
Success Metrics
- 100% of P1/P2s have RCA completed within 24 hours
- 0% of RCAs with vague root causes
- 100% of preventive actions have named owner + deadline + status tracking
- Preventive action completion rate: >90%
1. What actually "closes" an RCA under this standard?
Filing the document is step one, not the finish line — the fix has to actually ship and be confirmed working.
2. Why is "fix the symptom" (restart) dangerous rather than just incomplete?
A restart clears the symptom, not the cause — the same failure mode is still sitting there waiting to trigger again, often worse.
This is the section most people will recognize themselves in, because several habits below were considered perfectly acceptable in other environments — they're only wrong here, at this scale, on this system.
This is a fully preventable incident — seasonal spikes like Double Day (3x) or Tet (3-4x) are known weeks in advance; one Tet forecast of 4x baseline still broke the DB connection pool within 20 minutes because load testing was deferred until 3 days before the event.
What It Is
Platform experiences seasonal traffic spikes (Double Day 3×, Lunar New Year 3–4×). Teams know these events weeks in advance yet load testing is skipped or rushed. The spike arrives, system breaks, and it's called 'unexpected'.
Why It Matters
- Completely preventable incident
- Revenue loss during peak selling periods
- SLA breach — contracts specify uptime during promotional events
- Brand damage: "We couldn't handle demand" is worse than any technical failure
Real Scenario
Tết promotion Feb 1–10. Forecast: 4× baseline (40k QPS). Load test deferred until 3 days before event. Test runs at 1× load. Tết day: 4× actual traffic breaks DB connection pool within 20 minutes.
Observable Symptoms
- No load test infrastructure or capability
- No capacity forecast for known events
- Post-event reports mention "unexpected traffic spike" (it was known months ago)
- MTTR for traffic-spike incidents >1 hour (manual scaling, no auto-scaling)
Prevention & Correction
- Capacity forecasting (2 months before): identify event, estimate multiplier
- Load test design (4 weeks before): simulate 1.5× highest forecast, test all critical paths
- Auto-scaling validation (3 weeks before): dry-run scaling event in staging
- Capacity reserve (1 week before): pre-provision 20% extra
Success Metrics
- 0 P1/P2 incidents from traffic capacity during events where load testing was completed
- Load test discovers >80% of real bottlenecks pre-event
- Peak latency during event stays <2× baseline
- Auto-scaling works without manual intervention
High coverage usually only exercises the happy path. Timeouts, partial failures, and concurrent access still slip through — one gap caused a double refund because a "pending" state was never tested.
What It Is
Teams write tests that pass happy-path scenarios to hit a coverage metric (e.g., 80%), while skipping edge cases and error handling. Metrics look good; production bugs don't care.
Why It Matters
- Metrics are meaningless: 80% coverage is false security if critical paths aren't tested
- Bugs escape to production and are harder to diagnose
- MTTR extends because edge case handling is never debugged
Real Scenario
Refund service has 85% code coverage testing the happy path only. In production: the Get Refund Status API times out; the service doesn't receive confirmation; database state is 'pending' but money was transferred. Next day: customer requests refund again — double refund occurs.
Observable Symptoms
- Coverage metric high (85%+) but P1/P2 incident rates still high
- Production bugs in 'tested code' according to coverage reports
- Tests don't cover: timeouts, rate limit hits, partial failures, concurrent access
- Post-mortem: "This code path wasn't tested"
Prevention & Correction
- Test strategy, not just coverage: measure critical path coverage (100% of payment flows)
- Edge case checklist: null/empty input, large input, timeout, partial failure, concurrent access
- Property-based testing: generate 1000 random scenarios and verify invariants hold
- Test review: reviewer asks 'What scenarios fail this code? Is there a test for that?'
Success Metrics
- Critical path code coverage: 100%
- All error paths have tests (null handling, timeout, rate limit, partial failure)
- Post-incident audit: "Bug was in tested code" = never happens
- New bugs escaping to production: <1 per sprint
Early detection means a small blast radius (100 customers) versus late detection (1 million customers) — one deployment bug went uncaught for hours and reached thousands of customers with duplicate SMS messages before customer support flagged it.
What It Is
Code is deployed to production. Teams don't monitor immediately post-deployment. A subtle bug goes undetected for hours. By the time the issue is caught, thousands of customers have been affected and data is corrupted.
Why It Matters
- Early detection = small blast radius (100 customers) vs. late detection (1M customers)
- Cost of fixing grows exponentially: fix in 5 min = 100 customers; fix in 2 hours = data corruption
- SLA breach is almost certain if bug isn't caught in the first hour
Real Scenario
SMS engine deploys an 'optimisation' at 10 AM. The change triggers the gateway API in parallel with the DB write, but fires before the DB write commits. 3 PM: CS reports duplicate SMS messages to thousands of customers.
Observable Symptoms
- Incidents occur hours after deployment; no one was watching
- Post-deployment errors started 2+ hours ago; caught by customer complaint, not alert
- Post-mortem: "We didn't catch this until customers reported errors"
Prevention & Correction
- Post-deployment monitoring window (first 30 min): dedicated monitor, dashboard with error rate + latency + business metrics
- Canary deployment: 5% of traffic to new version; if error rate >2× baseline, auto-rollback
- Feature flags: toggle off instantly; <1 minute decision, <30 seconds to execute
- Post-deploy checklist: canary health, smoke tests, dependency health — don't open full traffic until all pass
Success Metrics
- Deployment bug detected within 30 minutes of deploy
- MTTR for deployment bugs: <15 minutes
- Blast radius for any production bug: <1% of users (<10k customers)
Trust in the whole suite erodes — so when a real failure happens, it gets dismissed as "flaky" too, and the bug ships to production.
What It Is
Tests fail randomly 10% of the time — not due to code changes but timing, shared state, or random seed issues. Developers re-run instead of investigating. Real failures are masked.
Why It Matters
- Automated testing is only valuable if it is trusted
- Bugs escape to production because tests were assumed flaky
- MTTR extends — test failure was ignored, root cause not in logs
Real Scenario
Payment API test: 'Verify refund processed in <500ms.' Test queries DB immediately after request. DB replication lag is 50–200ms randomly. Test fails ~30% of the time. One day refund actually does fail — the test failure is dismissed as 'that flaky replication test'.
Observable Symptoms
- Test suite has >5% flaky tests
- QC engineers re-run failed tests 3+ times after deployment
- Team sentiment: "I don't trust these tests"
- Bugs escape that the test suite should have caught
Prevention & Correction
- Zero flaky test tolerance: any test failing <95% of the time is immediately triaged
- Root cause categories: timing, shared state, random seed, external dependency, concurrency
- Test isolation: each test runs in isolation; setup/teardown is explicit
- Flaky test quarantine: if root cause isn't clear, track in issue tracker; fix within 1 week or remove
Success Metrics
- Flaky test ratio: 0%
- No test marked skip or flaky older than 1 week
- Developer survey: "I trust the test suite" >90% agreement
- CI/CD test pass rate: 99%
A sequential suite of 2,000 tests run daily means results land the following week — stretching major releases to 10+ working days and creating pressure to "fast-track" and skip testing altogether.
What It Is
Full regression test suite takes days to run sequentially. Only specific QA engineers can verify deployable builds. Deployment cycle stretches to a week.
Why It Matters
- Slows innovation; developers get blocked
- Manual testing is incomplete — QA can't test all scenarios
- Bottleneck creates pressure to skip tests ('fast-track this, it's urgent')
Real Scenario
Regression suite: 2,000 sequential tests. QA runs daily; results land the following week. If tests fail Friday, the fix isn't deployed until the following Tuesday. Major releases take 10+ working days from code-complete to production.
Observable Symptoms
- Full regression suite runtime takes days
- Deployment cycle: Monday commit → Friday deploy (5 days)
- QA engineer is a known bottleneck; team waiting for their time
- Bugs found in manual testing → back to dev → cycle repeats
Prevention & Correction
- Parallelize test execution: 2,000 tests / 100 parallel = hours reduced to minutes
- Eliminate manual gatekeeper: increase automation coverage; QA shares knowledge across team
- Regression testing SLA: defined maximum run time enforced by CI/CD pipeline
Success Metrics
- Full regression suite runtime: <4 hours
- Deployment cycle: Monday commit → Wednesday deploy (2 days, not 5)
- Flakiness doesn't cause re-runs
UI issues hit revenue directly — an off-screen confirm button once cost 8% of TPV for two weeks before anyone connected the data.
What It Is
Minor UI changes (button colour, alignment, wording) aren't treated as critical. They go unfixed. Customers notice. They compound — impacting key metrics (user retention, TPV) — and eventually become P1/P2 business incidents.
Why It Matters
- UI/UX issues directly impact revenue — users drop off and don't complete checkout
- Five minor UI issues = perception of a broken product
- Fix time is much longer when UI is broken in production for weeks
Real Scenario
Checkout flow redesign: larger submit button, moved address field. All P1 tests pass. But on 5-inch phones, the 'Confirm' button is partially off-screen. 12% of iOS users can't complete checkout. TPV drops 8% for two weeks before anyone connects the data.
Observable Symptoms
- Known UI issues tracked but not prioritised
- Analytics show: user drop-off at checkout increased 2% correlated with a UI breakage
- TPV declined post-deployment; UI changes broke a key workflow
Prevention & Correction
- Responsive testing: test on multiple device classes before every release
- P1-level UI criteria: any UI breakage in a customer-facing flow = P1 fix regardless of functional status
- Visual regression testing: automated screenshot comparison to catch unintended layout changes
Success Metrics
- 0 UI-related P1 incidents post-deployment
- Business metrics (conversion, retention) don't decline post-UI-change
- Average time to fix UI issues: <2 hours
A known bug can be exploited at production scale — a Risk API known to time out on ~2% of transactions was deployed anyway, the timeout rate hit 4% in production, and an abuser exploited it to bypass payment verification, causing a fund-loss incident.
What It Is
Teams approve production deployment with open Sev-2 or Sev-3 bugs. No formal risk assessment. No stakeholder sign-off (QA, Ops, Product). When known bugs interact with live traffic at scale, they explode.
Why It Matters
- Deployment risk is unmanaged; any known bug could cascade in production
- No shared understanding of acceptable risk
- When it fails, response is chaotic: "Who authorised this deployment?"
Real Scenario
Payment API deployed with 3 open bugs. Sev-2: Risk API occasionally times out (~2% of transactions). Deployed anyway. The Risk API timeout rate hits 4% in production — an abuser exploits it to bypass payment verification, causing a fund-loss incident.
Observable Symptoms
- Deployments have open bugs visible in Jira
- No approval gate for deployment (deploys without QA/DH sign-off)
- Post-mortem reveals: "That bug we knew about caused the P1"
- Severity ratings are inconsistent across squads
Prevention & Correction
- Severity definitions: Sev-1 (crash/data loss/security) = must fix before deploy; Sev-2 = risk assessment + DH sign-off
- Pre-deployment checklist: all Sev-1 fixed; Sev-2 risk-assessed and approved; tests pass; rollback ready
- Stakeholder approvals: QA sign-off, Squad Lead sign-off, DH sign-off
- Deployment pipeline blocks if checklist is incomplete
Success Metrics
- 0 P1 incidents caused by deployed bugs that were known pre-deployment
- 100% of deployments have completed pre-deployment checklist
- Known Sev-2 bugs have documented risk assessment + approvals
The same bug class (e.g. a race condition) resurfaces in a different service, because the lesson from one RCA never made it into the shared test suite.
What It Is
QA and engineering don't update test strategies based on production defects or cross-team post-mortems. The same bug class escapes in different forms. The test suite is static; production issues are dynamic.
Why It Matters
- Systemic learning doesn't happen
- Bugs caught once resurface in different components
- Test suite doesn't evolve with the platform
Real Scenario
Q1: Payment timeout causes double-charging (race condition). Tests updated for concurrent payments. Q2: Refund timeout causes double-refund — same race condition class. The Refund team hadn't seen Q1's RCA; their test suite had no concurrent-access scenarios.
Observable Symptoms
- Post-mortem: "Similar bug happened in component X last year"
- Test suite unchanged in months despite ongoing production defects
- QA doesn't attend cross-team post-mortems
- Defect categories repeat: race conditions, null handling, boundary cases
Prevention & Correction
- Test suite evolution: bi-weekly review of production defects and RCA post-mortems; identify test gaps
- Cross-team post-mortem attendance: QA attends all P1/P2 retrospectives
- Defect classification: maintain spreadsheet of defects by category; invest testing in high-escape categories
- Test strategy quarterly review: update strategy based on common defect types from prior quarter
Success Metrics
- Defect escape rate: <0.5 per sprint
- Recurring defect categories: 0 (same bug class repeats <1× per year)
- Test suite evolves every quarter
1. Why doesn't high test coverage guarantee safety at Zalopay?
Coverage measures lines executed, not scenarios that actually matter — edge cases and failure paths need to be tested on purpose.
2. What's the real cost of skipping load testing before a known peak event like Tet?
Knowing about a peak months in advance and not testing for it turns a predictable event into a self-inflicted incident.
Several of these habits look like "efficient work" elsewhere — fast reviews, bundled PRs, prioritizing speed — but they're the single biggest source of P1/P2 incidents here.
A payment service's MySQL-to-TiDB migration PR got a "Looks good, tests pass" — nobody asked about indexes for the real query patterns or the rollback procedure. Production: missing indexes caused table scans and a 6-hour P1.
What It Is
Code reviews focus on formatting and naming. Critical technical decisions — architecture, data handling, edge cases — are skipped. A 500-line DB migration PR is approved in 3 minutes. Shadow changes bypass review entirely.
Why It Matters
- Critical bugs slip through review
- Architectural debt accumulates silently
- Performance problems aren't caught until production
- Operational surprises: no one trained to handle new schema
Real Scenario
Payment service migrates from MySQL to TiDB. PR reviewed: 'Looks good. Tests pass. +1.' Nobody asks about indexes for real query patterns, connection pool size for peak load, or the rollback procedure. Production: missing indexes cause table scans; DB slows under load; P1 for 6 hours.
Observable Symptoms
- Code reviews are quick (<5 min per PR)
- Review comments are cosmetic: "fix typo", "reformat"
- No questions about design trade-offs
- PR descriptions are empty; reviewer has no context
Prevention & Correction
- Code review focus by PR type: DB/Schema (indexes, rollback, migration testing), API (backward compatibility), performance-critical (complexity, caching)
- Designated reviewers: critical areas (payment, data integrity, security) require senior engineer
- Review time budget: target 30–60 min for non-trivial PRs
- Review feedback quality: avoid cosmetic; focus on logic, design, correctness, testing
Success Metrics
- Code review time: >30 min per non-trivial PR
- Review feedback: >50% focused on logic/design, not cosmetics
- Production bugs in reviewed code that review should have caught: <1 per quarter
If it breaks, there's no full team to respond — a new refund flow deployed Friday at 4 PM broke silently at 6 PM, and on-call spent until 2 AM Sunday diagnosing and hotfixing it alone.
What It Is
Major code changes are deployed late Friday (or before holidays) without automated rollback or approval. If the deployment breaks, there is no full team available and the system stays broken over the weekend.
Why It Matters
- Worst timing for incident response (weekend; no full team)
- MTTR extends from minutes to 24–48+ hours
- Revenue loss, customer frustration, SLA breach
- Engineer burnout from solo weekend troubleshooting
Real Scenario
Friday 4 PM: new refund flow deployed. Tests pass. Developer leaves for the weekend. 6 PM: refunds not appearing. On-call tries to rollback — schema change already applied. Spent until Sunday 2 AM diagnosing and hotfixing.
Observable Symptoms
- Deployments happen Friday afternoon or before holidays
- Rollback requires database reversal (schema changes are one-way)
- Post-incident: "We deployed Friday with no rollback plan"
- On-call incidents on weekends correlate with Friday deploys
Prevention & Correction
- Deployment windows: no major deployments on Friday after 2 PM; no deployments before holidays
- Rollback automation: 1-click rollback; database migrations always reversible; feature flags allow feature-level rollback
- On-call readiness: rollback is automated + tested; on-call briefed; deployment ticket includes rollback procedure
- If Friday deploy necessary: rollback automated + tested; full team available until 9 PM
Success Metrics
- 0 Friday/pre-holiday major deployments without rollback automation
- Rollback time: <5 minutes (automated)
- On-call incidents on weekends caused by Friday deploys: 0 per quarter
When something breaks, it's impossible to tell which change caused it, and rollback becomes all-or-nothing — a 1,800-line refactor once got 50 payments stuck.
What It Is
Developer bundles a new feature, refactoring, bug fix, DB schema change, and config update into a single PR. If any piece breaks, the entire bundle fails — impossible to tell which change caused it.
Why It Matters
- Hides changes from scrutiny (reviewers can't deep-dive; too large)
- Rollback is all-or-nothing (can't revert just the bug fix)
- Root cause analysis is hard (which of 6 bundled changes caused the incident?)
Real Scenario
Payment core team ships 'Implement Charge v2' PR: 45 files, 1,800 lines. Reviewer approves. The refactor accidentally changed a boolean OR into an AND in the state check. 50 payments get stuck. Rollback reverts all 7 changes including an unrelated bug fix.
Observable Symptoms
- Large PRs (hundreds of lines, multiple files) are common
- PR description says "Includes X, Y, Z" (multiple unrelated features)
- Code review comments are shallow on complex PRs
- Post-mortem: "We didn't realise the refactoring was part of that PR"
Prevention & Correction
- PR size limits: target <200 lines, <5 files; if larger, require justification + 2 reviewers
- Logical separation: one PR = one logical change (one feature, one bug fix, or one refactor)
- Schema + code changes: never bundle schema migration with application logic in one PR
- Small PRs = safe rollback; large PRs = dangerous rollback
Success Metrics
- Average PR size: <200 lines, <5 files
- PRs >200 lines: <5% of total
- Individual changes can be rolled back without affecting unrelated features
- Production bugs from coupled changes: 0 per quarter
The debt silently kills the very speed it was meant to protect — one team's velocity dropped from 50 to 28 story points over 18 months because the same payment logic ended up duplicated across three services.
What It Is
Teams prioritise feature delivery speed at the expense of code quality. Shortcuts accumulate: incomplete error handling, untested edge cases, hacky configurations. Eventually velocity slows, MTTR rises, and incidents increase.
Why It Matters
- Velocity eventually tanks (can't move without refactoring first)
- Production incidents increase (fragile code breaks easily under load)
- New engineers take 3+ months to become productive
- MTTR increases (bugs hard to diagnose in convoluted code)
Real Scenario
Early sprint velocity: 50 points. Teams ship fast by duplicating payment logic across 3 services and skipping documentation. 18 months later: velocity is 28 points. Every feature now touches payment logic in 3 places, each slightly different.
Observable Symptoms
- Velocity increased short-term but plateaued or declined long-term
- Developers: "Can't add feature without refactoring 10 other things"
- Code duplication high; same change needed in 5+ places
- Technical debt acknowledged but never addressed: "We'll refactor later"
Prevention & Correction
- Structured debt management: allocate 20–30% of sprint capacity to tech debt payoff — not optional
- Debt tracking: technical debt in backlog with cost estimate and priority
- Debt prioritisation: >6 months old = schedule in next sprint; blocking features = fix first
- Quarterly refactoring sprint: dedicated sprint every quarter; no new features; measure reduced incident rate
Success Metrics
- Velocity stable or increasing (not declining over 2 years)
- No service duplication: no 2 services serving the same purpose
- Bugs per sprint: <1 (after refactoring)
- New engineer productivity: productive in <6 weeks
Only when it's actually needed does the team discover it doesn't work — one schema change with a dropped column turned an estimated 5-minute rollback into 4 hours of "fix forward" under production pressure.
What It Is
Major code changes are deployed without a tested rollback procedure. The rollback plan exists on paper but has never been executed. When deployment fails, rolling back is unclear, slow, or destructive.
Why It Matters
- MTTR extends dramatically when rollback is unavailable or untested
- Teams try to "fix forward" instead of reverting, extending the incident
- Incident severity increases with every minute the system is broken
Real Scenario
Payment API deploys a schema change + data migration. Estimated rollback: 5 minutes. Actual: the schema change is non-reversible (a column was dropped). Engineers spend 4 hours writing a forward-fix under production pressure.
Observable Symptoms
- Rollback documented in runbook but never practised
- Rollback involves manual steps ('SSH into 5 servers and run script X')
- Post-mortem: "Rollback didn't work; we had to fix forward"
- Deployments are infrequent (monthly) due to high perceived rollback risk
Prevention & Correction
- Automated rollback: feature flags (instant toggle); blue-green deployment (switch traffic back instantly)
- Database migrations always reversible: add-then-use-then-remove; never drop columns during rollout
- Rollback procedure SLA: <5 minutes from decision to complete revert; all steps automated
- Testing: rollback plan must be tested before the change is executed, not after
Success Metrics
- Rollback automation: 100% for all critical deployments
- Rollback time: <5 minutes
- Rollback success rate: 100%
- Deployment bugs caught via canary: 100% before full rollout
Skipping validation lets hidden dependencies slip through — a refund service designed to use an async queue was implemented with a synchronous HTTP call instead, and a slow partner response cascaded into a 30-second hang on the customer-facing API.
What It Is
Code is implemented without formal validation against the approved system design. Implementation drifts from design; dependencies aren't handled as planned; architectural decisions are missed.
Why It Matters
- Architectural debt accumulates unnoticed
- Cascading failures when undocumented integration points fail
- MTTR extends when debugging reveals missed dependencies
Real Scenario
Design: payment service uses an async queue to process refunds. Implementation: developer uses a synchronous HTTP call (simpler). In production, a slow refund partner causes request timeouts — cascading back to the customer-facing API, which hangs for 30 seconds.
Observable Symptoms
- Design documents exist but aren't referenced during implementation
- Implementation has dependencies not in the design
- Post-mortem: "Design called for async queue; code uses synchronous HTTP call"
- No one is responsible for validating implementation against design
Prevention & Correction
- Design review + approval: before coding, design reviewed by architecture team (data flow, failure modes, dependencies)
- Design changes during implementation require approval
- Implementation validation checklist (before PR merge): code matches design; all dependencies handled
- Architecture review: implementation reviewed against design before merge
Success Metrics
- 0 production incidents from unhandled dependencies that were in the approved design
- Architecture reviews completed for 100% of major features
- Post-incident: "Implementation didn't match design" = triggers mandatory process improvement
1. Why is bundling several changes into one PR riskier than opening separate ones?
Bundling trades short-term convenience for long-term traceability — exactly what you need most in an incident.
2. What's the specific danger of deploying a major change on Friday afternoon?
The code risk doesn't change on a Friday — the response capacity does, and that's what turns a minor bug into a weekend-long outage.
All three items share one trait: the consequences aren't linear. A "small" delay or shortcut can still end in a full breach or a legal violation.
Every day of delay raises the odds of exploitation — a CVSS 8.5 SQL injection deferred for 30 days ended with an attacker reading other customers' transaction data.
What It Is
A known security vulnerability with an exploitable code path is in production. SLA requires fixing within 24 hours. Team defers fix to the next sprint or applies a temporary workaround while the root cause remains exploitable.
Why It Matters
- Risk is quantifiable and time-bound — every day increases breach probability
- Regulatory compliance: if breach occurs while known vulnerability is unfixed, company is liable
- Financial loss: breach = fines, remediation costs, customer compensation, reputation damage
Real Scenario
Security report: SQL injection in customer dashboard (CVSS 8.5). SLA: fix within 24 hours. Day 1: received. Day 2: deferred ('sprint is full'). Day 5: WAF workaround. Day 30: breach — attacker reads other customers' transaction data. Regulatory notification required.
Observable Symptoms
- Known vulnerabilities in Jira older than 7 days (past SLA)
- Workarounds deployed instead of fixes
- Security team escalates; engineering team doesn't prioritise
- Post-incident: "That vulnerability was reported 30 days ago"
Prevention & Correction
- Vulnerability SLA: Critical = fix within 24 hours or take offline; High = fix within 2 days; Medium = 30 days; Low = 90 days
- Code fix is priority: eliminate root cause, not just workaround; workarounds are 24–48 hours max
- Automated weekly scanning + manual exploitability assessment
- Dependency updates: weekly check for security updates; deploy within SLA
Success Metrics
- 0 critical vulnerabilities in production older than 24 hours
- Vulnerability SLA compliance: 100%
- Time to fix known vulnerability: <7 days (tracked)
- Post-incident: never "vulnerability reported but not fixed"
Once a secret is committed, it's compromised — it stays recoverable in git history even after deletion. One exposed API key, from a repo briefly made public, caused a 50M VND loss.
What It Is
API keys, database passwords, encryption keys, and certificate private keys are committed to git. Even after deletion, they're recoverable via git log. Any developer with repo access can extract production credentials.
Why It Matters
- Complete unauthorised access to production systems
- No audit trail — attacker can impersonate legitimate users
- Immediate financial loss and data breach potential
- Regulatory liability
Real Scenario
Payment gateway API key hardcoded in a Java config class. Repo accidentally set to public for 6 hours. A former employee clones the repo, extracts the key, and uses it to process test transactions totalling 50M VND before the breach is detected.
Observable Symptoms
- Developers paste secrets directly into code as hardcoded strings
- Grep finds password:, api_key:, secret: in codebase
- Secrets visible in git history
- Former employees still have repo access and can retrieve secrets
Prevention & Correction
- Secrets never in code: use environment variables, secrets manager (CMDB), or config filled at build time
- History cleanup: if a secret was ever committed, it is compromised — remove from git history and rotate credential
- Access control: former employees removed immediately on departure; weekly access audit
- Automated scanning: weekly scan of entire codebase and git history; alert on any found; immediate action
Success Metrics
- 0 secrets in version control (code, config files, git history)
- All API keys, passwords, encryption keys in secrets manager
- Former employees removed from repo access: 100%, same-day
- Automated scanning: weekly, all codebase, 0 secrets found
This is a direct PCI-DSS/data-protection violation — a contractor once accidentally exported a file with 40,000 customers' card numbers from unredacted logs.
What It Is
Application logs contain PII: customer names, emails, phone numbers, card numbers. Logs are shipped to centralised log systems with broad access. Contractors and third-party tools can see raw customer data.
Why It Matters
- PCI-DSS violation: mandatory fines + termination of payment processor
- Violates data protection regulations
- Customer trust: if customers know their data was logged unmasked, trust is permanently eroded
Real Scenario
Payment API logs every transaction: 'Transaction from customer=Nguyen Van A, email=a@gmail.com, card=4111-1111-1111-1111.' Logs are shipped to OpenSearch. A contractor performing performance analysis inadvertently exports a file containing 40,000 customers' card numbers.
Observable Symptoms
- Logs contain customer names, emails, phone numbers, card numbers
- Contractors or third-party tools have access to centralised log system
- Post-breach: investigation reveals PII was in logs
Prevention & Correction
- Data classification: identify all PII fields; classify every schema field as Public/Internal/Sensitive/Secret
- Logging redaction: card number → last 4 digits only; email → hash or mask; phone → omit entirely
- Log retention policy: application logs auto-deleted after 30 days
- DLP (Data Loss Prevention): automated scan of logs before sending to central system; detect and redact PII
Success Metrics
- 0 PII in logs (automated scanning: 100% detection, 100% redaction)
- Access to logs: only authorised personnel
- Log retention: enforced (auto-delete after policy duration)
- Post-audit: 0 compliance violations related to PII logging
1. Why is a secret "temporarily" hardcoded for testing still compromised, even after it's deleted?
Deleting a file doesn't delete its history — anyone with repo access can still dig the secret out of an old commit.
2. What makes deferring a known critical vulnerability especially dangerous?
Risk here is time-bound and compounding — "we'll get to it next sprint" is a decision to accept a growing probability of breach.
Squad Lead
Why individual good will isn't enough without squad-level enforcement.
No owner, no deadline — that RCA is theater. A June 5 RCA concluded "fix max-connections setting," the Squad Lead acknowledged it, but no Jira tickets were created — the same pool exhaustion recurred on September 2 because the fix was never prioritized.
What It Is
Following a P1/P2 incident, the Squad Lead closes the RCA without creating discrete, owned, deadline-bound action items. Corrective measures are captured as vague notes ('we should improve monitoring') rather than concrete tasks.
Why It Matters
- RCAs without tracked actions are post-mortems in name only
- The same root cause reappears, often within 90 days
- Engineering culture degrades: engineers learn that writing an RCA is theatre, not accountability
- Leadership visibility into systemic risk is blocked
Real Scenario
June 5: DB connection pool exhaustion in Transfer API. RCA completed June 7. Conclusion: 'Fix max-connections setting.' Squad Lead reads and acknowledges. No Jira tickets created. September 2: the same pool exhaustion recurs — the fix was never prioritised.
Observable Symptoms
- RCA document closed but no corresponding Jira tickets exist
- Post-mortem note: "Team to discuss improvements" — no owner, no deadline
- Same incident root cause reappears in the next RCA
- When DH asks 'what's the status of the December incident fix?', Squad Lead doesn't know
Prevention & Correction
- RCA closure gate: RCA cannot be closed until action items are in Jira with owners and due dates
- Action item format: what changes (not what to investigate), who owns it, when it is done
- Backlog assignment: RCA action items go directly into current sprint as non-deferrable tasks
- Squad retro: every retrospective opens with a review of open RCA action items
Success Metrics
- 100% of P1/P2 RCAs produce Jira action items within 24 hours
- 0 open RCA action items older than 30 days without explicit documented escalation
- Incident recurrence rate for known root causes: 0 within 90 days
The same risk gets reopened before it's patched — a Visa EVN Fee hotfix deployed on January 13 was never merged to master, and the same bug reappeared on February 25.
What It Is
A P1/P2 incident occurs. Root cause identified. Fix planned. Before the fix is confirmed deployed, the system or a closely related service is pushed to production again — re-exposing the same risk.
Why It Matters
- Re-deployment before fix verification is re-exposing the same quantified risk
- Trust is destroyed: "they had a fix plan but deployed before fixing it"
- The next occurrence is worse — higher customer impact, harder to explain
- The RCA process is undermined: why write fix plans if nobody checks they are executed?
Real Scenario
Jan 13: Visa EVN Fee — hotfix deployed without merging to master. RCA: 'Merge to master before hotfix.' Feb 25: the same bug reintroduced because the process gate was never enforced before the next deployment.
Observable Symptoms
- Two incidents share the same root cause within 60–90 days
- RCA shows "fix was scheduled but not yet deployed" at time of second incident
- Sprint board shows RCA fix ticket as 'In Progress' while related service deployed
- Deployment approval checklist does not include an 'RCA fix status' field
Prevention & Correction
- Pre-deployment checklist: includes 'All open P1/P2 RCA actions for this service are deployed or formally risk-accepted'
- RCA action status visible in deployment pipeline: open action item requires Squad Lead sign-off
- Verification before RCA closure: action item marked done only when confirmed deployed to production and verified working
- DH review: Department Head reviews recurring incidents for this failure pattern quarterly
Success Metrics
- 0 incidents caused by a risk identified in a prior RCA and not yet fixed
- 100% of deployments to services with open RCA actions have documented risk acknowledgment
- Incident recurrence rate for same root causes: 0 within 90 days
Unreviewed "shadow changes" are the single biggest driver of P1/P2 incidents — an unpartitioned 400-million-row table was created outside the standard process, and tracing the root cause took hours.
What It Is
Engineers push changes to production outside the defined change management process: bypassing code review, skipping pre-deployment checklist, deploying directly from feature branch, or making config changes without a change record.
Why It Matters
- Change management is the primary defence against preventable incidents
- Unreviewed changes are the single largest driver of P1/P2 incidents
- 'Shadow changes' make root cause analysis nearly impossible
- It signals that process is optional — engineers treat everything as optional
Real Scenario
Feb 2026 Money Transfer incident: an unpartitioned 400M-row table caused DB connection exhaustion. The schema had drifted — the table was created without going through the standard DB change process. No change record existed. Root cause tracing took hours.
Observable Symptoms
- Production changes that don't appear in deployment records
- Config changes made outside any MR or change record
- Engineers deploy directly to production without code review approval
- Post-mortem: "We found a change that wasn't in the release log"
Prevention & Correction
- Zero-tolerance policy: any production change bypassing code review is a process violation — performance matter for Squad Lead
- Configuration changes: all config changes go through an MR, linked to a change record, reviewed before deploy
- Weekly deployment audit: diff of production deployments vs. change records; any gap escalated to Squad Lead immediately
- Process simplification: if engineers bypass because the process is too slow, fix the process — do not normalise bypassing
Success Metrics
- 0 production changes without a linked change record or MR
- 100% of critical-path deployments have code review + Squad Lead approval
- Post-incident: "We found an unrecorded change" = never happens
Bus factor drops to zero for that system — one incident took 40+ minutes to triage because only one person understood the routing rules, and they'd moved teams three months earlier.
What It Is
An engineer leaves or transitions. Their systems, runbooks, and operational knowledge are not formally transferred. The squad continues operating those systems with no documentation, no named owner, and no one who understands incident response procedures.
Why It Matters
- Bus factor drops to 0 for critical systems
- Incident response time multiplies when the 'person who knew' is gone
- Systems become dark technical debt; new engineers inherit with no context
- Mistakes compound: new engineers make the exact errors the previous engineer learned to avoid
Real Scenario
Aug 2025 K8S PCI incident: HAProxy configuration was only understood by one engineer. When the failure occurred, triage required 40+ minutes because nobody else understood the routing rules. The engineer had moved to a different squad 3 months earlier. No handover documentation existed.
Observable Symptoms
- RCA note: "We contacted the previous engineer (now in another team) to debug"
- Critical systems have no assigned owner in the service registry
- Runbooks reference engineers who have left the company
- New on-call engineers can't find documentation for components they are responsible for
Prevention & Correction
- Mandatory handover checklist: (1) system documentation updated, (2) runbook tested by backup engineer, (3) new owner assigned, (4) Squad Lead sign-off
- Knowledge distribution: Squad Lead ensures no system has only one person who can operate it — bus factor >1 is a squad health metric
- Documentation as a done criterion: features aren't 'done' without runbook, architecture context, on-call guide
- Quarterly ownership audit: Squad Lead reviews service registry for single-owner systems
Success Metrics
- 0 critical systems without documented runbooks and named owners
- Bus factor >1 for all services owned by the squad
- Engineer transition: 100% completed with handover checklist sign-off
- Post-incident: "We had to find the previous engineer" = never happens
Stale tests create false confidence — an AcquiringCore refactor introduced concurrent goroutine writes with no concurrency test written, and caused a fatal crash even at 78% code coverage.
What It Is
Engineers add features and change behaviour, but the automated test suite is not updated to reflect the changes. Tests become stale (pass but no longer test changed code paths) or flaky. Squad Lead doesn't track test health as a quality metric.
Why It Matters
- Stale tests provide false confidence — 'tests passed' but the changed code was not tested
- Bugs escape to production that the test suite should have caught
- Technical debt accumulates silently until a P1 incident exposes it
- CI/CD pipeline is meaningless as a quality gate if tests are stale or flaky
Real Scenario
Sep 2025 AcquiringCore race condition: concurrent goroutine writes caused a fatal crash. The squad had 78% code coverage and tests passed — but no test exercised concurrent goroutine access, introduced in a refactor with no concurrency test written.
Observable Symptoms
- Test coverage high (80%+) but incidents still frequent
- Engineers: "They always pass; I don't trust them"
- New features delivered with 'existing tests cover this' but no new tests written
- Post-mortem: "The test should have caught this but it didn't cover this code path"
Prevention & Correction
- Test updates as mandatory part of done: every MR that changes behaviour must include test updates
- Flaky test SLA: any flaky test is a P3 bug for the squad; must be resolved within 1 week or removed
- Coverage tracking by Squad Lead: Squad Lead reviews test coverage report at each sprint retro
- Post-incident test gap analysis: every P1/P2 includes 'what test would have caught this?' — that test is written before fix is closed
Success Metrics
- Test suite flakiness rate: 0%
- Post-incident: "The test didn't cover this" triggers immediate test addition — 100% of the time
- Production bug escape rate: <1 per sprint for issues tests should have caught
1. What makes an RCA action item real rather than "theater"?
An action item with no owner or deadline has no mechanism forcing it to actually happen — it just quietly ages in a document.
2. Why can verbal handovers fail so badly when an engineer leaves?
What isn't written down leaves with the person — the next incident has to be solved from scratch, under pressure.
Department Head
Where squad-level fixes become org-wide learning — or stay siloed.
Without a real retrospective, no one asks the system-level question — VietQR had 4 separate incidents in 9 months, each with its own squad-level RCA, before a cross-squad retrospective asked "Why does VietQR keep failing?"
What It Is
A P1/P2 incident is resolved. The DH either doesn't organise a retrospective, or facilitates one that is a formality — brief meeting, no structured agenda, no blameless framing, no concrete outputs.
Why It Matters
- Retrospectives are the only scalable mechanism for organisational learning from incidents
- Without structured retrospectives, the same incident class recurs across squads
- Engineers read absence of real retrospectives as 'leadership doesn't take this seriously'
- Systemic patterns, only visible across multiple incidents, are never surfaced
Real Scenario
VietQR had 4 separate incidents in 9 months, each with its own squad-level RCA. No DH-level retrospective was held to ask: 'Why does VietQR keep failing?' A cross-squad retrospective would have surfaced the systemic pattern: missing idempotency, no rate limiting, no self-degradation path.
Observable Symptoms
- Retrospective meetings scheduled but cancelled when engineers are 'too busy'
- Meetings last <30 minutes with no documented outcomes
- Action items from retrospective are vague or have no owner
- Same class of incident repeats and post-mortem written fresh with no reference to prior
Prevention & Correction
- Mandatory DH retrospective SLA: any P1 incident triggers DH-facilitated retrospective within 72 hours
- Structured agenda: (1) timeline reconstruction, (2) contributing factors, (3) systemic vs. local cause, (4) org-level changes, (5) action items with owners
- Blameless culture enforced: DH ensures retrospectives focus on system and process, not individual blame
- Cross-squad sharing: if root cause has applicability beyond the squad, DH ensures it's shared with all relevant squad leads within 1 week
Success Metrics
- 100% of P1 incidents have DH-facilitated retrospective within 72 hours
- 100% of retrospectives produce documented action items with named owners
- Retrospective action item completion rate: >90% within defined timelines
- Repeat incidents from the same systemic root cause: 0 within 180 days
A different squad repeats the same failure class months later, because the RCA's lesson never left one team — a config-schema bug that caused 9,009 failed payments stayed inside Cashier's own squad-level RCA, though the same root cause applies to any service with config-driven behavior.
What It Is
A P1/P2 incident contains a root cause that is systemic — could affect other squads. The DH resolves it within the squad but doesn't share the learning across the department. The same failure class then occurs in a different squad.
Why It Matters
- Each squad re-discovers the same mistakes independently — wasted incident cost
- Systemic patterns (race conditions, connection pool exhaustion) repeat across the platform
- Engineering knowledge is siloed; the organisation is weaker than the sum of its squads
- Trust erodes: "Why didn't anyone tell us?"
Real Scenario
Sep 2025 Cashier config-path incident: config schema changed without updating the loading code — 9,009 failed payments. The root cause (config schema and loading code must be atomic in the same MR) applies to every service with config-driven behaviour. The lesson stayed in Cashier's squad-level RCA; months later, a different service made the same class of mistake.
Observable Symptoms
- Two squads experience similar incidents within 6 months with no awareness of each other's RCA
- RCA database isn't searched before new post-mortems are written
- DH receives the RCA, acknowledges it, but doesn't forward to adjacent squads
- Cross-squad architecture reviews don't reference recent incidents from other squads
Prevention & Correction
- Incident communication standard: every P1/P2 RCA includes 'cross-squad applicability' field — if Yes, DH sends a brief to all applicable squad leads within 1 week
- Department incident digest: monthly brief of significant incidents + lessons learned, sent to all engineers
- Architecture review integration: significant architectural failures discussed at next department-wide architecture review
- RCA database is cross-searchable: before writing a new RCA, author must search for similar root causes
Success Metrics
- Cross-squad RCA sharing: 100% for incidents with cross-applicability
- Similar incidents in different squads within 180 days of a shared root cause: 0
- Qualitative: engineers report 'I learned about this from another squad's RCA'
A known risk keeps getting pushed sprint after sprint — the 400-million-row unpartitioned table behind the Feb 2026 Money Transfer incident had a fix plan, but every sprint's capacity went to feature delivery (Tet promotion, payment flow improvements) instead, and the DH kept approving those plans without asking about the known risk.
What It Is
A P1/P2 incident occurs. The RCA produces action items. But the next sprint fills entirely with feature work. Reliability fixes are deferred repeatedly. The DH approves the sprint plan without questioning zero reliability allocation.
Why It Matters
- Deferred reliability work is a quantified liability — it will cause another incident
- Engineering culture degrades: "fixing root cause matters less than shipping the feature"
- The reliability plan becomes theatre: published but never funded with real sprint capacity
- DH is implicitly authorising the risk when approving a plan with zero reliability capacity
Real Scenario
Feb 2026 Money Transfer incident: 400M-row unpartitioned table. A fix plan existed, but every sprint's capacity was consumed by feature delivery (Tết promotion, payment flow improvements). No capacity was ever allocated to the partitioning task, and the DH approved sprint after sprint without asking about the known risk.
Observable Symptoms
- Reliability fix tickets age 30–60+ days in backlog without being worked
- Sprint planning consistently shows 100% velocity allocated to feature delivery
- DH approves sprint plans without asking 'Where are the RCA action items from last month?'
- Engineers: "We want to fix this but product keeps pushing it back"
Prevention & Correction
- Reliability capacity floor: DH mandates minimum 20% of sprint velocity for tech reliability, debt, and RCA actions — not negotiable
- DH sprint review gate: before approving any sprint, DH checks 'Where are open RCA action items? Are they in this sprint or formally risk-accepted?'
- Risk acceptance is explicit: if known risk deferred past 30 days, requires written risk acceptance signed by DH — not silent deferral
- Reliability roadmap: DH maintains visible roadmap of deferred items with risk level and timeline — presented to leadership monthly
Success Metrics
- Sprint plans: minimum 20% of velocity allocated to reliability/tech debt every sprint
- RCA action items: 0 items older than 30 days without being in-sprint or having formal risk acceptance
- Incidents caused by deferred known risks: 0
A squad with a lighter process can be "compliant" with its own weaker standard — the Jan 2026 Merchant Platform incident involved a config change deployed without a change record or review, which wasn't required by that squad's local process even though another squad's process would have required DH sign-off.
What It Is
Each squad defines its own change management process independently — different approval gates, different pre-deployment checklists, different code review requirements. The DH hasn't established a department-wide baseline.
Why It Matters
- Risk control inconsistency creates unpredictable incident patterns across the department
- Squads have no shared reference for 'what does good look like?'
- Cross-squad work falls into gaps between squad-level processes
- Incidents cluster in squads with lighter-weight processes, but DH never investigates why
Real Scenario
Jan 2026 Merchant Platform incident: a configuration change was deployed without a change record or review — not required by that squad's local process. The same change in another squad would have required DH sign-off. The lack of a department standard meant the Squad Lead's lighter-weight process was technically 'compliant' but insufficient.
Observable Symptoms
- Code review requirements differ between squads (1 approver in Squad A, 2 in Squad B)
- Pre-deployment checklists are squad-created, not department-standardized
- Some squads deploy without post-deployment monitoring windows; others have strict 30-minute canary
- DH has no visibility into what each squad's change management process contains
Prevention & Correction
- Department-wide change management baseline: DH publishes and enforces minimum standard covering code review requirements, pre-deployment checklist, post-deployment monitoring window, rollback requirement
- Squads can add requirements above the baseline but cannot go below it
- High-risk change categories defined at department level: DB schema changes, payment-path code, security changes all require DH-level visibility
- Annual process audit: DH reviews each squad's change management process against the baseline and publishes findings
Success Metrics
- 100% of squads have change management process reviewed against department baseline annually
- High-risk deployment categories: 100% have DH-level awareness before deployment
- Incident root cause: 0 incidents where 'lighter squad process' is a contributing factor
1. What's the risk of a Department Head approving a sprint plan with 0% reliability capacity?
Approval isn't neutral here — signing off on a sprint with no reliability capacity is a decision, even when it's made by omission.
2. Why does a lesson learned in one squad's RCA still need to be spread department-wide?
Knowledge siloed in one squad protects only that squad — the rest of the department re-discovers the same mistake the hard way.
This standard isn't only about avoiding violations — these four areas are where a squad gets recognized for going beyond compliance, not just avoiding mistakes: