Q3 Infrastructure Review.pdfPage 2 of 14 Search
Platform Engineering · Q3 Infrastructure ReviewConfidential — internal distribution

2. Reliability and capacity

Across the quarter the production fleet served 7.4 billion API requests, up 18 percent from Q2, while the monthly availability of the public API stayed at or above the 99.95 percent objective in all three months. The single largest contributor to the error budget was the payments upstream incident on 14 August, which consumed 41 percent of the quarterly budget in under two hours. Excluding that event, the remaining burn was spread across routine deploys and two short regional network events.

Capacity planning moved from quarterly spreadsheets to the forecasting job introduced in July. The job projects request volume per service from the previous twelve weeks and flags any service whose projected p95 utilisation exceeds 70 percent within the next 30 days. In its first ten weeks it raised eleven warnings, nine of which led to scaling changes before the threshold was reached; the other two were seasonal spikes that the model now accounts for.

2.1 Availability by service

ServiceRequests (M)Availabilityp95 (ms)IncidentsBudget used
api-gateway7,41299.982%142136%
checkout-svc61299.961%318278%
payments-svc20199.912%4663176%
search-svc2,90499.990%211020%
catalog-svc3,37799.994%88012%
auth-svc1,14599.987%64126%

Table 2. Quarterly availability and latency per tier-one service. Budget used is relative to each service's own objective.

Payments remains the only tier-one service that exceeded its error budget. The follow-up work agreed in the August review — request hedging towards the secondary provider, a shorter client timeout, and an explicit circuit breaker in the checkout path — shipped in September. Since then the service has had no incident longer than four minutes, and its p95 latency has dropped from 466 to 301 milliseconds over the last three weeks of the quarter.

2.2 Cost

Compute spend grew by 6 percent against an 18 percent increase in traffic, which brings the cost per million requests down to $3.12 from $3.47 in Q2. Most of the saving came from moving the search and catalog tiers onto the newer instance family and from right-sizing forty-two over-provisioned deployments identified by the utilisation report. Storage costs were flat; the log retention change planned for Q4 is expected to reduce them by roughly a fifth.

For the next quarter we propose to keep the availability objectives unchanged, to add latency objectives for the three checkout endpoints, and to extend the forecasting job to databases and queues, which are currently planned by hand.