Case study

Twelve petabytes in six hours

A live-service title with 4.1 million concurrent-ish players shipped its largest content update. Here is what the delivery layer did.

The update touched 41% of the installed build — new zones, a shader recompile and a full audio repack. Naively that is a 58 GB download per player, or 238 petabytes in aggregate. Nobody has that budget.

Chunking took it to 3.1 GB

Content-defined chunking found that most of the audio repack was byte-identical to the previous build, merely relocated inside the package. Fixed-block diffing would have missed all of it. The published patch was 3.1 GB — a 95% reduction against the naive figure.

chunks total          1,942,884
chunks new                87,204   (4.5%)
chunks relocated         612,441   (31.5%)
chunks unchanged       1,243,239   (64.0%)
published patch size        3.1 GB

Peers took two-thirds of the rest

Peer assistance came online as soon as the first few thousand clients had complete chunk sets. Within eleven minutes the peer share crossed 50%; at peak it was 71%. Total delivered was 12.8 PB, of which 4.1 PB came from our CDN.

The interesting engineering problem is not the peak. It is the first eleven minutes, when there are no peers and everyone arrives at once.

We handle that with pre-seeding: the manifest and the chunk set are pushed to every edge before the announcement, and the launcher is allowed to pre-download on a schedule the studio sets. Sixty-one percent of players had the patch on disk before it unlocked.

What went wrong

Two things. A carrier in one country blocked the peer protocol's UDP port range, so 340,000 players fell back to CDN-only and saw slower downloads — we now probe and fall back to TCP within eight seconds instead of ninety. And the verification pass on spinning disks took up to fourteen minutes, which surprised players who thought the download had stalled; the launcher now shows verification as a separate stage with its own progress.