Pieza independiente
us-east-1 · Northern Virginia
The region
I
On 28 February 2017, at 9:37 in the morning Pacific time, an Amazon technician ran a command to remove some billing servers from the storage service.
The procedure was routine and established. A parameter was mistyped in the command, and instead of removing a few servers many more were removed, among them those of two subsystems that were not meant to be touched: the one that keeps the indexes of where each object is, and the one that decides where to place new ones.
The two subsystems had to be restarted entire. They had not been fully restarted for years, the volume of data had grown a great deal since then and nobody knew how long the startup would take. It took hours.
During that while, project management, messaging, home automation, photograph storage and accommodation booking services stopped working. The dashboard Amazon publishes to report incidents also stopped updating, because its status icons were stored in the service that had gone down. For a while, the only way of finding out that the service was down was that it did not work.
II
The region is called us-east-1 and it is in northern Virginia, spread across several data centres in Loudoun County and thereabouts.
It is the first one Amazon opened, in 2006, and from that comes almost everything else: the inertia accumulated since then has mass. It is the cheapest, it is where new services are launched first, it is the one that comes up by default in the tools and in the documentation examples, and it is where several global services live that have no copy anywhere else.
That last sentence is the important one. A region is supposed to be an independent unit, and if one goes down you use another. But some central functions —certain identity registries, certain name directories— live physically there and nowhere else, so that a customer with all their infrastructure in Europe can find themselves unable to authenticate requests because a building in Virginia has failed.
Loudoun County has for years advertised itself as the place through which a very large part of the world's internet traffic passes. The figure the county itself uses in its promotional material is that around seventy per cent passes through there. It is a brochure figure and it is as well to take it as such. What is documented is the concentration of square metres of data centre, which is the largest in the world.
III
It happened again on 7 December 2021, by another route.
That time it was an automated scaling activity that provoked a storm of connections on the region's internal network. The internal network became congested, and the monitoring systems [which use that same network] stopped seeing what was happening.
For several hours the engineers worked blind, with no telemetry. The status dashboard could not be updated either, for the same reason as four years before. Delivery, video, domestic vacuum cleaner and camera doorbell services stopped working; in several of Amazon's own warehouses operations were interrupted.
There is a pattern in the two outages that is not accidental. In both cases the system that was supposed to report the failure depended on the system that had failed, and in both cases the recovery was slow not for lack of diagnosis but because cold-restarting something that has been switched on for years is an operation nobody has rehearsed.
It is as well besides to give the size of the matter, because without it this looks like an anecdote about one provider. Three companies share around two thirds of the world market in cloud infrastructure, and the first of them has in Virginia its oldest and most used region. It is not that there is a single point of failure; it is that there are three, and one of them is considerably bigger than the other two.
IV
The interesting thing is not that a provider goes down. Providers go down; they go down all over the world and they will go on going down.
The interesting thing is that nobody had the list of who depended on what.
When a company contracts a cloud service, it knows what it has contracted. What it does not know is what its payment gateway provider has contracted, nor where the message-sending service used by its identity provider is hosted, nor which region holds the repository from which its own deployment downloads a dependency at the moment of starting up.
That chain is not declared anywhere and there is no obligation to declare it. It is discovered the day it breaks, by looking at what stopped working, and the inventory left afterwards is the only one that exists until the next time.
On the eighth of June 2021 there was an especially clean demonstration of this. A customer of the company that accelerates content delivery changed one line of its configuration; the change woke a fault that had lain latent in the program for a month; for some forty-nine minutes, a considerable part of the web returned an error — several newspapers, a payments platform, the British government portal. None of them had contracted anything from that company that day. They had contracted it years earlier and had forgotten.
There is a simple proof that the chain is not known, and it is that nobody passes it. Ask any company which of its services would stop working if a particular region of a particular provider disappeared for six hours. The honest answer, in almost every case, is that it would have to be found out by switching it off.
There is a name for that exercise when it is done on purpose [chaos engineering], and there are companies that practise it. They are few.
V
Baran measured the survival of a network assuming that nodes go down one at a time and separately.
That assumption is what holds up his whole result. If the nodes go down independently, with three or four links each the network withstands the loss of half. If they go down together, the calculation says nothing, because what looked like a hundred nodes was one.
The industry's word for that is correlated dependency, and it is exactly what a cloud region manufactures: thousands of services that would be drawn as separate points on any architecture diagram, and that share a building, an internal network and a control plane.
The right-hand drawing is still correct. What has changed is what counts as a node.
A microservices architecture drawn on a whiteboard looks like the distributed panel: many small pieces, each talking to several. Drawn on the map of Loudoun County it looks like the left-hand panel, and the whiteboard is not lying — it is measuring something else.