9798235433472_

The systems making decisions about your life were never elected. No one asked if you approved. And most of them cannot be stopped.

Between 2016 and 2019, the Australian government sent 526,000 welfare recipients automated debt notices using a calculation method its own lawyers had identified as unlawful — before the programme launched. The legal advice was not shared with ministers. The programme ran for four years after a court found it illegal. People paid debts that did not exist. Two mothers gave evidence to the subsequent Royal Commission about sons who had not survived the process.

In New Orleans, a predictive policing programme operated in secret for six years, generating risk scores on 3,900 named individuals, informing federal prosecutions, and remaining entirely unknown to the city council member who had chaired the criminal justice oversight committee for the entirety of that period. She learned about it from a journalist.

At Facebook, internal documents showed that the company's public claim of removing 94% of hate speech reflected a carefully chosen metric. The actual removal rate, per the company's own researchers: 3 to 5 percent. A whistleblower brought the evidence to the United States Senate. The Senate listened, thanked her, and went home.

Invisible Governors argues that these are not isolated failures of technology, ethics, or political will. They are the predictable consequences of a single structural problem: democratic institutions have deployed automated systems that move faster than their oversight mechanisms, without building the feedback loops that would allow those systems to be seen, contested, corrected, and stopped.

There are five such loops. Each has a name, a documented history in other high-stakes governance domains — central banking, pharmaceutical approval, aviation safety, nuclear deterrence — and a clear institutional architecture. None of them are present in most automated governance systems currently operating.

This book names them. It traces their absence through the most thoroughly documented cases of automated governance failure of the past decade. And it makes the case — without false optimism, but with the weight of historical precedent — that closing them is not a technical challenge but a political one, and that the politics, while difficult, are not unprecedented.

Democracies have built institutional architecture for dangerous new technologies before: after the railways, after thalidomide, after the financial crisis. The governance architecture for automated systems is not a mystery. It has simply, consistently, not been built.

The loops can be closed. The question is who closes them first, and whether they understand what they are building.

Invisible Governors: How Automated Systems Took Power — and How We Take It Back

Comentarios