Crypto news

The Avon and Somerset Police, in collaboration with the Bristol City Council, have discontinued the use of at least two artificial intelligence models designed to assess the risk of crimes against children. The reason is critically low prediction accuracy and complete opacity of the algorithms. Independent auditors encountered a situation where the source code and the list of variables used were simply absent, making verification of the results impossible.

My analysis shows that this situation is not an isolated failure, but a systemic problem with the implementation of AI in law enforcement. When models become a "black box," they not only lose practical value but also carry reputational and legal risks.

Think Family Database: Data Collection Without Consent

The failed models were based on the Think Family Database, launched in 2016. It accumulated information on nearly 500,000 Bristol residents, consolidating police reports, social services, housing data, mental health records, teenage pregnancies, school truancy, and even free school meal eligibility. The key point: collection occurred without direct citizen consent, based on legal norms for data sharing between government agencies. One police data specialist cynically described this process as "throwing everything into a big bucket."

At least 23 different machine learning models were built on this database, ranging from predicting burglaries to assessing the risk of becoming a victim of domestic violence. Concurrently, the Offender Management App was operational, which, according to a senior officer, served as the basis for a "leaderboard" of the most dangerous criminals.

Why the Models Failed

One of the early models for assessing the risk of crimes against children used data from the police, the city council, and even anonymized information from the charity Barnardo's on 1,000 children who had already been victims of such crimes. Factors influencing the scoring included a child's status as in need of help, persistent school truancy, and mental health issues.

As early as 2016, the police ethics committee warned of a high risk of algorithmic bias. Later, an audit conducted by the non-profit organization Social Finance confirmed the worst fears: the risk scoring was deemed the weakest element of the system. Low accuracy completely undermined the practical value of the models. By the time of the review, two of them had already been deactivated.

The root of the problem lies in data degradation. When attempting to scale the approach to the entire Avon and Somerset region, the police failed to agree on data sharing with all local councils. As a result, the models lost crucial social indicators, leaving only a "police core" that did not provide an adequate picture. City service employees directly complained that vulnerable children were not captured in the results, and underage crime victims received lower scores than individuals involved in theft cases.

Eticas Audit: Accuracy Below 10%

Journalists obtained over 36,000 performance evaluations for 13 models spanning 2017-2024. An analysis conducted by the auditing firm Eticas revealed a catastrophic picture: most models had low positive predictive accuracy. The system erroneously flagged a significant proportion of people as risks.

The model for identifying potential burglars was particularly telling. For over three years, its positive predictive accuracy remained below 10%. This means that fewer than one in ten individuals flagged by the AI as a threat actually committed such a crime. The auditors emphasized that such metrics are absolutely atypical for well-managed models in operational use.

The police attempted to justify themselves, stating that some models, including the burglary tool, were never deployed, and that the multi-year evaluations were the result of automatic checks on a static file that had not been deleted. The Bristol City Council acknowledged that it currently uses only one NEET risk model (assessing the likelihood that a child will not be in education, employment, or training after school) but insists it does not replace professional judgment.

Context: PoliceAI and Systemic Risks

This scandal unfolds against the backdrop of the launch of the national PoliceAI center, which has been allocated £75 million over three years to scale AI across all 43 police forces in England and Wales. An ironic twist: the center is headed by former Avon and Somerset Police Chief Constable Andy Marsh—the very region where the risk model failure occurred.

Expert Opinion: The Bristol case is not just a story about bad code. It is a clear demonstration that deploying AI in critical areas, such as child protection, requires not only perfect algorithms but also impeccable data management, absolute methodological transparency, and mandatory independent auditing. Without these three pillars, any model, no matter how promising, risks becoming a dangerous and useless tool that undermines trust in justice.