Crypto news

The Avon and Somerset Police, together with the Bristol City Council, have discontinued the use of at least two artificial intelligence models designed to assess the risk of crimes against children. The reason is critically low accuracy and complete opacity of the algorithms. Independent auditors were unable to find either the source code or even a full list of variables used in the systems.

During a journalistic investigation based on hundreds of pages of documents obtained through freedom of information requests, systemic problems were uncovered. This story unfolds against the backdrop of the launch of the national PoliceAI center, intended to test and scale AI tools across 43 police forces in England and Wales.

Think Family Database: A "Big Bucket" Without Consent

At the heart of the problem lies the Think Family Database, a database launched in 2016. It combined police and social data on families, including housing status, mental health information, teenage pregnancies, school truancy, and even the receipt of free school meals. It is estimated that the database could have contained records on nearly 500,000 residents, with data collected without people's direct consent, based on general legal provisions for information sharing between government agencies. One police data specialist cynically described this approach as "throwing everything into a big bucket."

Based on this database, at least 23 machine learning models were built — ranging from predicting burglaries to assessing the risk of becoming a victim of domestic violence. However, it was the models for assessing the risk of crimes against children that performed the worst.

Failure on All Fronts: From Bias to Vanishing Code

As early as 2016, the police ethics committee warned of a high risk of algorithmic bias due to the chosen variables. The model, for example, included a child's status as being in need of help and mental health issues, which could create a vicious cycle of stigmatization. Later, the non-profit consulting organization Social Finance, in its review, called the risk scoring the weakest element of the project, noting that low accuracy completely undermined the practical value of the models.

An attempt to scale the system across the entire Avon and Somerset area only worsened the situation. Due to the inability to agree on data sharing with all local councils, social indicators disappeared from the models, leaving only the police "core." As a result, social services staff complained that vulnerable children who were victims of crimes received a lower risk score than individuals involved in theft cases. An audit conducted by the company Eticas showed that for most models, the true positive rate was below 10%. This means the system incorrectly labeled more than nine out of ten people as "high-risk." By the time of the Social Finance review, the source code and documentation for the models had already been lost.

Conclusions and Prospects

The Bristol case is not just a story about an algorithm failure. It is a demonstration of the fundamental risks associated with deploying AI in critically important social spheres. The problem lies not only in accuracy but also in transparency, data quality, and the possibility of independent audit. It is telling that the head of the new national PoliceAI center, a former chief constable of Avon and Somerset Police, previously oversaw the very jurisdiction where these controversial models were developed and used.

Cryptalist Commentary: This case is a stark reminder that the "black box" of algorithms in the hands of the state can be more dangerous than the crime it is supposed to prevent. While regulators and the public debate the future of AI, police departments around the world are already conducting dangerous experiments with human lives. The Bristol story should serve not just as a warning, but as a catalyst for introducing strict standards of audit and transparency for all government AI systems.