Bristol AI scoring of children: errors, bias, and closed models
The Avon and Somerset Police, together with the Bristol City Council, have discontinued the use of at least two machine learning models designed to assess the risk of crimes against children. The reason is critically low accuracy and complete opacity of the algorithms. Independent auditors faced the impossibility of verification: the source code and list of variables were lost.
These systems were based on the Think Family Database, launched in 2016. It aggregated police and social data on nearly 500,000 city residents — from housing status and mental health to school truancy and receipt of free meals. Information was collected without explicit consent from citizens, based on legal norms of interagency data exchange. One police officer described this process as "dumping everything into one big bucket."
Systemic Errors and Algorithmic Bias
As early as 2016, the police ethics committee warned that the selected variables could lead to algorithmic bias. However, the warnings were ignored. Later, the non-profit consulting organization Social Finance, in its review, called the risk scoring the weakest link of the project. Analysts found that vulnerable children who had recently been victims of crimes could receive a lower score than individuals involved in burglary cases. City service employees openly stated they did not trust the system due to its opaque methodology.
A separate audit conducted by the company Eticas, based on 36,000 performance evaluations of 13 models, showed that for most of them, the precision of positive predictions was below 10%. This means that over 90% of people flagged by the system as potential offenders had not actually committed any crimes. For the model identifying potential burglars, this indicator remained below 10% for more than three years.
Failure Amidst the Scaling of PoliceAI
This incident takes on particular significance against the backdrop of the launch of PoliceAI — a national center for testing and scaling AI tools across 43 police forces in England and Wales, with a budget of £75 million. Notably, this center is headed by former Chief Constable of Avon and Somerset Police, Andy Marsh — the very region where the controversial AI models were developed and subsequently rejected.
The Bristol case is not just a story about a flawed algorithm. It is a systemic failure demonstrating that the risks of implementing AI in law enforcement are linked not only to the accuracy of the models themselves but also to the quality of the underlying data, the lack of transparency, and the impossibility of independent auditing. By June 2023, the Bristol Police and City Council had not even retained documents regarding the decision to abandon the two models for assessing the risk of crimes against children. This calls into question the ability of these agencies to manage such technologies and be held accountable for them.
Expert Opinion: The story of the Bristol AI models is a classic example of how the pursuit of technological modernization without proper control over data quality and algorithmic transparency leads to the discrediting of the very idea of using AI in socially significant areas. Until police departments learn to maintain basic documentation and ensure the possibility of external auditing, any talk of "safe" and "ethical" AI will remain empty declarations.