Failure of AI Predictors in Bristol: Child Risk Assessment Algorithms Disabled Due to Fatal Errors
The Avon and Somerset Police, together with Bristol City Council, have discontinued the use of at least two artificial intelligence models designed to assess the risk of crimes against children. The reason is critically low prediction accuracy and complete opacity of the systems, which proved virtually impossible to independently audit.
A journalistic investigation conducted with the participation of the human rights group Liberty Investigates, the local publication Bristol Cable, and the non-profit editorial team Lighthouse Reports, revealed systemic problems in the operation of these algorithms. An analysis of hundreds of pages of internal documents showed that the models, built on the basis of the Think Family Database, suffered from fundamental flaws.
How data was collected and models were built
The Think Family Database, launched in 2016, combined police and social data on Bristol residents. It included information on housing status, mental health, teenage pregnancies, school attendance, and even receipt of free school meals. Notably, data was collected without the direct consent of citizens, using legal grounds for inter-agency information sharing. One police data specialist cynically described this process as "throwing everything into a big bucket."
On this shaky foundation, 23 machine learning models were built, assigning risk scores to adults and children—from predicting burglaries to the likelihood of becoming a victim of domestic violence. In parallel, the Offender Management App operated, which, according to one senior officer, served as the basis for a "leaderboard" of the most dangerous criminals.
Why the models failed
The key problem lay in the quality of the data. One model for assessing the risk of crimes against children used anonymized data from the charity Barnardo's on 1,000 children who had already suffered from such crimes. However, in 2016, the police ethics committee warned of inevitable algorithmic bias due to the selected variables, such as a child's status as needing help or mental health issues.
Later, an audit conducted by the non-profit organization Social Finance confirmed the worst fears. The risk scoring was deemed the weakest element, and low accuracy was cited as a factor undermining the practical value of the models. By the time of the review, two models for assessing the risk of crimes against children had already been deactivated.
Social Finance linked the deterioration in model quality to a change in the dataset. The police attempted to scale the approach across the entire Avon and Somerset region but failed to reach data-sharing agreements with all local councils. As a result, the models lost social indicators and turned into a purely police "core," further reducing their accuracy.
Of particular concern is the complete opacity of the systems. Independent auditors could not find either the source code or the list of variables used in the models. Moreover, neither the police nor Bristol City Council had, by June 2023, retained documents regarding the decision to abandon the two models for assessing the risk of crimes against children.
Results of the independent audit
Journalists from WIRED obtained from the police over 36,000 performance evaluations for 13 models used or tested between 2017 and 2024. An audit conducted by the company Eticas showed that most models had critically low positive predictive accuracy. The system erroneously flagged a significant proportion of people as high-risk.
For example, a model for identifying potential burglars showed a positive predictive accuracy below 10% for over three years. This means that fewer than one in ten people flagged by the system actually committed such a crime. The auditors emphasized that such metrics are atypical for well-managed models in operational use.
A case against the backdrop of PoliceAI expansion
This story unfolds against the launch of PoliceAI—a national center for testing and scaling AI tools across 43 police forces in England and Wales, with a budget of £75 million over three years. Notably, the center is headed by former Avon and Somerset Police Chief Constable Andy Marsh—the very region where the controversial AI analytics were developed.
The Bristol case is not just a story about a technical error. It is a systemic failure that demonstrates that the risks of such models are linked not only to the accuracy of algorithms but also to data quality, documentation retention, and the possibility of independent verification.
Expert opinion: This situation is a classic example of how the pursuit of technological innovation in law enforcement can turn into a disaster without proper oversight of data quality and algorithm transparency. AI models trained on biased or incomplete data are not just useless—they are dangerous, as they can create false accusations and undermine trust in the justice system. The Bristol failure should serve as a warning to all who blindly implement AI in socially significant areas.