Bristol AI failure: police shut down child crime prediction models due to catastrophic accuracy
The Avon and Somerset Police, in collaboration with the Bristol City Council, was forced to abandon the use of at least two artificial intelligence models designed to assess the risk of crimes against children. The reason: low accuracy and complete opacity of the algorithms. Independent auditors were unable to find either the source code or even a list of the variables used, making any verification of the systems' functionality impossible.
At the heart of the scandal lies the Think Family Database, launched by the Bristol City Council in 2016. It accumulated information on nearly 500,000 residents, including police data, social services data, information on mental health, housing status, school attendance, and even participation in parenting courses. Data collection was carried out without the direct consent of citizens, based on legal provisions for information sharing between government agencies.
How the models worked and why they failed
Based on this database, 23 machine learning models were built, including those predicting theft, the risk of domestic violence, and, most critically, crimes against children. One of the models for assessing risk concerning children used anonymized data from the charity Barnardo's on 1,000 already victimized minors. Risk factors considered included the child's status as needing help, school truancy, and mental health problems.
As early as 2016, the police ethics committee warned of a high risk of algorithmic bias due to the chosen variables. Later, an audit conducted by the non-profit consulting organization Social Finance confirmed the worst fears: the accuracy of the models was deemed the weakest link, completely undermining their practical value. Social Finance linked the degradation in quality to a change in the dataset. When attempting to scale the models to five local councils, the police could not agree on sharing social data, and as a result, the models relied only on the police "core," losing critically important social indicators.
Eticas Audit: Accuracy Below 10%
A separate analysis conducted by the auditing firm Eticas, based on 36,000 performance evaluations, revealed that most models had extremely low positive predictive accuracy. For example, a model designed to identify potential burglars showed accuracy below 10% for over three years — meaning fewer than one in ten people flagged by the system actually committed a crime. The auditors emphasized that such metrics are absolutely atypical for professionally managed models in operational use.
City service employees complained that the system missed vulnerable children, while individuals in theft cases could receive higher risk scores. Other workers openly stated they were unwilling to rely on the assessments due to the complete opacity of the methodology.
My analysis: This case is not just a story about an AI failure. It is a stark example of how ambitions to implement technology in sensitive areas, such as child protection, can be undermined by neglecting fundamental data science principles: data quality, reproducibility, and transparency. The absence of source code and decision-making documentation is not a technical oversight but a systemic management failure that calls into question any future PoliceAI initiatives.