Google was forced to urgently deactivate its new AI image generator, integrated into Google Earth and based on the Nano Banana model. The tool lasted less than 24 hours — such a rapid rollback indicates serious flaws in content moderation.
The essence of the feature was to draw arbitrary objects or events onto any point on the map based on a user's text prompt. However, within the first hours after launch, fake images generated by the neural network began spreading online, depicting, for example, an explosion crater in Los Angeles or a flooded U.S. Capitol. Such content grossly violates the service's rules, which became the formal reason for the shutdown.
The reaction of open data researcher Henk van Ess is telling: during testing, he found that his prompts — from creating an image of refugees at the Mexican border to visualizing a nuclear facility in Iran — were not blocked by the system at all. This directly points to the absence of adequate safety filters at the initial launch stage.
The situation once again demonstrates a fundamental problem with large language models: generative algorithms trained on vast datasets are unable to independently assess the ethical and political context of their "creations." For Google, which actively promotes its AI solutions in both corporate and consumer segments, such an incident is a reputational blow that could force the company to reconsider the pace of introducing such features into geoservices.
My take: This is not an isolated glitch but a systemic pattern. As long as developers prioritize speed to market over the quality of censorship, such incidents will recur. The lesson for the entire industry is simple: before giving AI access to geospatial data, content-filtering mechanisms must be strengthened tenfold, otherwise user trust will be undermined for good.