A new report from the Common Sense Media’s Institute for Youth and AI Safety has revealed that Google’s AI Overview and AI Mode features do not provide an adequate level of safety for children and teenagers, as tests showed they failed to handle several sensitive scenarios, such as suicidal thoughts, self-harm, psychosis, and eating disorders. In a statement to Mashable, Google said the research is flawed and does not reflect how users interact with its search products.

Between May and July of this year, Common Sense Media’s Institute for Youth and AI Safety tested the performance of the “AI Overview” and “AI Mode” features on 2,600 interactions. The researchers evaluated how well the products handled several sensitive topics, including suicide and self-harm, child sexual exploitation, and psychosis.

The report’s authors wrote: “Our tests showed that both AI Overview and AI Mode failed to provide adequate support for children in crisis, including missing clear signs of suicidal thoughts, reinforcing signs of psychosis and obsession, normalizing eating disorders including vomiting, and encouraging cannabis use.” Additionally, the AI mode often provided different answers to the same history questions than AI overview.

Common Sense Media made important recommendations based on the findings, emphasizing that schools and families should have the option to disable and review AI features. They advised elementary school teachers to steer students away from Google’s AI-powered search engine entirely, instead relying on library databases or other research tools. Families were also recommended to seek out a search engine that doesn't provide AI-driven answers.

Google, which reviewed the report before publication, reported it couldn’t verify or reproduce many of the findings in the report. Google claimed its own tests resulted in higher quality responses for the same queries used by the lab.

In a statement to Mashable, Google said: “Our AI search features are a very useful tool for kids and teens to learn, explore, and understand information and the world around them.” This report tests a limited set of vague and artificial queries that don’t reflect how people use search, nor is it an effective way to measure product safety and usefulness.

The Risks of Google’s AI-Powered Search

Jeffrey A. Fowler, head of public engagement at the Institute for Artificial Intelligence Safety for Youth (AI4Youth) at Common Sense Media, told Mashable that turning off the “AI Overview” and “AI Mode” features by default for children and teens is a straightforward solution. “We don’t think this is a revolutionary position,” Fowler said, noting that in other parts of the world, these two features are disabled for minors.

Google told Mashable that parents can block access to search on Android devices and Chrome browser through parental control tools. Parents must manually add Google.com to the list of blocked sites in their Google Chrome and web settings for this option to work on Chromebooks.

Recommendations for Schools and Parents

Based on the study’s findings, researchers recommended allowing schools and families to turn off AI Overview and AI Mode features, especially for children. They also called on elementary school teachers to steer students away from AI-powered search and instead rely on library databases or traditional search engines that do not provide AI-generated answers.

Google rejects the findings

Google said it had reviewed the report before publication, but could not verify or reproduce many of the results. It confirmed that its internal tests provided higher-quality answers when using the same queries and added that AI-powered search features provide a useful way for children and teenagers to learn and explore information. The company considered that the study relied on a limited set of "artificial and ambiguous" questions that do not reflect natural usage of the search engine, and therefore are not suitable for measuring the level of safety or actual benefit of the product.

Demands to disable features for minors

Jeffrey Fowler, head of community engagement at the Institute for the Safety of Artificial Intelligence for Youth, said that the most obvious solution is to disable the AI Overview and AI Mode features for teenagers and young people by default. He noted that some countries already apply restrictions preventing minors from using these features. He added that the United States still allows their use without similar restrictions. In contrast, Google explained that parents can block access to the search engine on Android devices and in the Chrome browser using parental control tools. It also imposes restrictions on minors' access to the Gemini chatbot.

Failure in Handling Psychological Crises

Researchers tested 652 queries related to mental health, including anxiety, suicide, self-harm, and eating disorders. Although both systems sometimes identified critical cases and provided links to assistance, they did not meet the researchers' safety standard of detecting 95% of critical cases.

Of the 214 queries that required directing users to helplines or medical services, AI Overview provided appropriate resources in only 58% of cases, compared to 77% for AI Mode. The report cited an example of a query containing a colloquial expression referring to suicide. AI Overview responded with technical instructions on setting up a “digital contact,” instead of recognizing that the user might be in crisis.

Errors in Information and Homework

The report also raised concerns about the accuracy of information provided by AI-supported search. Tests showed that AI Mode completed all homework assignments requested, whether in mathematics or humanities essays, during 180 tests. Researchers tested over 275 questions on historical facts, finding that AI Overview and AI Mode's results differed in 43% of cases, despite identical questions, in terms of accuracy, detail, or the angle of information presentation.

In one example, the system was asked about the artist ranked number one on the Billboard Artist 100 chart and confidently answered that it was Taylor Swift, when in fact the correct answer is Drake. This was considered by researchers as evidence that the system may provide information that appears confident but is not accurate. Fowler concluded his statements by emphasizing that these products “were never really designed for children, but they’ve become widely used by them,” noting that it is unrealistic to expect a 10-year-old child to consistently verify the accuracy of every answer provided by AI.