Highly Risky: Insights from a Mental Health Specialist on Google’s AI Summaries

A year-long commission has been initiated by Mind with the purpose of investigating the intersection of AI technology and mental health. This decision follows a revealing investigation by The Guardian, which highlighted that Google’s AI Overviews, currently reaching 2 billion individuals each month, have been providing “very dangerous” mental health advice.

In this context, Rosie Weatherley, the information content manager at Mind—the largest mental health charity operating in England and Wales—articulates the significant risks associated with these AI-generated summaries that prominently appear above regular search results on the most visited website globally.

“Over the span of more than three decades, Google has painstakingly crafted a search engine designed to ensure that credible and accessible health content can rise to the top of its results. The traditional online search for health information, albeit not perfect, was generally responsive and effective. Users were usually directed to legitimate health websites that proficiently addressed their queries.”

“However, the introduction of AI Overviews has fundamentally altered this landscape. These clinical-sounding summaries offer an illusion of certainty and conclusiveness, effectively replacing the richness of information users previously relied upon. This transformation can be alluring but is ultimately irresponsible, often truncating the information-seeking journey and leaving users with only partial answers.”

“To assess the impact of this change, I tasked both myself and my team of mental health information experts at Mind to conduct a brief search session. We focused on common queries known to be used by individuals facing mental health challenges. Surprisingly, we found that our exploration barely required 20 minutes.”

“In just under two minutes, Google presented AI Overviews that inaccurately suggested starvation is a healthy practice. Another colleague was informed that mental health issues arise solely from chemical imbalances in the brain. Yet another was misled into believing her perceived stalker was real. Lastly, one of us was told that a staggering 60% of welfare claims related to mental health conditions are simply attempts at malingering. It should be abundantly clear that none of these assertions hold any truth.”

“Each of these instances illustrates how AI Overviews tend to oversimplify critical information on sensitive and intricate subjects, reducing complex topics to neatly packaged responses. The removal of essential context and nuance in the presentation transforms potentially harmful inaccuracies into plausible assertions.”

“This phenomenon is particularly detrimental for individuals who may already be experiencing a degree of distress. A tech behemoth like Google, which benefits financially from AI Overviews, should be allocating more resources towards ensuring the accuracy of the information it disseminates. Unfortunately, their existing approach appears to largely involve reactively retraining or withdrawing AI Overviews only when issues have been flagged by individuals, organizations, or journalists. This method of addressing problems feels superficial and doesn’t adequately match the size and resources available to a corporation reaping immense profits from such technologies.”

“Although search engines have made considerable progress in rendering the most harmful content, such as methods of self-harm or suicide, less accessible, the risk remains for individuals searching for information in a vulnerable state to encounter dangerous inaccuracies presented as calm, uncontroversial assertions deemed factual by the world’s leading search engine.”

“A recent search for crisis information revealed that AI Overviews combined various contradictory signals in long, confusing lists. While AI undoubtedly holds immense potential to enhance lives, the current risks presented are genuinely concerning. Google only appears to implement protective measures when it identifies users in acute distress. However, individuals deserve equitable access to constructive, empathetic, and nuanced information, regardless of their current emotional states.”