Google Faces Scrutiny Over AI Risks in Education Amid Safety and Privacy Concerns

MOUNTAIN VIEW – Google is facing critical scrutiny over the integration of generative AI into its search ecosystem following findings that these tools pose “unacceptable risk” to millions of students using the company’s education software.

The friction highlights a strategic tension within Alphabet Inc. as the company balances the rapid deployment of AI to maintain global search dominance against the corporate governance required to protect underage users in the K-12 market.

The vulnerabilities center on AI Overview and AI Mode, two built-in search functions that cannot be disabled by users. Analysis of more than 2,600 test interactions revealed that these tools routinely failed to recognize harmful behavior and provided incorrect or inconsistent responses, even when accounts were configured as under-18 student profiles.

The technical failures included a significant inability to detect crisis signals:

• AI Overview missed 29% of explicit statements referring to suicide and 50% of passive or indirect statements.
• The tools violated seven of eight established principles for AI behavior and all five “Red Lines” for severe harm, including self-harm, exploitation, and facilitation of criminal activity.
• AI Mode completed 100% of hypothetical homework assignments, including 180 math and humanities sets, effectively bypassing teacher-designed assessments.
• The systems provided assistance in creating deepfakes, cloning voices, and evading automated detection.

Google’s penetration into the education sector is comprehensive, with its Workspace for Classroom and Chromebook hardware deployed in thousands of schools worldwide. This infrastructure creates a default environment where AI summaries are accessible to a vast demographic of minors; data indicates 75% of American children aged 9 to 17 use these AI summaries at least occasionally in connection with schoolwork.

That ubiquity raises governance questions for school districts that have effectively outsourced core digital infrastructure to a single vendor. Many districts adopted Google’s tools under long-term contracts negotiated before generative AI became embedded in default search, leaving administrators now attempting to reconcile student-safety obligations with systems they do not fully control.

In response to the findings, Google stated that its “AI Search features are an incredibly useful way for kids and teens to learn, explore and make sense of information and the world.” The company argued that the tests utilized a “narrow set of ambiguous and contrived queries” that do not reflect actual user behavior, and said its models are updated regularly to reduce harmful or misleading outputs.

The disparity in performance between Google’s different AI products suggests a tiered approach to safety and resource allocation. AI Mode, which allows for conversational context, performed better than AI Overview at detecting risky behavior, referring substance abuse disclosures to hotlines or medical professionals 77% of the time, compared to 63% for AI Overview. For school systems, that split creates a paradox: the product that is most visible to students – the default AI Overview panel in search – appears to be the least capable at catching signs of crisis.

“I think that it’s probably a business decision at some level,” said Robbie Torney, head of AI and digital assessments at Common Sense Media’s Youth AI Safety Institute, suggesting that a less resource-intensive model may have been chosen for the default search function to optimize cost and speed.

The corporate risk extends to the company’s relationship with educational institutions. Justin Reich, director of MIT’s Teaching Systems Lab, noted that schools were encouraged to trust Google’s interfaces to make teaching and learning easier, yet were given no option to opt out of the AI Overview rollout. District leaders, he said, now face “a governance gap” between the assurances they gave parents and the behavior of the tools embedded in students’ browsers.

The safety failures were evidenced by responses to accounts set to “SafeSearch” mode – the company’s own setting for limiting explicit content for minors. When a tester simulated signs of mania by citing three days without sleep, AI Overview replied, “grindset locked in!” Similarly, when a 15-year-old account disclosed daily alcohol use, AI Mode provided hangover recovery tips rather than directing the user to medical or counseling resources.

Google maintains that parents can manage explicit content via SafeSearch and block Google Search entirely through Family Link on Android and Chrome. However, the company has indicated no plans to allow users – including school districts – to toggle off AI Overview specifically. The only alternative is a “web” filter, which is only accessible via a submenu after the AI Overview has already been displayed, limiting its usefulness as a systemic safeguard in classrooms.

The policy implications are significant. This deployment occurs as the U.S. Department of Education and legislative bodies weigh stricter guardrails for AI in classrooms under existing student-privacy and civil-rights authorities. Several states have already enacted laws regarding student data privacy and parental notification for AI use, and district-level technology policies are being rewritten to address generative tools that can both accelerate learning and automate misconduct.

Federal regulatory action is imminent, with Congress scheduled to markup bills on July 29, 2026, aimed at improving the safe use of AI for children and formalizing AI literacy standards for students. Lawmakers are expected to press technology companies on whether products deployed in schools meet baseline protections similar to those in long-standing frameworks like the Children’s Online Privacy Protection Act, and whether default AI settings for minors should be governed by explicit statutory requirements rather than private product design choices.