AI Tool for Social Workers Generates ‘Nonsensical’ Transcripts from Children’s Reports

The integration of AI tools into social work practices has raised significant concerns due to their potential to make critical errors in official records. Frontline workers have reported instances of inaccurate reflections, including unfounded alerts about suicidal thoughts and nonsensical entries, which can greatly affect the quality of care provided.

Last year, Keir Starmer endorsed what he described as groundbreaking technology capable of expediting social work transcription. However, a recent study conducted by the Ada Lovelace Institute across 17 local councils in England and Scotland indicates that AI-generated inaccuracies—sometimes referred to as “hallucinations”—are increasingly infiltrating these important records.

As many local authorities adopt AI transcription tools to streamline the documentation and summarization of meetings with both adult and child service users, the aforementioned research suggests that “potentially harmful misrepresentations of individuals’ experiences are appearing within official care documents.”

In one instance, a social worker utilizing an AI transcription tool explained that the technology wrongly “suggested there was suicidal ideation” when the client had never mentioned such thoughts or plans.

Another professional highlighted that the AI’s notes sometimes referenced irrelevant items like “fish fingers or flies or trees,” when a child was describing domestic conflicts. Experts in social work emphasized the gravity of such inaccuracies, warning that they could result in missed indications of risky behavior.

Concerns were also raised by social workers regarding the precision of transcriptions, especially those involving individuals with regional accents. One worker described their transcriptions as occasionally containing “gibberish,” while another remarked, “It’s become somewhat of a joke in the office.”

Numerous councils from locations ranging from Croydon to Redcar and Cleveland have provided social workers with AI transcription tools designed to capture and summarize case discussions. The promise of significant time savings is especially enticing for regions grappling with staffing shortages.

One particularly notable transcription system, known as Magic Notes, is offered to councils at rates between £1.50 and £5 per hour of transcription. Most social workers interviewed either used this specialized AI tool or the general-purpose Microsoft Copilot AI.

Interestingly, the research indicated that AI transcription can produce noticeable time savings and free up social workers to enhance their connections with service users. “Our findings illustrate that these tools can bolster relational aspects of care work along with the quality of information documented by social workers,” stated the study following interviews with 39 unidentified social workers.

However, concerns arose when one social worker employed an AI tool to reword care documents into a more “person-centered” tone. The AI inaccurately injected “words that were never spoken,” raising questions about the distinction between the social worker’s assessment and that generated by AI. Another practitioner voiced worries that the technology had blurred the lines between their professional judgment and the AI’s output.

The implications of AI-driven inaccuracies could be significant, potentially leading to misguided decisions about a child’s welfare and posing both safety risks for the child and professional repercussions for the social worker, as outlined in the report.

Reports of disciplinary actions have already surfaced in the profession, specifically for social workers who failed to adequately scrutinize the results of AI note-takers and overlooked glaring mistakes, according to the British Association of Social Workers (BASW). The organization is advocating for social work regulators to provide explicit guidelines regarding the usage of AI tools.

Despite a sense of “genuine anticipation” surrounding these technologies among some in the field, “they also introduce new dangers to social work and society, ranging from potential bias in report summaries to inaccurate ‘hallucinations’ found in transcripts,” commented Imogen Parker, the associate director at the Ada Lovelace Institute. “These risks are yet to be thoroughly assessed or addressed, leaving frontline workers to contend with these challenges without adequate support.”

Training for social workers on AI utilization is often minimal, sometimes just an hour long. Some professionals indicated they spend up to an hour verifying AI transcriptions, while others only check for a couple of minutes. One stated they took “five minutes to just scan it quickly” before pasting it into the system, and another noted that AI-generated care plans could be “horrifically” inaccurate.

It was also mentioned that some colleagues lack the motivation or time to evaluate the transcripts properly.

Andrew Reece, BASW strategic lead for England and Wales, articulated the risk of unchecked AI outputs: “The time spent documenting helps solidify your understanding of what was conveyed. If the computer is performing that task for you, you may miss vital elements of reflective practice.”

Seb Barker, co-founder of Beam, which operates Magic Notes, emphasized that AI outputs are merely first drafts rather than finalized records. “Social workers are adopting AI tools for valid reasons,” he asserted. “There are overwhelming service demands and documentation requirements that are ever-growing, leading to a risk of burnout for a new generation of social workers.”

Barker also mentioned that evaluations aimed at detecting bias revealed that Magic Notes performed “consistently and equitably,” showcasing features tailored for social work, including automated checks for hallucinations. “Not every AI tool is created equal; generic or low-quality tools frequently do not meet the specialized needs of the sector,” he concluded.

The UK government along with Microsoft has been contacted for their input regarding these concerns.