Artificial intelligence note-taking tools utilized by Ontario doctors were flagged for providing inaccurate and incomplete information, as well as showing signs of “hallucinations,” according to a recent report by the province’s auditor general. Shelley Spence, the Ontario Auditor General, highlighted these issues while investigating the broader use of artificial intelligence within the provincial public service.
During the evaluation of AI medical systems meant to transcribe doctor-patient conversations, government assessors encountered significant errors in the transcripts generated by 20 different programs. The report emphasized the potential risks associated with inaccuracies in medical notes produced by AI scribe systems, which could lead to substandard or even harmful treatment decisions affecting patient well-being.
To test the capabilities of these AI systems, evaluators conducted simulated conversations between fictitious doctors and patients. The assessment revealed that nine systems exhibited “hallucinations,” fabricating information or suggesting treatment actions that were never advised by the doctors. For instance, some systems recommended therapy or blood tests without any actual doctor input.
Among the findings, transcripts from 12 programs contained incorrect details such as recording incorrect drug prescriptions, while 17 systems overlooked crucial information related to patients’ mental health. Additionally, several vendors failed to provide essential documentation like third-party audit reports and certifications during the procurement process, yet four systems were still approved for use by the government.
Despite the identified shortcomings, these AI scribe systems are currently operational across the province. To address concerns regarding accuracy, the government has issued guidelines for doctors to manually review AI-generated notes. The auditor general recommended implementing IT controls in the systems to ensure doctor attestation, confirming the thorough review of notes.
Minister of Public and Business Service Delivery and Procurement, Stephen Crawford, acknowledged the errors discovered during the testing phase of the process. Stressing the oversight of professionals in the decision-making process, Crawford highlighted the systems’ purpose of streamlining administrative tasks for physicians, allowing them more time for patient care.
Health Minister Sylvia Jones did not participate in a public response to the report but emphasized that approximately 5,000 physicians in Ontario are currently utilizing AI scribes without any reported patient harm incidents. All documentation generated by these systems requires doctor review and approval before being added to a patient’s medical record, ensuring patient consent and awareness of AI usage during appointments.
The audit, which prompted ten recommendations for enhancing AI utilization within the public sector, primarily focuses on bolstering security and privacy measures. The government and its procurement agency have agreed to implement nine of these recommendations, aiming to address the highlighted issues and improve AI functionality across various sectors.
