In an unfamiliar medical experiment, China has allowed artificial intelligence to take over a large part of the workflow in an eye disease clinic, rather than just being used as a tool for doctors. The system took on tasks including analyzing eye images and helping detect diseases such as glaucoma and age-related macular degeneration while still under human supervision.

Although its performance was initially limited, it improved significantly after being trained on 1,426 high-quality images classified by specialist doctors, achieving over 0.93 in some performance tests. But the experiment revealed another challenge: doctors stopped using the tool when it was slow and complex, before it was optimized to be faster and easier. The results showed that AI success in medicine depends not only on its accuracy but also on data quality and ease of integration into daily workflows.

Researchers led by a team from Beijing Institute of Vision Science and Transient Eye Research conducted an experiment called AI-TEC, which is a clinic at Tsinghua University designed from the start to maximize the use of artificial intelligence while still involving doctors and human supervision.

The system included multiple stages in the patient journey, starting with the initial consultation, through eye examinations and image analysis, up to follow-ups after visits.

The experiment showed that the AI’s performance in diagnosing some eye diseases, such as glaucoma and age-related macular degeneration, was initially not high. But accuracy improved significantly after it was provided with 1426 high-quality images that were accurately classified by specialist ophthalmologists.

These data were more effective than the original training set which included about 27 thousand images, but they were of lower quality and less comprehensive in terms of case classification.

Using the AUROC scale to evaluate diagnostic performance, the system’s accuracy exceeded 0.93 after updating the training data, a level close to that achieved by other modern systems for analyzing eye images.

But the surprise was not only in the AI’s accuracy but also in how much it was actually used by doctors and staff.

After five months of operation, the AI was only used in 41 examinations out of 1113 during one month, which is about 3.8% of cases where it could have been used.

In the following month, the percentage rose to 23%, after researchers modified the system to make it faster and easier to use, with fewer clicks and manual inputs required from staff.

The experiment thus showed a problem that goes beyond the capabilities of the algorithm itself: an AI system can be accurate but not necessarily clinically useful if its use is complex or does not fit in with how doctors work.

Researchers believe that the success of this type of clinic depends on three key elements: high-quality medical data, an efficiently designed workflow, and rapid and regular feedback from doctors to develop the system.

The experiment also revealed a fundamental difference in how AI and doctors think; AI primarily focuses on determining the outcome, such as whether or not a disease exists, whereas a doctor usually starts with the patient's symptoms and complaints before reaching a diagnosis.

As a result, researchers concluded that the success of an AI-designed clinic should not be measured only by how well the algorithm performs in tests, but also by its ability to change clinical workflows and improve patients' health outcomes. The study was published in Nature Medicine.

• China
• Artificial Intelligence
• Eye Clinic
• AI Management