University at Buffalo researchers have developed what they describe as the first Artificial Intelligence system built specifically to distinguish radiology reports written by humans from those generated by Artificial Intelligence. The effort is aimed at reducing the risk that fabricated medical documents could be used for insurance fraud, falsified disability or malpractice claims, and other cybercrimes. The focus is on radiology because the field relies on highly specialized structure, vocabulary, and writing conventions that make general-purpose detection tools less dependable.
The team presented its study, “Detecting Synthetic Radiology Reports Using Style Disentanglement,” at the 2025 GenAI4Health workshop held during the Conference on Neural Information Processing Systems in San Diego in December. As part of the work, the researchers built a dataset of 14,000 pairs of radiologist-authored and Artificial Intelligence-generated chest X-ray reports. The synthetic reports were created in two ways: paraphrasing real radiologist reports with large language models, and generating full reports directly from chest radiographs using medical vision-language models. Researchers said the dataset is the first to combine both text-based and image-based synthetic radiology reports, with all samples limited to the findings section of reports.
The detection framework was designed to separate stylistic features from clinical content, based on the idea that Artificial Intelligence systems can reproduce medical terminology but still leave recognizable writing patterns in phrasing, punctuation, and word choice. Built on a BERT-Mamba-based model, the system distinguished human-written reports from synthetic ones with high accuracy and consistency, achieving Matthews correlation coefficient (MCC) scores between 92% to 100% in both text-to-text and image-to-text categories. Even when Artificial Intelligence outputs closely resembled the original reports, text-to-text detection accuracy still exceeded 99%. The framework also performed well in cross-LLM tests, identifying Artificial Intelligence-generated reports from models it had not previously encountered.
The researchers said stylistic differences helped drive those results. Large language models tended to produce more polished and expansive wording, while clinicians were more concise and direct. Examples included simple terms such as “heart” or “lung” being replaced by more elaborate phrasing such as “pulmonary vasculature,” which became a detectable signal for the model. The team is now refining both the dataset and the benchmark detection system ahead of a planned public release, while also expanding the work to more radiology categories and a broader range of Artificial Intelligence models.
Although the project centered on medicine, the same style-based detection approach could extend to other sectors vulnerable to fabricated records and synthetic narratives, including insurance, finance, journalism, education, and the legal profession. The researchers also emphasized that Artificial Intelligence can still be beneficial in radiology, particularly as a way to save time and help radiologists manage growing workloads, provided the technology is deployed safely and evaluated rigorously.
