Dr Taylor on Assessing ChatGPT's Ability to Offer Specific Information in Hematologic Malignancies

“We saw, even amongst experts, there was a bit of difference in opinions about some of the specific questions. We did find that the experts disagreed more with [ChatGPT] on the specific questions as opposed to the general questions, where most of the experts agreed that the information was pretty accurate.”

Justin Taylor, MD, an associate professor in the Division of Hematology at the University of Miami Miller School of Medicine, discussed findings from a study that evaluated the ability of ChatGPT to offer specific information in hematologic malignancies.

Taylor was the coauthor of a study that was published in Future Science OA which evaluated the ability of ChatGPT version 3.5 to provide current and specific information in terms of patient-specific queries and novel therapies compared with more general inquiries in the management of hematologic malignancies. Taylor explained that although a literature review revealed some prior research on this topic, most existing studies were either conducted outside of the field of oncology or focused on other solid tumors, leaving an unmet need for data examining how large language models perform within the context of hematologic malignancies.

Additionally, Taylor highlighted that previously conducted studies often asked broad questions that did not reflect the specific, treatment-focused questions patients typically have about their disease, further underscoring the need for a more nuanced approach, which was employed in this study.

Study authors observed a difference in opinions about some of the specific questions, Taylor said. Experts disagreed with ChatGPT on more specific questions compared with the general inquiries. Findings from their study showed that ChatGPT’s responses in terms of general questions regarding hematologic malignancies received a higher average score (3.38) compared with questions on novel therapies and specific mutations (3.06), according to 4 human reviewers who were hematologic oncologists.