Quality of Information on Hypersensitivity Pneumonitis Generated by Artificial Intelligence Chatbots for Clinical and Patient Use

Abstract

Background and aim

Hypersensitivity pneumonitis (HP) is a complex immune-mediated interstitial lung disease that requires integration of clinical, radiological, and exposure-related data for accurate diagnosis and management. With the increasing public use of artificial intelligence (AI)–based chatbots for health-related information, concerns have emerged regarding the quality, reliability, readability, and patient usability of AI-generated medical content. This study aimed to evaluate AI chatbot–generated information on HP from a patient education perspective.

Materials and methods

Four frequently searched patient-oriented questions related to HP were identified using Google Trends. These questions addressed disease definition and causes, clinical features, treatment, and diagnosis. Each question was submitted verbatim to eight widely used AI chatbots, generating 32 responses. Four pulmonologists with expertise in interstitial lung diseases independently and blindly evaluated all responses. Content quality and reliability were assessed using the DISCERN instrument; understandability and actionability using the Patient Education Materials Assessment Tool for Print Materials (PEMAT-P); overall written clarity using the Written Readability Rating (WRR); and structural readability using the Flesch–Kincaid Grade Level (FKGL).

Results

All chatbot-generated responses required advanced literacy, with FKGL scores ranging from 20.17 to 29.07, corresponding to college-level or higher readability. None met recommended patient-appropriate readability thresholds. WRR scores declined as clinical complexity increased, with the lowest scores observed for diagnostic explanations. DISCERN scores varied across chatbots (35.00–57.10), indicating generally fair to good quality but incomplete information. PEMAT-P understandability scores were moderate, while actionability scores were consistently low across all models.

Conclusion

Although AI chatbots can generate clinically detailed information on HP, current outputs remain difficult for patients to understand and apply. Until improvements in readability, actionability, and reliability are achieved, AI-generated content should be used cautiously and under professional supervision in patient education.