The Hallucination Model
A clinician receives a microbiome report and see that 5 bacteria are outside of reference ranges according to the report. The clinician then searches the biomedical literature for an intervention on the US National Institute of Health that appears to move each of those taxa toward the report’s reference range and prescribes that intervention. This is prescribed to the patient. Mission accompanished.
Of course, any experienced clinician will know that he will not find such a study. A well-read clinician may be ROFL (Rolling on the Floor Laughing) with this approach for a variety of reasons:
- Reference ranges are typically done using means and standard deviation. This imposes an assumption of a normal distribution on the data. Microbiome data often exhibits a skew of 20-30; a skew over 2 excludes the use of a normal distribution. The term “normal” is often misunderstood; applying a casual conversation meaning instead of the statistical meaning. Laboratory reference intervals should not be confused with clinically meaningful targets. A reference interval is generally a statistical description of a selected comparison population, not evidence that a value outside that interval causes disease or that moving the value inward improves patient outcomes.
- It is extremely unusual for a clinical microbiome test to be the same as that used in any study. The National Institute of Standards and Technology has for over 10 years identified a severe lack of standardization as undermining microbiome research. Their lead has accurately stated in 2019: “There are currently 97 different ways to analyze the same raw data, and they will give you 97 different answers“. You cannot safely assume that the study results apply to the patient test.
- Finding all of the targeted bacteria in the same study is extremely unlikely. A common response is to find a study in isolation for each bacteria and then synthesize a combination of substances. This assumes complete independence of each substance and its bacteria influence. This is typically false with many substances helpful for one bacteria and contraindicated for another. To be safe, the clinician will need to read every published study for each substance; a time requirement impractical in a clinical setting.
- The impact of a substance on a bacteria may be inconsistent. Baseline microbiome composition and function, diet, lifestyle, antibiotic exposure, age, comorbid disease, and concomitant medications can all influence both microbial response and clinical effect. The direction of change observed in one population may not generalize to another, and a change in microbial abundance is not necessarily accompanied by an improvement in symptoms or disease outcomes.
A frustrated clinician (or patient) may simply asked some Large Language Model for an answer. The response would often be called hearsay in a court of law.
A fuller description is below.
Suggested Process
Ask the “expert” to read the above and explain how they work given this background paper. Expressions like “from experience” or “trust me” indicates a high risk of the person going to harm you instead of help you.