Evidence Matters

Use Case #1

Using a web-conferencing platform, parents of a child diagnosed with a rare genetic disorder meet remotely with a world-renowned clinical geneticist who can assess their child’s baseline level of function from the comfort of their living room.1

Use Case #2

With an app marketed under emergency use authorization during the coronavirus pandemic, a patient with schizophrenia learns self-management skills.2

Use Case #3

Using an artificial intelligence decision support system, a patient with Type 1 diabetes adjusts her own insulin dosing.3

Evidence is all around us. However, with over 2 million research articles published every year, it is impossible to keep up with the sheer volume of “evidence.” But perhaps there is some evidence that matters less, or doesn’t matter at all. In the British Medical Journal, Smith argued that Parachutes reduce the risk of injury after gravitational challenge, but their effectiveness has not been proved with randomised controlled trials.”  (Apparently, finding volunteers for the no-parachute control group was a challenge). In this light-hearted jab at evidence, Smith suggested two approaches for evaluating technologies, “[t]he first is that we accept that, under exceptional circumstances, common sense might be applied when considering the potential risks and benefits of interventions. The second is that we continue our quest for the holy grail of exclusively evidence based interventions and preclude parachute [or other technology] use outside the context of a properly conducted trial.”

At NODE.Health’s virtual Digital Medicine Conference (DMC) 2020, we introduce the term “Digitally Recognized as Safe (DRAS),” a digital equivalent of Generally Recognized as Safe (GRAS), a term that was unveiled in 1960 for food chemicals and additives for which years of experience substituted for study-based evidence.  DMC 2020 “Evidence Matters” will help you understand when evidence matters and when, perhaps, it doesn’t. Using the examples above, and lessons from insightful sessions such Why Evidence MattersInvestors: What you Need to Know About Evidenceand Understanding and Designing Clinical Trials, we’ll highlight for you a few of the takeaways you’ll learn at this year’s DMC. Evidence is what many of us think of as “proof.” In healthcare and research, we might narrow the concept of evidence down to “proof of causation;” that a given intervention is responsible for an outcome. 

Evidence is the result of a systematic observation (such as the scientific method) to: 1. Establish facts, and 2. Reach conclusions. But in reality, very little today comes with such “proof.” And in the world of digital health and big data, real-world evidence (RWE) – evidence not collected from a controlled clinical trial, but from everyday use at home and on the go  – can be generated in volumes and at rates that far outpace the gold standard of the randomized controlled trial (RCT). So, will the RCT go the way of the dodo in the looming shadow of its nimbler RWE cousin?

Use Case #1:

For the parents using a telehealth platform to bring a remote, world-renowned geneticist into their own living room to observe their child’s baseline level of function, “digital health tools,” defined as “technologies, platforms, and systems that engage consumers for lifestyle, wellness, and health-related purposes…” might fall under DMC 2020’s newly coined term, DRAS. Certainly, some direct-to-consumer apps for general wellness and lifestyle, which do not manage or treat a specific condition but promote overall well-being by reducing stress, promoting optimal sleeping habits, or improving one’s outlook, are typically regarded as low-risk and safe, and are not subject to regulatory review. A platform for telehealth visits may offer much greater benefit than harm, with the added benefit that the geneticist can observe more accurate baseline behavior in the child’s natural environment. Barring potential cybersecurity and privacy risks of telehealth, does a study necessarily need to demonstrate the clinical effectiveness of such a telehealth platform to observe behavior, or is this another parachute example?

For clinical cases where audio, video, and image capture are sufficient for the clinician, we may not need further evidence. Case closed, DRAS, right? Not so quick…because just as a parachute would not be effective in preventing trauma in a jump from the top of a ladder, the telehealth platform might not be safe or effective in encounters where vital signs and a physical exam involving palpation (touching) and auscultation (listening with a stethoscope) would be needed. Finally the Evidence for Video Health is sure to be of interest.

Goalposts for Evidence

In its own efforts to stratify when evidence is needed for regulatory approval of Software as a Medical Device (SaMD), the Food and Drug Administration and the International Medical Device Regulators Forum (IMDRF) have taken a risk-based approach based on two dimensions (Figure 1).  One dimension is the state of the patient’s healthcare situation. The more serious the condition, the more evidence ought to be required of the SaMD. The other dimension is the way in which the information provided by the SaMD is to be used, ranging from simply informing to treating or diagnosing. The higher the risk score, the more evidence required.

Figure 1. FDA’s SaMD Risk Categorization Framework

Use Case #2:

In our example of the patient who uses a digital medicine tool to self-manage schizophrenia, the use may fall into the FDA risk categorization framework of informing clinical management all the way to treating the patient. Depending on the patient’s condition at the time, the risk could be anywhere from low to high. So, is evidence needed? In the current pandemic, under which the FDA declared an emergency use authorization for mental health apps, enabling them to accelerate to market and bypass customary regulatory pathways, the FDA has seemingly signaled that the mental health benefits of such products may currently outweigh the risks. So, in some cases, when evidence matters may be a matter of moving goalposts. Mental health apps are at the leading edge, being among the frontrunners in digital health to have received FDA approval. However, inconsistent results about their ability to improve health outcomes are common. For those interested in how the regulatory environment sets standards for evidence, don’t miss Digital Regulations: What you MUST know.

New Data Models for Evidence

The randomized controlled trial (RCT) has long been the gold standard for evidence. But in an era when digital therapeutics, defined as “products that deliver evidence-based therapeutic interventions to prevent, manage, or treat a medical disorder or disease,” 8 iterate rapidly in the marketplace, generating data in real-world populations, the FDA has signaled its own intent to begin to consider other models for evidence collection, including RWE.

Use Case #3:

In our example of the patient using an artificial intelligence-based decision support system to titrate her own insulin, is RCT-style evidence necessary? If one considers the risk/benefit ratio, knowing that insulin in excess, can cause hypoglycemic events that may be fatal, it seems quite reasonable that high quality, RCT evidence would be warranted. But the same conclusion might not apply to an artificial intelligence system that helps patients titrate other medications without such severe consequences. 

In fact, some have argued thatUnder certain conditions, real-world studies can provide relevant evidence to decision makers, even in absence of RCTs, therefore becoming not only a complementary source of evidence but also a low-cost, rapid and valuable substitute for technologies whose diffusion process has already started.”  DMC attendees will not want to miss the Balancing Regulation, Innovation, and Safety During and Pandemic and Beyond session.

Welcome to the NODE.Health Digital Medicine Conference: “Evidence Matters” (except when it doesn’t)

At the NODE.Health conference, we love to understand the evidence behind the effects of digital health tools on health outcomes. But many digital health companies also promote versions of evidence on operational parameters of their tools too, including patient engagement, Net Promoter Score, and reliability. Marketing is not evidence unless it is supported by good science. But let’s keep in mind when evidence matters, and when it’s simply a parachute.

We look forward to seeing you at the conference. As we explore emerging digital technologies such as artificial intelligence and machine learning where algorithms are re-writing the rules about evidence, one may remember a famous 1980 lyric about disruptive technology “written by machine on new technology,” from the one-hit-wonder Buggles’ pop hit “Video Killed the Radio Star.” If RCT evidence in digital is threatened by new, more nimble models of RWE, must we all begin to hum a new tune and should we begin wonder whether “Digital Killed the RCT Star?”  Come to the NODE.Health conference and hum along with us. The evidence awaits you!

1Myers K, Stoep AV, Zhou C, et al. Effectiveness of a Telehealth Service Delivery Model for Treating Attention-Deficit/Hyperactivity Disorder: A Community-Based Randomized Controlled Trial. Journal of the American Academy of Child & Adolescent Psychiatry. 2015;54(4);263-274. doi.org/10.1016/j.jaac.2015.01.009

2Lovett L. Pear rolls out digital therapeutic for schizophrenia after FDA loosens regulations in COVID-19 response. MobiHealthNews. April 29, 2020. Accessed October 14, 2020. https://www.mobihealthnews.com/news/pear-rolls-out-digital-therapeutic-schizophrenia-after-fda-loosens-regulations-covid-19

3Nimri R, Battelino T, Laffel LM, et al. Insulin dose optimization using an automated artificial intelligence-based decision support system in youths with type 1 diabetes. Nature Medicine. 2020;26;1380–1384. doi.org/10.1038/s41591-020-1045-7Â

4Smith GC, Pell JP. Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ. 2003;327(7429):1459-1461. doi.org/10.1136/bmj.327.7429.1459

5Goodrich WW. Safe Food Additives and Additives Generally Recognized as Safe— There is a Difference. The Business Lawyer. 1960;16;98-106. https://heinonline.org/HOL/LandingPage?collection=journals&handle=hein.journals/busl16&div=14&id=&page=

6Rada G. What is the best evidence and how to find it. BMJ Best Practice. Accessed October 20, 2020. https://bestpractice.bmj.com/info/us/toolkit/discuss-ebm/what-is-the-best-evidence-and-how-to-find-it/

7Goldsack J, Coder M, Fitzgerald C, et al. Digital Health, Digital Medicine, Digital Therapeutics (DTx): What’s the difference? Digital Therapeutic Alliance. November 11, 2019. Accessed October 20, 2020. https://dtxalliance.org/2019/11/11/digital-health-digital-medicine-digital-therapeutics-dtx-whats-the-difference/

8Food and Drug Administration. Enforcement Policy for Digital Health Devices for Treating Psychiatric Disorders During the Coronavirus Disease 2019 (COVID-19) Public Health Emergency. Docket no FDA-2020-D-1138. April 2020. Accessed October 25, 2020. https://www.fda.gov/media/136939/download

9Food and Drug Administration. General Wellness: Policy for Low Risk Devices. Docket no FDA-2014-N-1039. September 2019. Accessed October 25, 2020. https://www.fda.gov/media/90652/download

10Roakdia H, Zawada S, Rosner B, et al. Telehealth visits with your doctor expanded rapidly during COVID…and the hackers are right on their heels. NODE.Health. July 30, 2020. Accessed October 20, 2020. https://nodehealth.org/2020/07/30/telehealth-visits-with-your-doctor-expanded-rapidly-during-covid-and-the-hackers-are-right-on-its-heels/

11Food and Drug Administration. Global Approach to Software as a Medical Device Software as a Medical Device. Accessed October 25, 2020. https://www.fda.gov/medical-devices/software-medical-device-samd/global-approach-software-medical-device-software-medical-device

12Lau N, O’Daffer A, Colt S, et al. Android and iPhone Mobile Apps for Psychosocial Wellness and Stress Management: Systematic Search in App Stores and Literature Review. JMIR Mhealth Uhealth. 2020;8(5):e17798. doi.org/10.2196/17798

13https://nodehealth.org/2020/09/16/the-changing-regulatory-landscape-during-and-after-covid-19/

14Tarricone R, Boscolo PR, Armeni P. What type of clinical evidence is needed to assess medical devices? European Respiratory Review. 2016;25(141);259-265. https://err.ersjournals.com/content/25/141/259Â