The Intelligent Trial: AI Comes To Clinical Trials

By Maxine Bookbinder

September 29, 2017 | Artificial intelligence is revolutionizing health care and advancing the clinical trial process faster than any Hollywood AI robot could predict. Besides cutting costs, improving trial quality, and reducing trial times by almost half, AI is finding biomarkers and gene signatures that cause diseases, recruiting eligible clinical trial patients in minutes, reading volumes of text in seconds, and is on the cusp of breakthrough discoveries involving new diagnostic tools and treatments for Alzheimer’s disease, cancer, and other chronic and terminal illnesses.

Created in the 1950’s, AI is the ability of a computer or machine to simulate human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages. It is dependent upon three elements: massive amounts of data, sophisticated algorithms, and high performance parallel processors. One aspect, machine learning (ML), can rapidly assess multiple texts, graphs, and other data simultaneously.

The decreasing cost of computers and software, the increasing push for precision medicine, the abundant availability of an exponential amount of data, and the public’s impatience with the rising cost of drugs and medical care may contribute to the recent surge in AI adaptation and acceptance.

“This is not wizardry. It sounds like a mystical concept,” says Atul Butte, Director of UCSF’s Institute of Computational Health Sciences, Distinguished Professor of Pediatrics, and Executive Director for Clinical Informatics for the University of California Medical Schools and Medical Centers. “But in fact, you can go on Amazon and buy Machine Learning for Dummies for less than $20.” He cites logistical regression and “old-fashioned statistics” as examples simple machine learning. “We’ve always been using these techniques in trial designs.”

Artificial intelligence is an opportunity to personalize and optimize health care, says Trials.AI’s Product Manager Josh Stanley. “Trials are slow to progress; some researchers still use paper and pen. Their philosophy is, why change when it’s working? But it’s not. A huge percentage of trials fail, and a huge percentage don’t reach their recruitment and retention goals.”

Despite progress already made and promise ahead, many voices warn of potential negative consequences.

Pandora’s Box

For example, Tesla creator and Space-X CEO Elon Musk once said that AI is a “fundamental risk to the existence of human civilization,” and donated $10 million to the Future of Life Institute to run a global research program designed to keep AI beneficial to humanity. Physicist Stephen Hawking warned that AI “may be the best, or the worst thing, ever to happen to humanity” and later added that AI is “crucial to the future of our civilization and our species.”

Accenture estimates that AI will increase economic growth across 16 industries by 1.7% by 2035 and has the potential to increase productivity by 40%. A separate study done by researchers at Carnegie Mellon University and Albert Ludwig University in Germany, estimates that AI could cut the cost of drug discovery by about 70% (DOI: 10.7554/eLife.10047).

Unlike HAL, real-life artificial intelligence is not self-learning or self-aware. Humans must feed data, good and bad, successes and failures, into the software so it can learn patterns and experiences. “Before you have a [medical] case that needs to be fixed, it’s best to have given the computer 1000 or 10,000 more examples so it can learn,” says Butte. “Then when you have the 10,001st case, the computer now has a good guess of what went wrong. We have to teach computers, feed data, and let them discern the patterns.”

Trial Intelligence

Trials.AI’s Stanley says the role of AI is not to replace, but to augment and assist human intelligence, incorporate user-friendly efficiency, leverage data and make predictions in clinical trials to detect trends and outcomes. For example, AI can analyze operational data from historical cases, measure responses to drugs, predict site performances and use predictive criteria to determine whether taking a drug will result in a positive or negative outcome, whether the patient will drop out, and whether a trial will be successful. Clinicians can review a patient’s medical history and receive lab results within one system simultaneously. AI can ingest millions of words of text, molecules, genomic sequences, and images in minutes, aggregate the data and devise hypotheses beyond human ability. It does not argue dogmatic ideas, push personal theories, or maintain judgmental ideas.


Every year in the U.S., approximately 2 million patients participate in roughly 3000 clinical trials; six million patients are needed to meet U.S. recruitment goals. Consequently, up to 90% of trials are delayed or over budget. “The more patients who enroll in trials, the faster the pace of research and healthcare innovation in general,” says Wout Brusselaers, CEO and co-founder of Deep 6 AI, which uses AI to mine medical records to accelerate finding and recruiting patients for clinical trials.

Deep 6 AI’s software can find eligible patients for complex trials within minutes, depending upon the criteria. For example, in an early comparison search, a principal investigator using traditional recruitment methods to validate 23 eligible patients in six months for a biomarker for a non-small cell lung cancer trial. The Deep 6 AI software found and validated 58 eligible matches in less than 10 minutes. “We must find patients quickly,” says Brusselaers, “to develop cures faster.”

Researchers estimate that to hasten cures for cancer 25%-50% of cancer patients should be enrolled in trials. But, says former Vice President Joe Biden, “less than five percent of cancer patients enroll in a clinical trial often because patients and doctors don't know what trials are available.”

That’s where AI comes in. Deep 6 AI, winner of the 2017 Accelerator Competition Enterprise and Smart Data category at the SXSW Conference and Festivals, uses Natural Language Processing (NLP) to read doctors’ notes, pathology reports, diagnoses, recommendations, and to detect hard-to-find lifestyle data, such as smoking and activity history. It then graphs this information and matches appropriate patients with clinical trial criteria, potentially reducing recruitment time from years to just days or weeks.

The platform lets researchers compare patient graphs to find common traits in symptoms and diseases and disease progression and patient outcome, for example: "Show me a patient that looks like this one.” When researchers isolate 10 common characteristics of Alzheimer’s patients, the software can find undiagnosed patients with the same traits, and examine just how predictive of Alzheimer’s are these 10 conditions.

Another technique, called fuzzy matching, detects patients with undiagnosed symptoms and characteristics, such as cognitive decline or chronic pain. This allows sponsors who want to test new drugs for Alzheimer’s, chronic pain, and other disorders to find symptomatic, but undiagnosed, patients, and to offer these patients the chance to enroll in a trial they did not know about. 

Using algorithms based on both historical and incoming data throughout a trial, AI can predict which patients have a higher chance of dropping out or not following protocols. Clinicians can track patients individually, and email or text them reminders to take medications at specific times, journal, submit forms, or keep appointments. “We keep them engaged to help them stay in the trial. This also makes it much easier to manage, especially if you have 5 or 6 trials going on simultaneously,” says Trials.AI’s Stanley. This also gives clinicians more time to strengthen communication with patients.

Electronic reminders and check-ins give rural patients who have Internet access but don’t live near major medical centers or trial sites a chance to participate in trials. This also saves them from stressful distance travelling and long waiting room queues. “We should all have the same access to cutting edge care,” says Wout Brusselaers. “It doesn’t have to be expensive.”

Intelligent Speed

The clinical trials managed by Trials.AI have increased protocol adherence and decreased lost time. In its first trial, AI retained 98% of patients, had one critical deviation throughout the entire trial, and continued to completion without interruptions. “This is a huge opportunity to shorten, maybe even cut in half time to market,” says Stanley. The program allows investigators to more easily monitor cross-country sites, to know which ones are underperforming, to proactively address problems early, and to measure data they otherwise would not have access to in real time.

It is also a chance to leverage what is known about genetic biomarkers, says Stanley. “We can test therapies against patient cohorts with different gene mutations, and monitor and drop cohorts from the trial that aren't responding positively."

Working in concert with human intelligence, AI identifies specific genetic markers in populations to develop drugs for individual patients. “Humans are good for seeing patterns, but have built-in biases,” says Stanley. “It can be overwhelming; humans can’t filter out the noise in data. AI can discover patterns in data. We then can go back and see if it’s actionable, if it’s a meaningful discovery.” Computers, not doctors, analyze the mountains of data producing personalized medicine.

Hospitals, pharma, CROs and other sponsors can use the software to find patients with specific diseases, genomics, mutations, or any other characteristics. Pharma or CROs see a different platform with aggregated results, exclusion and inclusion criteria. Although they don’t see personal information, they know where prospective patients are.

AI is “crucial” in determining if a patient is suffering anxiety, depression, Alzheimer’s disease, or other cognitive impairments, says Frank Rudzicz, co-founder and president of Toronto-based WinterLight Labs.

Traditional paper and pen testing methods for Alzheimer’s disease “are crude, and up to the person administering the test to judge whether a person passes or not,” says Rudzicz, also a scientist at the Toronto Rehabilitation Institute. Instead, the WinterLight Labs team uses quantifiable data from algorithms that measure thousands of variations in voice patterns, including pitch, frequency, amplitude, grammar idiosyncrasies, subject matter, and its emotional impact on the patient.

Deaths from Alzheimer’s disease have increased by 89% since 2000, according to the Alzheimer’s Association and, along with other dementias, are expected to cost the US $259 billion in 2017 and top $1 trillion by 2050. 

“AI can hone in on very subtle, data-driven differences,” says Rudzicz. “This is beyond human perception. The input we get is the raw, direct measurements from the person as opposed to interpreted results. The patterns are in the data, and may not conform to a theoretical model a person came up with to describe a disease.”

For example, a depressed patient may use more personal pronouns and negative-sounding words. An Alzheimer’s patient replaces proper nouns (Aunt Elsie, TV remote) for pronouns (she, that) and hesitates more often between words. The software can then determine the average number of times a person uses negative words, find statistics related to a measure, and subsequently determine a patient’s positivity and negativity and whether he is at risk for depression. According to the National Alliance on Mental Illness (NAMI), depression is the leading cause of disabilities worldwide.


 “We can’t predict what the most important measurement will be,” says Rudzicz. “That’s where the machine learning comes in. If the AI says a feature or obscure measure is important, and it can make accurate predictions on someone’s cognitive health, at some point we have to be satisfied that the machine found something we didn’t.”

WinterLight Labs is conducting pilot programs and clinical trials to test the software’s accuracy. It is not yet used to diagnose, due to complex regulatory processes, and instead maps and validates existing diagnoses. It is currently working with two pharma companies to screen and recruit eligible patients for Alzheimer’s disease trials, expediting the process, reducing patient drop-out rates, and hopefully slashing millions of dollars in costs. Rudzicz also hopes to use the software as a remote device to monitor and assess patients’ cognitive health from their homes, replacing stressful distance traveling and long waiting room lines.

Crowded Field

The list of companies using AI or partnering with companies already using it is growing exponentially. Deep 6 AI partners with Translational Drug Development (TD2), and in one project is working to recruit participants for an oncology clinical trial testing a new drug. The team is searching multiple hospitals to find patients with a rare form of cancer who have not yet begun therapy. American technology company NVIDIA teamed with the National Cancer Institute, the U.S. Department of Energy and national research laboratories to use AI in creating a common discovery platform for cancer called CANDLE. Goals include uncovering the genetic DNA and RNA of common cancers, predicting how patients will respond to treatments, how each patient’s cancer evolves, building a database of disease metastasis and recurrence, and getting new therapies to market faster.

Deep Genomics, another Toronto company, looks for patterns in genomic data to find causal relationships with specific diseases. It is in early-stage drug development for inherited diseases caused by a single genetic mutation, which affect about 350 million people globally.

BERG Health is using AI to analyze tissue samples, genomics, and other data pertinent to a disease, which has resulted in a potential new drug for topical squamous-cell carcinoma. It passed early safety and efficacy trials and is waiting for full-scale testing. The company also is financing Project Survival, a seven-year project to improve the efficiency and price-point of drug research.

Korea Pharmaceutical and Bio-pharma Manufacturers Association are collaborating to buy an AI platform that will be used by around 20 Korean pharma companies. IBM, Atomwise, and Insilico Medicine are forming research partnerships with universities and nonprofits. GKS and Lawrence Livermore National Laboratory will use AI for pharmaceutical R&D. England’s Benevolent AI has an exclusive rights agreement with Janssen, giving the AI company exclusive access to an undisclosed number of clinical-stage drug candidates.

Last year, A Chinese team won a $1 million contest aimed at automating the detection of lung cancer using algorithms that most accurately identified signs of the disease in low-dose computed tomography images. The winning algorithms will be released to the public for free, hoping to inspire future medical imaging innovation.

Learning Better Questions

What happens to all these millions of points of structured and unstructured data from biomarkers, genomics, wearable devices, EMR, social media, labs, and imaging? It isn’t all used immediately. Some researchers store raw data in the event that, after more research, unused data might answer an unresolved problem and find a cure. James Streeter, Global Vice President of Oracle Life Sciences Product Strategy, says some clinicians “want to collect data because they think the answer is in there somewhere. Some don’t know what they are looking for. They have a hypothesis but haven’t proven it.” If, for example, says Streeter, “we can predict from genomics when a person will get cancer, we need millions of points of data to eventually be able to figure it out.”

“Other companies have made great strides in artificial intelligence, but today’s companies don’t know what to ask it. If they don’t know which questions to ask, they won’t get an answer. We are barely touching the surface of what we can do with data,” says Streeter. “We don’t know all the challenges. Companies do a great job bringing data together, but if you don’t know what questions to ask, you can’t get answers.”

Finding answers, or deciphering what questions to ask, is one of the challenges facing the industry. Algorithms are not infallible and can still have biases; datasets may have inherent limitations or biases built into them by humans. “It’s still an art to develop methods and models to eliminate errors built into the subsystems,” says Bruce Palsulich, VP of product strategy at Oracle Health Sciences.

Is AI more successful with certain patients or diseases? What are the inherent unknowns? What don’t researchers know? These are all questions for data scientists. Will an over-reliance on machine learning really result in a lack of learned skills, as some posited in a Viewpoint published in JAMA? (DOI: doi:10.1001/jama.2017.7797)


“We can’t expect AI to be perfect,” says Oracle’s Streeter. “We may not be asking the right questions.” Computers are only as smart as the information fed to them. Basically, researchers don’t know yet what they need to know.

Brusselaers says it is important not to overpromise and to realistically manage user expectations. While AI strips weeks and months off traditional tedious recruitment practices, medical staff are still making all the final decisions. “We are big believers in synergistic relationships between artificial and human intelligence.”

AI and humans are reducing these errors together. The FDA receives about 12,000 serious adverse event reports annually per individual reviewer, up 15%-35%, says Oracle’s Palsulich. “The volume is so high that it will never be feasible to support that without having machines do that work for us.” Machine learning extracts meaningful attributes from documents, sometimes in seconds, such as adverse reactions across multiple trials in multiple populations, pharmacogenetics, results from markets where the drug was previously approved, and medical record observations, to determine which documents humans need to review.

Increased automation does not necessarily mean job losses. Teaching machines how to question may be the next phase of AI. “It’s still very basic and we still don’t have enough people in the industry with experience yet,” says Streeter.

Data scientists who know the different types of data available, how to do the analytics to find these answers, and how to find the questions that need to be asked, are in demand. Balancing data with regulatory needs is an industry challenge, says Streeter.

This is giving a new dimension to clinical healthcare. Some administrative and tech jobs that AI replicates will need to be retooled. But, there are new trade needs, covering the ethics of AI, data science, and teaching machines. Companies are looking for data scientists to help find questions and answers. The industry must find people to bring structured and unstructured data together to find the answers they need. “There aren’t too many scientists yet in the field,” says Streeter. “We know if we bring different kinds of data together, we can find answers.”

And that is the purpose of examining all this data: to find new drugs, new cures, and new hope. “We want to help doctors already overloaded with assessments and streamline a more accurate picture of cognitive health,” says Rudzicz. “We want to make the world a better place.”

Editor’s Note: Artificial intelligence and machine learning are increasingly making their way into clinical trials. The technologies are the subject of Cambridge Healthtech Institute’s Artificial Intelligence and Machine Learning in Clinical Research conference, part of 9th Annual SCOPE Summit held in Orlando, Florida, February 12-15, 2018. For more information visit

Editor's Note: Changes were made on 10/6/2017 to better explain Deep 6 AI's offering. An earlier version suggested that Brusselaers was doing research, instead of representing a tool.