00;00;05;04 

MAPS 

 Welcome to this episode of the Medical Affairs Professional Society podcast “Elevate”. The views expressed in this recording are those of the individuals, and do not necessarily reflect on the opinions of MAPS or the companies with which they are affiliated. This presentation is for informational purposes only and is not intended as legal or regulatory advice. And now for today’s “Elevate” episode. 

00;00;33;17 

Ruth Nicholson 

Welcome to episode three of the Sci-Tech Critical Evaluation of Literature and AI series. Humans and the AI making sense of scientific papers at scale. I’m Ruth Nicholson. I am a Global Medical Excellence and Medical Affairs Leader, and I’m joined today by two wonderful experts. We have Sarah and Adeola, and Sarah, would you like to introduce yourself to everyone? 

00;00;56;11  

Sarah Snyder 

Absolutely. Thanks so much for the opportunity, Ruth. My name is Sarah Snyder and I am from MSL Mastery, and I’ll pass it over to Adeola. 

00;01;05;13 

Adeola Davis 

Ruth, happy to be here. I’m Adeola Davis, Senior Director of R&D, Learning and Development at a global pharmaceutical company. 

00;01;14;07 

Ruth Nicholson 

So I’m really excited to be discussing this topic with you both today. And we all have, I think, a really good sense in Medical Affairs of how we can use AI. We know it’s got a fantastic potential, and we know that we can use it to help us understand the papers, the publications, and to help us keep up. But how does it all actually work? Right? How does AI actually process scientific literature and what does that look like? So, Sarah, maybe you can tell us a little bit about how AI actually reads papers. 

00;01;51;13 

Sarah Snyder 

Thanks, Ruth. We’ll start with some basic definitions. So NLP and LLM are terms you might heard of related to AI. So that just means natural language processing in large language models. What AI actually does is it identifies patterns. And the key word there is patterns in language. So it’s not like us. It doesn’t understand the science. It’s just identifying the patterns. So Adeola, can you expand on that a bit? 

00;02;27;19 

Adeola Davis 

Yes. So you know as we think about that and because you had mentioned patterns, it predicts likely text based on training data but not necessarily always to truth. And that’s really where we have to keep the human element in mind as we’re using these platforms. 

00;02;44;12 

Ruth Nicholson 

Great so AI can be used to help us summarize, but we still need that human aspect of human oversight. So one question I have is some kind of examples in practice any tools in practice. So what tools do we have at our disposal to actually help us to summarize these papers and read papers and help us with our evaluations. 

00;03;08;19 

Adeola Davis 

Yeah. One of the tools that is commonly used is copilot. And it does work well for drafting summaries or slide outlines or even pulling through themes from information that you may ask it to look at from a meeting. 

00;03;21;23 

Sarah Snyder 

An additional tool and my personal favorite. If you’re allowed to use it, always make sure that you follow your individual companies policies. But notebook LM is a wonderful tool. And what you can do with notebook LM is use a defined set of information for example, a Congress abstract or a publication. You can put those directly in and it will create a summary. And it can also create a podcast or a video summary and highlight of that information. 

00;03;56;17 

Ruth Nicholson 

That sounds fantastic. But is it too good to be true? Are there risks that come with this? I’ve heard of hallucinations. 

00;04;05;29 

Sarah Snyder 

Hallucinations are a real risk. They are going down. The more that these models get improve for example, notebookLM is one where you’re going to have a low risk of hallucinations because you’re putting specific data in. But going back to your question, it is a real risk. We’ve got to be aware of that. Just like any data we would review in Medical Affairs. AI also will fabricate data if you don’t prompt it well. 

00;04;35;19 

Ruth Nicholson 

And that, of course, for those who don’t know, is the definition of an AI hallucination. 

00;04;40;26 

Sarah Snyder 

Yes. 

00;04;41;21 

Ruth Nicholson 

What about validation? Adeola? 

00;04;44;07 

Adeola Davis 

And add on to that validation is non-negotiable. I think as we mentioned in a previous podcast, we really need to make sure that we trace outputs back to a source text. I mean, just think about the scenario. A team was pulling together a quick literature review and use an AI tool to help summarize the data. Right. That’s something you may think about if you want to quickly get through, the data. As we mentioned, everyone’s being inundated with information and since the output in this scenario sounded confident and polished, no one double checks their original studies. And then later they realized that I had mixed up results and overstated claims that weren’t actually supported by the papers. As you could imagine, in this scenario, fixing the mistakes meant revisiting the entire review and delaying key decisions and really, you know, as we try to make sure that we’re pulling in AI responsibly in our everyday functions. This was really a clear reminder that AI can speed things up, but without validation, it can quietly steer you in the wrong direction. 

00;05;48;00 

Sarah Snyder 

Adeola that’s a really important example to share. One tip that I’ve heard from Medical Affairs experts using AI well, is that they will ask AI to reference and be very specific about where they obtained, where the AI not there, but where the AI obtained the specific piece of data. So what that means is that if it spits out something that it’s recommended to put on a slide, it will actually tell you the paragraph or the page from the trial that it was in. It makes it a lot easier on your end as well, so that you’re not having to go back and try to find where the AI came up with the information. So going back to Ruth’s point about hallucinations, you can quickly find those and minimize those. 

00;06;42;27 

Ruth Nicholson 

And any other top tips? Sarah or Adeola. That’s that you’ve heard in the past so that you use yourself in terms of trying to preempt or negate the risks that AI brings. 

00;06;58;18 

Adeola Davis 

Absolutely. I just want to build on what Sarah mentioned. As we create these prompts, and if it’s something that you do regularly, it’s okay to save them. Have a list of prompts that you use regularly that you have worked on, and leverage that for future inquiries into your copilot or whatever system that you’re using. And I think as you fine tune that and really create a script for AI, then you can feel more confident in what it provides back, providing the referencing and just seeing, in training the system essentially, I think can really help you use this technology responsibly. And Sarah, additional thoughts as well. 

00;07;39;21 

Sarah Snyder 

I’ll add on to your prompting technique there, adding Adeola, prompting the AI to only evaluate the information that you’ve given it will minimize the risk that it starts going out and trying to find additional information. That tool that we mentioned before, notebookLM, that’s only going to search the information that you upload. So you’re going to have a lower risk of problems because it’s only scanning that specific information. 

00;08;08;14 

Ruth Nicholson 

Wonderful. Thank you both so much on your thoughts and your comments on how we work with AI to make sense of scientific papers at scale. But before we close today, let’s leave everyone with some very clear takeaways. Sarah, what are your what are your thoughts? 

00;08;25;12 

Sarah Snyder 

First and foremost you got to try it out and you have to use it. And the more you use it, the more comfortable you’ll get with it. AI is excellent at two things. What? Many more than two things, but I’ll say speed and scale for today. It can scan, summarize, and synthesize literature way faster than any human team. 

00;08;48;08 

Adeola Davis 

Yeah. And I’d like to add AI does not understand clinical truth or relevance. It predicts language, not evidence. And that’s why clinical judgment remains non-negotiable. 

00;08;58;29 

Ruth Nicholson 

And building on that, responsible use means use of validation tools. So every output should be traceable back to source data, scope the question asked,  reviewed for context. And I think really by bearing all these points in mind, we can really ensure that we can credibly make use of fantastic AI tools that are now available, to really make sense of these scientific papers at scale. So thank you so much, Sarah and Adeola, for your thoughts today. That’s the end of our episode. Our next episode will be looking at smarter summaries, better decisions and AI across Medical Affairs.