00;00;05;04  

MAPS 

Oh. Welcome to this episode of the Medical Affairs Professional Society podcast “Elevate”. The views expressed in this recording are those of the individuals, and do not necessarily reflect on the opinions of MAPS or the companies with which they are affiliated. This presentation is for informational purposes only and is not intended as legal or regulatory advice. And now for today’s “Elevate” episode. 

00;00;33;09  

Adeola Davis 

Welcome to episode one of the SciTech Critical Evaluation of Literature and AI series. Evidence unlocked. Separating gold from gravel. I’m Adeola Davis, Senior Director of R&D, Learning and Development at a global pharmaceutical company. And I would like to introduce my colleagues who will be the experts on this session, Sarah, would you like to introduce yourself? 

00;00;56;03 

Sarah Snyder 

Hey Adeola. Great to talk to you and Ruth today. I’m Sarah from MSL Mastery and I’ll pass it over to Ruth. 

00;01;04;05 

Ruth Nicholson 

Thanks, Sarah. I am Ruth Nicholson. I’m a Global Medical Excellence and Medical Affairs Leader. Somebody very excited to be speaking to you both on this subject today. 

00;01;15;03 

Adeola Davis 

All right. Fantastic. So again, welcome to Evidence Unlocked, where we help Medical Affairs professionals separate gold from gravel because not every forest plot deserves a standing ovation. And again, I’m joined by Ruth and Sarah. So let’s start with the basics, but with a useful kind. So we hear hierarchy of evidence all the time. Ruth, what is it really? 

00;01;35;08 

Ruth Nicholson 

Well, at its simplest, the hierarchy of evidence ranks study designs based on how confident we can be that the results are true and unbiased. 

00;01;45;01 

Sarah Snyder 

Think about it as, at the very top, you’ve got your systematic reviews and your meta analyzes of those high quality randomized controlled trials or RCTs. Right below that, you got those RCTs. And then followed by that, number three would be the observational studies, your cohort or your case control studies. The last tier would be real world data so your case series or expert opinion. 

00;02;19;00 

Adeola Davis 

And what about preprints? 

00;02;21;08 

Ruth Nicholson 

So preprints aren’t really a level. They’re a status. And anything, any study type can be a preprints, a preprint, hasn’t yet passed peer review. 

00;02;32;10 

Adeola Davis 

And you know, Sarah, would you say is higher or the hierarchy always better. 

00;02;37;16 

Sarah Snyder 

I used to think so, Adeola, but not always. So those RCTs answer efficacy under the ideal conditions. So those pivotal clinical trials where everything is set up exactly the way that they want it to be, but observational and real world data actually tells us effectiveness in real life. So the key to remember is that in Medical Affairs we need all types, but we’ve got to be able to interpret them correctly. 

00;03;11;07 

Adeola Davis 

No. Very interesting. And so I say let’s let’s discuss frameworks. Let’s demystify them. So we hear things such as grade Prisma consort who does what. 

00;03;22;16 

Ruth Nicholson 

That’s a great question, Adeola. And we’re very lucky because Medical Affairs have a number of different frameworks, checklists available to really help us evaluate and decipher which studies are the gold and which studies need to be in the gravel box. So as you mentioned, there’s a number of very common ones used – CONSORT. You mentioned some CONSORT is the consolidated standards of reporting trials that’s used for randomized trials. It includes a really comprehensive 30 item checklist. There’s a flow diagram and it helps check whether the study is reported transparently. Check for randomization blinding endpoints, attrition. 

00;04;09;00 

Sarah Snyder 

Ruth, I love that you’re going to put trials in the gravel box. One example would be RCT doesn’t clearly explain how the patients were randomized. Or if you’re seeing unexplained dropouts, that’s going to be a red flag. So even if the results look positive, you got to go backwards and see some of those explanations. 

00;04;30;15 

Ruth Nicholson 

And then we also have Prisma. So Prisma is preferred reporting items for systematic reviews and meta analysis. And as the title suggests, applies to systematic reviews and meta analysis. Again, it includes a comprehensive item checklist, 27 items, this time a flow diagram. And it asks in a was the search undertaken comprehensively where studies excluded appropriately? Or are they cherry picking what data is included here? 

00;05;01;14 

Adeola Davis 

So if I’m hearing you correctly, Ruth, you would say that Prisma helps us avoid being impressed by a meta analysis that really just a selective pooling. 

00;05;09;29 

Sarah Snyder 

Oh, you got it. That’s exactly right. 

00;05;12;14 

Ruth Nicholson 

then Adeola you also mentioned grades. So grade goes a step further. So grades is, a framework for the grading recommendations, assessments, development and evaluation of data. And it rates a certainty of the evidence as high, moderate, low or very low. Sarah have you used grades. 

00;05;32;10 

Sarah Snyder 

Yes. And most Medical Affairs professionals I speak with use grade a lot internally. The reason that they justify the use of grade is that it helps shift the conversation from this proves it to this suggested keeping in mind limitations. 

00;05;50;23 

Adeola Davis 

So we’ve we’ve talked about hierarchy. We’ve talk about frameworks. But what about real life application. Give us a practical example of why it’s important to consider the big picture of critical appraisal. 

00;06;02;16 

Sarah Snyder 

Okay. So let’s describe the RCA duplicate project. What happened there is that research took about 30 classic RCTs, those randomized controlled trials and then built real world analogs. And what they used to do that is health care claims data. They were trying to see if the results would be similar. In about half of cases they did match. But in others the real world data told a different story. So you might wonder why that was. And it’s mainly because the patients in the care delivered in routine practice weren’t the same as in trial setting. I don’t think that will surprise anyone in Medical Affairs, but it’s good to have that data that proves this. So this is a great, real world example of why we’ve got to look beyond just those RCT results and why earlier we were talking about in that pyramid, we need all those different types of data. 

00;07;02;22 

Ruth Nicholson 

So that’s a great example to use, Sarah, the RCT duplicate project is really fascinating. And if I can expand on what you said. So one of the trials looked at was for a COPD therapy. The, real world data hazard ratio and actually show disease exacerbation compared to the RCT. And and of course, as you see demonstrates how different, routine care and patient characteristics can really impact the overal picture as opposed to really tightly controlled RCT. 

00;07;38;07 

Adeola Davis 

So what’s your rapid ma or Medical Affairs appraisal checklist? 

00;07;43;11 

Ruth Nicholson 

So, Adeola, as we mentioned earlier, there’s lots of different frameworks, lots of checklists available. They could be tailored to different data types, publication types, but it can be quite overwhelming. But at MAPS. So we recognize how important the critical evaluation skills are for those working within Medical Affairs and MAPS have actually just published, a very easy to use, Rapid appraisal checklist that those in Medical Affairs can use when they are really trying to critically evaluate and analyze data. Now, this framework is called the Clear framework. And very simply, it focuses on evaluating the credibility of the data. It looks at the logic and design the strength of the evidence. It evaluates the applicability. But also most importantly, it looks at the relevance and impacts of the data being presented. So in short, if I had to do a very rapid appraisal, I’d always think of about  clear in my mind. And that’s what I would look at. 

00;08;50;06 

Adeola Davis 

Oh fantastic. So let’s go ahead and wrap this up. I’d like to hear one takeaway from each of you. 

00;08;57;15  

Ruth Nicholson 

Oh Adeola, I would say all that glitters may not be gold. Don’t be fooled by sexy titles. We’ve all been sucked in by a publication that’s promised us the world but ultimately disappointed us. So I’d say ensure you appropriately evaulate the literature before you come to any, any conclusions. 

00;09;20;17  

Sarah Snyder 

I love that, Ruth. I’ll add that using these frameworks doesn’t slow you down. It often seems easiest to just jump to those RCTs and assume that they’re the gold standard. But applying these frameworks is going to make you much more strategic and effective in the long run, and really keep you safe in the way that you analyze and use clinical data. 

00;09;44;16 

Adeola Davis 

Wonderful takeaway. Thank you both, Ruth and Sarah for spending time with us on this first episode. I think our next episode will be. Literature never sleeps, but stay tuned for that and we hope that you enjoy this podcast. We’re going to continue delving in to all the wonderful aspects, of critical evaluation of the literature. So stay tuned.