Linn Southall   00:00

Welcome to the Medical Affairs Professional Society’s three-part podcast series on how to prioritize, track and measure your medical communications plan. In this third podcast episode, we will discuss measuring and improving the reach and engagement of scientific publications. I am Linn Southall. I will be the moderator for this podcast. I am a Medical Communications Director at Novartis Gene Therapies based in Zurich, Switzerland, and I currently serve as a member of the Medical Communications Focus Area Working Group. The views expressed in this recording are those of the individuals and do not necessarily reflect on the opinions of MAPS or the companies with which they are affiliated. This presentation is for informational purposes only and not intended for legal or regulatory advice. This podcast is for Medical Affairs professionals asking how do I know that my Medical Affairs programs have an impact beyond just counting outputs, and what can be done? I would like to say welcome and a big thank you to today’s panelists for sharing their subject matter expertise with the MAPS members. Before we dive into the world of metrics, I will let the speakers briefly introduce themselves and their background and the industry. Ladies first, Louise, please go ahead.

Louise Ostergaard   01:22

Thank you so much, Linn. I’m very happy to be here today. My name is Louise Ostergaard and I’m a Global Publications Director with Novo Nordisk working out of Zurich, Switzerland. I’ve been a publications professional for almost eight years now. And prior to that, I worked as a group leader at the University of Zurich.

Linn Southall   01:40

Avishek?

Avishek Pal   01:41

Hi. Hi, Linn. Thank you. Thanks for having me. Hi, everybody. My name is Avishek. I’m a microbiologist by training, and I’ve been in Med Comms for 15 years now. And in my current role, I lead Global Publications and external Medical Education and Events for Cell and Gene Therapies

01:57

at Novartis Oncology.

Linn Southall   02:00

And last but not least, Darren.

Darren Fleetwood   02:04

Thanks, Linn. Hi, everyone. I’m Darren, I’m a Research Director at Madano. My backgrounds is in evaluation and Social Research, and I lead a lot of our publications measurement work at Madano.

Linn Southall   02:16

Thank you, everyone. So as we heard in previous episodes, we’re sensing a shift within Medical Affairs to make the case for and insist on better measurements. But how do we actually make that a reality? Measuring what really matters is often not straightforward, sometimes expensive, and always takes committed upfront thinking and planning. Today, we will look at novel and cutting edge approaches to measurement of publications. In the past episode, we looked at the critical importance of setting clear objectives and selecting measurements that capture the expected outcomes. Now, how can we apply this to publications? If I first go to average check, what are your thoughts around measuring the impact of publications? With what outcome in mind should we measure? And what is the value in investing in measurements beyond impact factors?

Avishek Pal   03:13

You know, this is such an interesting question limb, you know, each publication in a publication plan should have a predefined role to play. Now, it could be targeting a specific audience or you know, with the focus scientific narrative, you know, it has to be timed to perfection or the Congress, you know, orient journal, but how do we measure whether the really meeting the objectives that they were meant to achieve for the team in terms of supporting see a strategic imperative or a medical focus area? How do we know whether the data reported is changing the practice of medicine? This is why we need to measure the value of publication. And this needs to be done in a more holistic way, rather than simply focusing on something so traditional, like generally back factors, Louise, what do you think?

Louise Ostergaard   04:03

I fully agree with you, Avishek, like, these old measures, such as impact factors, a number of references are really not so relevant anymore, or at least they only tell part of the story, right? So in my opinion, I think we should instead begin to consider what it is that we wish to communicate, and to whom, and then we find ways of evaluating if that was indeed achieved. So an important point is also to consider the multichannel aspect of our publication plans, or actually, I should rather say our communication plans, since we as publication professionals have certainly moved on from just getting data presented at conferences and published in scientific journals. And this should obviously also be reflected in how impact is measured.

Darren Fleetwood   04:48

Yeah, absolutely. And, you know, as you say, Louise, what we measure needs to reflect what it is we want to communicate and to whom I think, you know, my company has changed so much over the last 510 years. seniors with new channels, but also new audiences as well, because there’s just so much to get missed if we’re just relying on things like impact factors or citations.

Linn Southall   05:12

So Darren, what are other relevant aspects that we should take into consideration?

Darren Fleetwood   05:19

So things like engagement among patients, healthcare professionals, policymakers, and so on. They’re just not reflected in number of citations. So we need a set of measures that are better focused on these audiences, I think the other thing we need our measurement to be able to do is to act off the back of what we’re measuring. And you know, citations just take so long to peak, you know, at what point this result, it’s often too late to do something to change the course of the paper. So this is where I think some of the newer measures, so things like social media engagement can actually be quite useful as a as an early read.

Linn Southall   05:56

So we’re trying to measure beyond the Impact Factor, what are the key challenges that we face?

Avishek Pal   06:03

When I think we are still so deeply entrenched in tradition? Plus, you know, there’s always there’s a perception that there’s no suitable, reliable, intuitively interpretable alternative. And then this continues to bind us to journal impact factors, you know, the proposed alternatives like Ultimatrix, plum, and a few others, you know, they’re often very skewed towards very specific individual high volume metrics, which are quick to your group, such as social media, which makes them a little bit difficult to interpret in my team. Darren, what do you think?

Darren Fleetwood   06:39

Yes, I completely agree. And I think, you know, that taps into this central problem that we have that, you know, as we’ve discussed, measurement needs to reflect our objectives, but we often can’t directly measure what it is we’re trying to achieve. So, you know, there’s no feasible way of directly capturing if or how papers influence someone’s attitudes or behavior, for example, at least not in the real world. So I think we’re often rely on pulling together the best proxies we can. And rather than necessarily measuring the change itself, we’re measuring more has a paper create the conditions for change?

Linn Southall   07:19

So if we want to start measuring more effectively, then how can we overcome these challenges?

Louise Ostergaard   07:25

Well, I think as we’ve already discussed, it should definitely begin with a very clear strategy, plus a detailed tactical plan of what kind of activities are needed to achieve our objectives. So that means, first of all, defining which audiences we wish to reach and how this is best accomplished. So in addition to a clear plan of where the data should be presented in terms of Congresses and journals, we definitely also now need to consider, for instance, what type of publication extenders we want to develop. And furthermore, I think it’s an it’s extremely important that we also make sure that our internal colleagues are aware of the available materials, and how they can best use them. And then at the end of it, we can combine all outputs such as views, downloads via QR codes, the use of Extenders by affiliates, or different meetings, the extent of media mentions as sterren also spoke to plus, of course, the more traditional metrics to really define this realistic overview of how we were able to communicate the data and ultimately meet our scientific objectives.

Darren Fleetwood   08:30

And actually, one thing we found quite helpful with that is to borrow some techniques from evaluation practice. So things like theories, change outcomes, frameworks, these kinds of things. So they all start by considering what is it we ultimately want to achieve. So things like improving standards of care within a particular disease area, for example, and then working backwards from there to identify the steps that need that are needed for this to happen. So for example, a paper needs to be published key messages need to reach the right audience, they need to be convinced by them and so on. Having done this, we can then find the best measures for each of these steps along the way. And that makes sure that what it is we’re measuring is always tied to do what type back to our objectives.

Linn Southall   09:16

That’s really interesting. Do either of you have any more specific examples of this? How this could be done?

Avishek Pal   09:26

Yeah, let’s we do actually, you know, we did a pilot in one of my previous roles, and we actually proposed a holistic, multi-channel measurement framework. We call it the Empire index, you know, just just to be just for fun. We present it That sounds to us is my meetings. This framework uses article level attention metrics across these various traditional and newer channels. You know, for example, journals, social media guidelines, pattern mentions everything, and then it recalibrates them. Based on some predefined waiting for those respective channels, and then it gives us some impact scores across three smaller group of channels, you know, social, scholarly, and societal. The idea is to provide insights to publication planners on what channels to focus on as a normalized approach that can be embedded into future publication planning strategies. One of my favorite case studies is a faceful study publication, which, you know, per our empire index, generate it’s so much more holistic impact, versus some of our key pivotal phase three study publications, purely because they engage in a multi channel format, rather than just sticking to the traditional aspects. Louise, any other ideas?

Louise Ostergaard   10:53

Yeah, absolutely. I think those are excellent points, Avishek, at Novo Nordisk, we’ve also developed a similar tools, we call it beam. And the idea is that this tool should help us on one hand to put a number of match in the same way, as you just spoke to on how well our publications are performing, taking into account both traditional as well as the more novel metrics. And on the other hand, it should also guide us as to where the gaps in our communication are, and what we can improve on. So this will in the future be part of our Medical Affairs insights generation. And it should really help us both evaluate projects that are already done, but definitely also guide future strategies.

Linn Southall   11:34

So these tools are standardized tools for measuring publications. That sounds fantastic. Does that also give us an opportunity to compare publications then? Or is there a way to benchmark also across specialties?

Avishek Pal   11:52

So it’s a million dollar question. It is extremely challenging to identify, you know, what, what could be an universal benchmark or score? I think everybody looks for one. But you know, whether it’s a journal impact or otherwise, you know, what qualifies as a good score, that can be applied irrespective of, you know, the variability across disease areas, the product life cycles, unfortunately, it’s quite tough. However, as an alternative, the aim should be, in my opinion, to assess whether the objective of the publication was met, rather than trying to make any broad comparisons across the board. What are your thoughts?

Louise Ostergaard   12:32

Yeah, I really agree with you here it this is the key challenge, right, that so many of us have been trying to solve, what does good look like? And could we put a number to it, because then it’s so much easier to communicate to our stakeholders. And I also agree with you, I think one part of the solution is to find a relevant way of comparing your set of publications to a set of similar publications. So I, for instance, work with rare diseases. So for me, it really wouldn’t make sense to start comparing with the interest generated by a publication reporting results from a major cancer or diabetes trial, for instance. So instead, I should rather define a set of comparable publications. And this could perhaps be from other companies that are working in the same field or or within a similar area. And again, I think it’s very vital that we are clear of our objectives. Are we benchmarking directly to competitors? Are we establishing ourselves in a new therapy area? Are there any other relevant parameters that we need to consider?

Linn Southall   13:33

Thank you, Louise, and thank you to Avishek and Darren there and as well for sharing your expertise and insights with our listeners today. That was a really interesting conversation, I learned a lot. We have spoken today about the challenges of measuring publications beyond impact factor, but also about the opportunities when we overcome them. So the takeaways that I take away from the episode today is that as with anything we want to measure, the most important step is to do fine expected outcomes, and performance indicators upfront and to prioritize what is important for our specific organization. By thinking beyond the publication itself, and incorporating a multi channel approach into the communications plan and update metrics accordingly, we can achieve a higher engagement and impact. So thanks again to all speakers. This was the third and last podcast in the series on the topic of meaningful metrics. We encourage all listeners to engage in conversations about medical communications with other MAPS members on the MAPS Connect on the MAPS website or mobile app. Simply log in with your email address and password so sated with your MAPS account, and access the global community. Then click on the discuss tab and scroll down to medical communications to post a question or review previous postings. If you’re a MAPS member, thank you for your support of MAPS. If you’re not yet a MAPS member, we will end would like to access the additional resources in this area. Please visit the MAPS website to explore joining today at MedicalAffairs.org/membership. This concludes the podcast.