INTRO   00:06

Welcome to this episode of the Medical Affairs Professional Society podcast: “Elevate”. The views expressed in this recording are those of the individuals and do not necessarily reflect on the opinions of MAPS nor the companies with which they are affiliated. This presentation is for informational purposes only, and is not intended as legal or regulatory advice. And now for today’s “Elevate” episode.

Jennifer Riggins   00:33

Welcome to the Medical Affairs Professional Society’s Podcast Series: “Digital First Scientific Communications – How Digital is Transforming Medical Communications”. In these podcasts, we speak with experts in the field of scientific and medical communications and discuss how digital transformation is opening opportunities for medical communicators. I’m Jennifer Riggins, a co-host of this podcast. I currently serve as a member of the Digital Focus Area Working Group. I’ve worked in pharma for 30 years with a focus on medical information communications and medical digital, and I currently work for factum, I and nonprofit consortium of medical information leaders. I’m joined by my co host D Casey of Omni healthcare communications or Omni HC for short, a leader in digital first medical communications. So this is episode three of our series that’s focused on omni channel communications. We embarked on this project to interview experts on omni channel and determine the consistencies discrepancies, challenges and then unknowns from several points of view. We interviewed representatives from several partners circle organizations at the map Smashville meeting, we’ve discussed omni channel and how they define it, implement it and measure it. So we’re tackling one of these areas of omni channel per episode. We have curated all the responses to each question we asked to allow you the listener to have full transparency. Then throughout the podcast, Steve and I will summarize and discuss some of the interesting things we’ve heard. So in this way, you know, we hope to help you learn more about omni channel and how best to use it.

Steve Casey   02:06

In our last two podcasts we discussed the definition of omni channel as well as implementing omni channel and Medical Affairs. If you haven’t had a chance to listen to those two podcasts, we’d highly recommend it for today’s podcast with our interviewees. We tackled the question of how does a Medical Affairs organization measure the success or value of omni channel activities? To begin, let’s hear from Sandeep Gantotti, Associate Vice President at Indegene, followed by Martin Callaghan, Director of Innovation at Pharmagenesis.

Sandeep Gantotti   02:39

Sure. So this is, this is the final most important question, I think, all this effort, what is it actually resulting right? Are we getting any benefit of value? I think that whatever we do, in terms of defining value, or success needs to be tied back to that clarity of purpose, which I talked about earlier. So if if the goal of your Omni channel engagement is to drive, let’s say, engagement in a virtual program or CME program, then that’s what you should be measuring, right? There’s no magic KPI or number index, or anything that sort of nicely captures everything, right? So we really feel that as unique as our customers are, I think that the measures of success also have to be unique and customized towards that. We use the OKR framework, which is objectives and key results framework, which is used across many industries. Basically, it defines an objective, and then a key result, which tells you whether you’re doing well in that, right. So we feel like that’s a good way to objectively measure whether you’re going in the right direction. So, in our view, again, in line with the comment I made earlier, that there’s no magical numbers scale or anything like that, we really see the measures of value and successes, leading and lagging indicators. Or leading indicators are kind of can imagine very obvious things that we all measure, we do digital engagement. So how many times is a website being viewed, how many times something being downloaded, registrations time spent? Those are kind of easy, quick things that tell you that number one, people are actually engaging with you. Right. So that’s, that’s the first hurdle we need to cross. But I think the lagging indicators are the ones that actually matter when he talks about value, so if it’s Medical Affairs, our goal or objective is to for example, drive better clinical practice, or enable access to medicine. That takes a long time to do we measure those typically through cross functional feedback. So we’re not going doesn’t necessarily get a measure every day, it may take six months or a year or a year and a half. So we continually work with our Medical Affairs stakeholders and colleagues as well as cross functional colleagues to see if we’re moving that needle on those objectives that we’ve set. That’s in a nutshell how we we measure success. And

Martin Callaghan   05:24

I think there’s two ways. First of all, it’s whether they meet the strategic goals are intended for whatever the use case is, whether that’s a behavior change, educational shift, being able to connect your congress activities, and your cycle has been able to do medical education better. And secondly, I think it’s all about how effective and efficient it means you as a brand or team or franchise division or entire organization has, that’s the end goal. It’s meant to be a way of doing what you do better over time.

Jennifer Riggins   05:52

So, Sandeep, and Martin had just talked about tying measurement to the clarity of purpose and the strategic goals, and that there is no magic KPI. So what do you think about that, Steve?

Steve Casey   06:04

Well, I fully agree with them. There is no magic KPI. But before I go further, let me first describe how I see KPIs and other metrics. To me metrics are part of a continuum of information. It’s usually a snapshot in time. I like to say that metrics is an indicator and that you’re on target moving towards your objective. And the objective is what we’ve what we’ve been determining as part of the necessary tasks that must be completed to get to the goal we’re trying to achieve. Because of how early in the commercialization process and Medical Affairs add that adds value. I think it’s rare that we’re going to actually be able to measure if we’re able to achieve that final goal. And on top of that, with all the other efforts that will occur on top of the Medical Affairs effort, it’s really hard to discern what effect medical had on achieving that goal. That said, I do think with metrics we can identify if omni channel or any Medical Affairs effort is moving in the right direction to achieve the objective, which should help the organization achieve the the end goal. As we discussed in earlier episodes, I think customization is a key principle of omni channel. And when measuring omni channel, I think it’ll depend. As Sandeep also said, what the objective of the omni channel effort is, I think there’s a need for measurement. But it has to be customized and therefore, each KPI will be different depending on the objectives and capabilities of the organization.

Jennifer Riggins   07:38

Definitely, you know, I really also liked some deeps thoughts around leading and lagging indicators. Knowing if you’re providing value or actually changing behavior takes a long time. We need to be patient and know that with many of our measures of value and impact, it takes a long time it’ll take a while to move that needle.

Steve Casey   08:01

As you know, Jen, we asked the interviewees about a link between omni channel and behavior change. Although we decided not to use those responses in this series, a quick summary for the audience. Everyone feels like there’s a link, but no one can actually point to evidence which shows the link, we may want to share those answers with the audience at some point. But for now, let’s just say I, I agree that there’s likely some connection between omni channel and behavior change. But yes, there’s a big butt. And I say that in air quotes, I believe we need to be cautious about how much we rely on omni channel to empower that desired behavior change. But let’s move on. Let’s listen to Tony Gottschalk, Senior Principal at IQVIA and Lars Bauerle, SVP and Product Manager for Definitive Healthcare. Tony and Lars focus more on using a mix of methods for measurement and on learning from that information that you collect how to change behavior.

Tony Gottschalk   09:00

Measurement can be a challenge, because as we talked about before, you’re trying to measure behavior change, you’re trying to measure changes in patient outcomes, which are always challenging to measure in any in any setting. But we advise clients to develop a core set of key performance indicators that they can track. Start by tracking simple things, engagements, number of engagements before you start to get into content. And once that content received well by the physician, did it trigger a behavior change? Those are sometimes challenging KPIs to track, but we do it through a variety of methods. We want to sometimes ask the physician what do they think about the engagement? What do they think about their medical science liaison? What do they think about a company’s scientific integrity or accuracy compared to other companies that they interact with? So some of it we capture through survey information from physicians To understand what’s important and what’s impactful to them. Other metrics are captured passively by monitoring and observing or listening to what physicians are doing online, reporting that back to the company. And it’s really a mix of methods, I think is is the best way to get to meaningful key performance indicators to track how you’re doing an omni channel.

Lars Bauerle   10:26

Yeah, I think we are Definitive Healthcare that we provide a lot of data, you know, round KOLs, round prescription or, or x data claims data and such. And that that can really be used, actually, I think, in seeing whether you were only channeled efforts have the effect that you hope for, right, so if this really works out, well, you can go in and look into the prescription data and see if you know the people that you’ve been working with all day starting to prescribe more of your drug or not, or even in some of the other claims data around the diagnosis and procedural information that exists there, right. Maybe even referral data can inform you a bit about the effect of your Omni channel efforts. So I think we can provide a lot of sort of real real data there on seeing the impact of what you have done. Right. So it’s not just about measuring, let’s say, if people are reading your documentation or clicking on the links and things like that, right, but are they actually learning from that information and taking it on and actually changing their behavior? So I think we can, we can help a lot with that. That’s a very easy thing to do and read, at least within the US, right? That’s where we have access to that type of information. We also provide social media related information, right? Do what do people say on Twitter? Or, you know, do they participate in videos on youtube put out by medical societies, for example. So that information to you can also look into and start to glean and see whether people are actually, you know, changing their behavior or taking on and talking about the things that that you have been trying to put out there. So I think we can provide lots of good, measurable information in those areas, showing you what’s going on and what your activities actually have resulted in.

Jennifer Riggins   12:36

What do you think, Steve? You know, how important is it to use a mix of methods to understand your customer and to develop a core set of measurements?

Steve Casey   12:44

I think the mixed methods approach to measurement is just the very beginning of our metrics journey and communications. As you know, I’m very interested in metrics and using them as a feedback loop to inform and gain insight about a process. In fact, my group is a big proponent of building feedback loops into publication and communication systems in order to really optimize the scientific platform and publication plan. With an omni channel efforts similar to other areas of Medical Affairs communication, I think that we need to begin to establish levels of communication measurement that are clearly defined. The old gold standard Impact Factor really doesn’t give us any type of real useful information for omni channel. Using a mix of methods gives us more information, but it normally calls for customization by project. This really leads to an inability to compare and contrast the data for you know, if we’re going to feed that back to improve the process or system, I believe we need something more along the lines of like Moore’s model for outcomes reporting that’s used in independent medical education. In the Moore’s model, they defined five different levels that are distinct from level one participation to level five performance. Each level shows a higher behavior change effect, by the ISP program. So if we develop something of this nature, we can easily begin to rank behavior change outcomes for communications, much likely due for AI me giving us a much better understanding of the effect of our effort, and then we can use that to feedback and help improve our process and work product.

Jennifer Riggins   14:26

Mm hmm. So it sounds like you need to put together a working group to define a new model for behavior change and communications measurement. You know, I can see it now the KC model for behavior change measurement.

Steve Casey   14:41

Thanks, but I can’t take that credit. I tend to fall to behavior science methods to empower behavior change. But let’s move on and listen to a couple more opinions from our interviewees: Gary Lyons, Head of Strategy and Excellence at Fishawack and Dylan Benton, Vice President of Digital Strategy and Engagement at Symbiotix, to provide their views and measuring success.

Gary Lyons   15:03

So that’s a really, that’s a really good question. And it can often come down to the type of activity that was taking place, really. So what I mean by that is the type of channel being used, each channel will have its own types of metrics. So we’re thinking about how we visually measure success, it will come down to measuring success of individual channels, and maybe the program of activity as a whole. So when we think of various qualitative and quantitative measures of success, it’s really important when thinking about omni channel planning to have a really clear view of the KPIs at the start. So you talk about the types of goals and objectives that are coming into play, and the science objectives that we’re sorry, the types of KPIs that will be behind those objectives. So upfront, you’ve got the KPIs defined, and so you’re able to measure them at the back end. And for some activities you can do, for example, from a qualitative perspective, you can survey, maybe a range of end users or range of HCPs, you can get some feelings from them on how the activity prospectively, you know, if things went well would impact their behavior, or their approach to clinical practice. And then you can ask the same question six months later, the activity you can do on a small cluster, a representative sample of HCPs, maybe from different markets and different functions. And so you’re able to see the shift from the outset to where their expectation has been met at the end, or how it’s impacted their clinical practice, as I say at the end, that the the type of survey could be based around Net Promoter Score, concept, so you’re able to actually get a validated scoring. And with NPS, as you probably know, you want to get something like a 30% plus feedback on NPS typically for for an activity to be deemed to be good, you know, and what we want, but with NPS is really, it’s gauging the level of recommendation, isn’t it? It’s the peer endorsements, you know, how did you find this activity? Would you recommend it to appear? So you can you can blend that into the generalized question about how much of this activity change your clinical practice. So you think about things like that. Obviously, there’s lots of infield metrics as well in terms of feedback from MSL teams, feedback from local medical teams feedback from global global teams, their direct feedback on the actual activity undertaken is really critical. That’s outside of the customer feedback, of course, which we just talked about a second ago. So I think both internal and external lenses on the activity measurement is really important. As I say, the different channel metrics will come into play, you know, it’s medical portal, it’s how much how many return visits have been having downloads of content, that kind of thing. But one critical metric that’s not really often judged is experience, you know, what kind of experience is the end customer receiving from, from that activity? Aside from Will the activity change their behavior or like other opinion around clinical matters? But have they actually enjoyed the experience? Because as we were saying before, when somebody enjoys doing something, they’re more likely to engage with the activity again, they’re more likely to have an enhanced impression of the company offering activity. So measuring experience in terms of how did this make you feel? What’s interesting, did it engage you? And so there’s different levels of engagement. You can look at, you know, how long did they view a video four? Did they go through the whole length of the 15 minute video? Or did they drop off? After 15 seconds? You can do that with web pages as well. Did they? Did they read the whole web page? Or they moved to another page? How much interaction that they have with the MSL? Was it a deep interaction? Was it two minutes? Was it five minutes? So there’s lots of different surrogate measures for measuring experience, but I think experience is a critical and underplayed metric for measuring the impact of programs. So I think those are some of the key things that I think were the standout.

Dylan Benton   18:48

You know, because Medical Affairs is committed to providing rich scientific communication, it’s always hard to measure success within Medical Affairs, it’s not exactly as easy as measuring scriptlet in the commercial, right. But as basic as it sounds, standard digital metrics provide at least foundational insights into whether certain platforms or content topics are being engaged with or not, you know, even understanding at a high level where the audience is engaging, will help inform and allocate resources to drive more visibility and engagement across the road audience segments, but to specifically answer your question, things like confident confidence based learning modules, and surveys really help us understand at least how we’re moving the needle from a knowledge and understanding perspective, but also from a confidence perspective as well. And you know, these can be event based, but also can be leveraged on an ongoing basis to sort of periodically check in on this. And then the last thing to mention is, you know, relying on experiences and feedback within ourselves and medical directors, it’s a bit anecdotal, but it’s, it’s really key to providing insights to what they hear directly from stakeholders. You know, channeling these insights into strategic considerations is really vital.

Jennifer Riggins   19:50

So it’s clear from Gary and Dylan that the feedback from the MSL is critically important as well as gathering both quantitative and qualitative input. You know, and additionally, Gary, he brings in the importance of knowing what type of experience your customers receiving, have they enjoyed the experience? How do they feel, and they engage with the content? I think this is so important, you know, and it can be easily missed and all of the collecting of numbers. So bottom line for me is did the customer get what they wanted in a user friendly and easy way that will bring them back the next time?

Steve Casey   20:27

Yeah, Jen, gathering these insights is key to better defining how the omni channel effort is working. They’re they’re really key across the organization, the challenge is surfacing these insights through a qualitative effort, maybe through MSLs, they can but it can be incredibly expensive if you do it that way. But if done right, and using the right technology stack, you can gain tremendous advantage from understanding the customer’s experience and the behavior change that that should ensue from them.

Jennifer Riggins   21:01

Definitely. So let’s listen to our last segment from Charlotte Moseley, Executive Director at IPG Health Medical Communications.

Charlotte Moseley   21:09

I think the value of omni channel actually is inherent just by actually the the channels that will be used for omni channel, it’s actually there’s going to be more metrics that are going to be available than kind of using the methods that we use today, arguably, however, the the age old discussion around the value of Medical Affairs that’s still in play, that doesn’t change just because you bring in omni channel. So and we mentioned behavior change earlier, that’s a big thing to look at in terms of measuring success, if you can change the behavior, and ultimately then change an outcome. That’s obviously what you’re looking for, particularly from a medical perspective. So I think the first thing is probably the one of the things to measure is certainly the impact of the behavior change. So can you get a customer from A to B, so knowing what your A to B is, is first thing to start with in terms of measuring the success, but also how you use all the information that you’re going to use for omni channel. So I think the value is ultimately the change in the behavior. But you’ve got a lot of that back in terms of, there are 1000 metrics that if you do omni channel, well, then you’re collecting a huge amount of information. That’s the whole kind of definition of omni channel, you adapt the content and what you’re providing to your customers. So by doing that, you will actually be creating a huge amount of data just by default. So it’s making sure you understand that data, but not just understanding that data is knowing what that data means. And ultimately, having a goal in mind at the outset. So it’s great developing, we’re having great content. But if you don’t know what the change you’re looking for is then just kind of, you know, there’s no point in pushing that out. So I think there’s going to be huge amount of metrics, because a lot of omni channel, sort of, by definition, to a certain extent is going to be via digital rights. But that’s not the only channels, you know, the still MSLs there’s still a lot of face to face interaction. All of that is still you know, a channel, and how do you use those insights? So again, back to data stack, kind of AI content, you need to pull that together do to find what your metrics are, your AI will certainly help that because that can then hopefully assimilate all the insights, which then will help feed into how do you measure certain metrics, I don’t think there are a predefined set of metrics here at all, you know, every therapeutic category who different, every customer will be different. But ultimately, you’ve got sort of collate that in some way. But I think, to say, the use of data, the use of data analytics, the use of AI, will actually help to start measure the true value of Medical Affairs, which I think hopefully may solve the sort of 10 Year 15 year old discussion about what is the value of Medical Affairs? So no golden bullet, no great nugget in terms of the true metrics, because I think it you know, it’s a very complex picture. But I think omni channel will certainly help define further the true value of Medical Affairs.

Steve Casey   24:25

You know, I agree with Charlotte that using Omni channel should give us more metrics to measure the effect of the effort, but we have to define what objectives the metrics are indicating movement toward, I’d have to say that what I found most interesting in Charlotte’s discussion is a reference to behavior change, as I was discussing before, to me a change in behavior is the most important measure that Medical Affairs could gather. I’m not saying that the other metrics are not important, but they’re just indicators that we can use to point to an anticipated change in behavior. If we can Your behavior change or a series of metrics that point to behavior change, then we can show the value of the Medical Affairs work product, and also optimize our processes to maximize those communication and the effectiveness of the communications.

Jennifer Riggins   25:15

So, good point. Okay, so, unfortunately, Steve, I think it’s time for us to wrap up. For me, there are three key takeaways from today’s discussion. Number one, there is no magic KPI. We have to really think about meaningful leading and lagging indicators that lead to behavior change. Number two, we need to use a mix of methods to measure value and impact. And then number three, to be successful customer experience matters. You agree with these?

Steve Casey   25:45

Yeah, Jen, I think your key takeaways are right on target. I have nothing to pontificate on.

Jennifer Riggins   25:51

Awesome. So we want to thank our interviewees and their companies for giving their thoughts on how a Medical Affairs organization can measure the success and value of omni channel initiatives. As you’ve heard and likely know from personal experience, this is so hard to do. There are no predefined sets of measurements. Your measures should be customized to your KPIs and sit include both leading and lagging indicators for the best results. So we hope you’ve enjoyed our little mini series on omni channel. We have a few overall takeaways from the series which include one make sure you are lined on your approach cross functionally, this is so important and so critical to the success of omni channel to each organization needs to have a customized approach to omni channel implementation within a defined framework that works for their specific organization. And then three measurement of success and impact should include a mix of quantitative and qualitative measures. So Steven, I wish you the best on your Omni channel journey. And you know, and I’m going to leave you with one final tip. Remember, Omni channel cannot fix bad content. Thank you for joining us today and listening to our podcast series digital first scientific communications, a podcast production of the Digital Focus Area Working Group of the Medical Affairs Professional Society.

Steve Casey   27:21

If you’re a MAPS member, thank you for your support. If you’re not yet a MAPS member, and I tell you this all the time, I want to encourage you to join so you can access additional resources visit the MAPS website today at MedicalAffairs.org/membership.