Share this entry
Paul Tebbey, PhD, MBA
Leah Williams, MSc
Metrics in Medical Affairs
Though much has been written describing the evolving uses of metrics in MA,1,4-9 there remains little consensus on best practices across organizations. Results from a small pilot survey of MAPS members (n = 25) reinforces this idea of fragmented metrics use, with 44% of respondents reporting that they either do not use metrics or are unaware of the use of metrics in their MA teams. Additionally, 64% of survey respondents either do not develop metrics or are unaware of metrics at the initiation of a strategy or tactic, and only 40% of respondents categorize the organization’s current metrics as useful (Figure 1). The survey also asked MAPS members to report which metrics are currently used (Figure 2). Notably, current metrics tend to be more quantitative, focusing on the number or volume of actions taken, for example, the number of medical science liaison (MSL)/key opinion leader (KOL) interactions per quarter, without necessarily taking into account the impact of these actions. When metrics are poorly integrated across the function or organization and when these metrics have magnitude but no direction, MA teams run the risk of measurement without meaning. Consequently, there is a missed opportunity for MA teams to differentiate what has been working vs what has not and thus to better refine future strategies and initiatives.
What is a Meaningful Metric
A meaningful metric details how closely a tactic or initiative delivers on its purpose as described in the Medical Affairs Strategic Plan. This alignment between a tactic and the metric(s) used to assess it in turn provides actionable insights, helping teams identify and communicate the impact of successful actions to the MA or cross-functional product team, while also offering the opportunity to rethink and reprioritize efforts shown to be less successful. These metrics capture not only that something has been done but also its effect. For example, metrics associated with data generation or communication may inform regulatory, reimbursement, and global market access requirements.10 A meaningful metric goes beyond the one-dimensional scalar measurement of magnitude alone (eg, the number of symposium attendees) and reflects the multidimensional value of a vector measurement that contains direction as well as magnitude (eg, the pre- and post-symposium assessment of knowledge acquisition of the attendees or post-symposium follow-up/educational opportunities with MSLs). Thus, the more meaningful metric measures the educational progress of all attendees aligned to the current medical objectives and scientific messages. It has value in driving updated strategy or actions of the teams making up MA and also organizational value in demonstrating the value/impact of the MA function as a whole.
The Art of Meaningful Metrics
The successful use of metrics in MA can be seen as 2 related challenges: how to use metrics and which metrics to use. Answering the first challenge starts by defining metrics as part of the Strategic Plan, thus ensuring that the use of metrics remains aligned to the overall Medical strategy. Top-line metrics within the Medical strategy are those that apply across functional groups within the MA team, providing one foundational assessment for the team to determine whether they have met the strategic objectives outlined in the strategy. Of course, groups within MA will then utilize their own impact metrics at a tactical level to optimize work or refine plans. This allows all MA team members to work toward common objectives while personalizing the actions within their individual plans that help to achieve these objectives. These ongoing assessments and evaluations influence updates to the Strategic Plan such that the use of meaningful metrics supports the ongoing evolution of the plan as a living and breathing document. This concept of metrics as an element in a cyclical process of planning, implementation, and evaluation is important for any stage of a product life cycle but especially in the peri-launch period wherein the impact of medical strategy should be reexamined at frequent intervals to account for new data and changing treatment paradigms.10 Within this framework of use are the metrics themselves. The list of possible metrics is long and ever-expanding, especially in the digital age (Figure 4), highlighting the need not only to choose from the existing list but to understand the factors that make metrics meaningful within the context of a team’s specific strategies and tactical plans. With that goal in mind, this paper offers the following ideas for use in conceptualizing metrics to support a team’s individual needs.
Qualitative vs Quantitative
A central challenge in the use of metrics in MA is quantifying outcomes that are inherently qualitative, such as stakeholder education or customer awareness of product messages. Unfortunately, it can be difficult to calculate and communicate the success of initiatives measured with qualitative metrics (eg, the value of insights gained across customer channels). Historically, this problem was addressed by focusing on quantitative measurements as proxies for desired outcomes, for example, the number of KOL interactions as proxy for the success of a team’s external engagement strategy. Current strategies seek to both formalize qualitative metrics and quantify the impact of MA actions. This concept of impact measurement can be understood as looking a layer deeper than traditional quantitative metrics to ask not only “What was done?” but “What was the impact of what was done?”
Scalar vs Vector
Related to the previous discussion of qualitative vs quantitative metrics, scalar metrics show size, whereas vector metrics show both size and direction. For example, measuring the annual number of publications or KOL interactions are scalar metrics demonstrating the quantity of actions but not their impact. Capturing the impact of a publication (eg, through altmetrics) or the depth of KOL interactions (eg, through qualitative survey) could help transform these scalar metrics into vector metrics.
Organizational vs Strategic
Where does a metric sit in the hierarchy of the MA Strategic Plan? For example, top-line goals like increased adoption of a product might sit at the organizational level, while the different groups within MA including Publications, Field Medical, and Health Economics and Outcomes Research (HEOR) may each use strategic metrics to support adoption of that product through initiatives such as guideline inclusion, key data dissemination, or medical education.
Reach, Relevance, and Resonance
Reach, relevance, and resonance combine a KOL’s reach with their relevance to the target market to predict their resonance. This can be seen as combining scalar and vector metrics such that how many times something is accomplished is multiplied by the impact of each iteration to determine an overall measure of success. For example, if this concept is applied to MA, a KOL with a significant publication history in neurology may have impressive reach, but if the Medical group is focused on oncology, the KOL may have little relevance, decreasing this KOL’s resonance in the oncology space.
Value vs Impact
MA teams often speak of metrics to demonstrate the “value” of the function. As MA evolves into increased prominence within the organization, it has the opportunity to shift from the language of justifying value to demonstrating impact, thereby shifting from a deficit perspective (“Is MA really worth it?”) to an asset perspective (“What opportunities does MA present?”). This shift can also influence the design of an MA team’s metrics framework, with teams focusing on metrics to drive opportunity and innovation rather than metrics meant to justify value.
Can vs Should
Eventually a strategic planning process with metrics at its core will generate more metrics than are feasible to implement. Keep in mind that every measurement comes with an associated cost not only in time and budgetary resources but in the danger that less useful metrics may obscure the learnings from more central and essential metrics. Just because something can be measured does not mean that it should be measured. Likewise, the choice to measure assigns value to the actions or outcomes being measured, meaning that metrics have the potential to incentivize behaviors, and it becomes essential to consider whether a specific metric may unintentionally incentivize negative actions.
Measuring vs Monitoring
Some metrics require no context—they stand alone as absolute measures. However, many metrics are only relevant as measures of change. This latter category of monitoring metrics can show a group or organization whether it is moving toward or away from its goals. Measuring and monitoring metrics can be cyclical and even periodic, with feedback from monitoring influencing the next iteration of metrics designed to measure outcomes. However, monitoring metrics present special challenges in implementation and interpretation. Who will do the measuring and how often? What types or magnitudes of changes over time are meaningful, and what actions should be taken based on these changes? Does a baseline exist against which to compare current and future measurements and, if not, how will baselines be established and at what point will the metric return meaningful insights? These questions must be answered in the Strategic Plan.
Internal vs External
Internal metrics demonstrate impact to other stakeholders within the organization and must be designed from the perspectives of these stakeholders; for example, Commercial may prioritize reporting on HEOR value, whereas Research & Development may prefer insights generated by MA leading to additional clinical studies. External metrics describe MA’s impact on stakeholders outside the organization and are likewise designed from the audience’s perspective, for example, measuring the increase in knowledge in a patient advocacy community around the effectiveness of investigational agents due to an MA external education program. When aligning metrics with the Strategic Plan, it is useful to keep these internal and external audiences in mind so that MA can choose metrics to match motivations of various internal and external audiences.
Leveraging Metric to Improve Outcomes in MA
The right to define strategies comes with the responsibility to measure results. In this way, solving the challenge of metrics is an essential step toward the progression of MA to function as a strategic partner and leader within the highest echelons of the organization. To achieve such, these metrics must have meaning, they must align with the Strategic Plan, and they must measure impact and influence future decision-making and outcomes. By becoming familiar not only with the menu of currently available metrics but also with the factors used to evaluate the design and purpose of metrics, an MA team can continue to evolve toward its strategic potential.
1. Matheis, R, et al. Metrics Driven Communication in Medical Communications. Miami: MAPS 2019 Annual Meeting; 2019.
2. Evers M, Ghatak A, Suresh B, Westra A. A vision for Medical Affairs in 2025. McKinsey & Company. Accessed June 1, 2021. www.mckinsey.com/industries/pharmaceuticals-and-medical-products/ourinsights/a-vision-for-medical-affairs-in-2025
3. Evers M, Fleming E, Ghatak A, et al. Open interactive popup. Pharma Medical Affairs: 2020 and beyond. McKinsey & Company. Accessed June 1, 2021. www.mckinsey.com/industries/pharmaceuticalsand-medical-products/our-insights/pharma-medical-affairs-2020-and-beyond
4. Chin J. Measuring performance of field-medical programmes: medical science liaison metrics consensus. J Commer Biotechnol. 2007;13(3):177-182. doi:10.1057/palgrave.jcb.3050057
5. Medical Affairs Professional Society. Communicating the Value & Measuring the Impact of Medical Affairs Functions. Accessed June 1, 2021. medicalaffairs.org/communicating-value-medical-affairsdashboards-kpis/
6. Envision Pharma Group. A white paper – the value of Medical Affairs. Accessed June 1, 2021. www. envisionpharmagroup.com/news-and-events/news-events/a-white-paper-the-value-of-medical-affairs/
7. Johnson H. Measuring Medical Affairs. Komodo Health. Accessed June 1, 2021. heatherraejohnson. com/wp-content/uploads/2020/09/MeasuringMedicalAffairs.pdf
8. Konkiel S, Hayes S. Measuring the value of Medical Affairs. Altmetric. Accessed June 1, 2021 altmetric.figshare.com/articles/presentation/Measuring_the_value_of_medical_affairs_with_ altmetrics/13317431/1
9. Melo-Pereira S, Dorfman S. Measuring the value of KOL interactions. PharmExec. Accessed June 1, 2021. www.pharmexec.com/view/measuring-value-kol-interactions
10. Farhood R, Keenan G, Gooljarsingh TL, et al. Medical Affairs launch excellence best practices for Medical Affairs. Medical Affairs Professional Society. Accessed June 1, 2021. medicalaffairs.org/medicalaffairs-launch-excellence-standards-guidance-templates/
11. O’Brien I. The metrics of Medical Affairs. EMG – GOLD. Accessed June 1, 2021. www.emg-gold.com/ post/the-metrics-of-medical-affairs