Four primary uses of CME data

Robinson points to the four primary ways CME providers use the data they collect during programs:

1. To show supporters the value of education they provided
2. To identify persistent clinical challenges that should be addressed in future education
3. To report back to accrediting bodies to demonstrate changes that resulted from the education
4. To share information with relevant stakeholders via publication, should compelling insights be identified

To compile the data that serve all these needs, CME providers must consider how data will be collected while they’re still in the activity-planning phase. Similarly, while providers are planning the focus of the educational content, they must also discuss what types of assessment questions must be asked to obtain meaningful data to enable the crafting of a compelling outcomes story.

Return on Education

The primary objective of a CME program is still to educate clinicians, and data empowers CME providers to demonstrate the Return on Education (ROE) of a program. With today’s technology, it’s easy to use a pre- and post-test to convey how understanding of a topic improved over the course of the program. Additional information such as what questions were asked by attendees and how they responded to polling questions help better frame ROE.

“That data is what makes up the meat of our outcomes analysis because that’s what’s informative,” says Mike LoPresti, Vindico’s Senior Vice President of Educational Strategy.  He adds that it’s important to also gather information that helps put the data in context for an accurate ROE story.

For example, if the pre- and post-test data show that people didn’t learn anything new, it would be beneficial to have also asked participants to self-report how much of the data was new to them. If most reported they only knew 20% of the content beforehand, that indicates a lot of new information was presented. LoPresti explains they would then look at the evaluations and speakers’ ratings for additional context as to why there wasn’t much difference in knowledge between pre and post.

“If we’re doing our job well, we’ve created the content and we know the audience, and we can see how they react to the content. At that point, we can come up with a data story and an outcomes story that we feel truly represents what happened during the program,” he says. “The supporter can then take our reporting and create their story internally as to whether the money was well spent and what to do strategically from there.”

Identifying themes and challenges

Among the many insights supporters receive from CME providers are those around themes, trends, and challenges. The provider uses data to answer questions such as, ‘What is this audience trying to understand?’ and ‘What is the next question to address in future programming?’ This is determined in part by looking at the questions asked by the audience; therefore, it’s important to have a versatile technology platform that makes it easy for them to ask questions while also gathering Q&A data at that individualized level.

For instance, a CME provider might look at the reporting it receives from a technology partner or platform to assess if participant questions indicate they didn’t understand content, questioned it, or needed more information to interpret it (so what was presented wasn’t enough). Also, is there a new, adjacent topic the questions raised? For example, were participants particularly interested in a comparison of study results versus the standard of care, which may not have been addressed in the program?

“At that point, we like to find creative ways to either develop a follow-up program or course or be proactive and develop programs that include a live activity and follow-up podcast focused on FAQs that came out of the live event,” says LoPresti.

Robinson adds they are often asked to do a breakout of data by cohorts to then look for trends or differences among them. “It is always interesting to know if we see any differences between specialty, years of practice, or patient volume, for example,” she says. “These are all the things that are very informative to our supporters as they evaluate their target audiences, and it also helps us to tailor our future education. It is a fairly common scenario that a particular specialty audience may not have been a target for the education, but they participate, we collect data, and uncover from the data that they may be diagnosing or managing more patients with a given condition than we thought.”

Another way providers use demographic data is to correlate the data to learners. Sometimes, programs are provided for multiple specialty or discipline audiences. The provider may have data showing a group of learners improved their knowledge by a certain percentage. “Were they doctors or nurse practitioners, primary care providers, or specialists? That can be meaningful to a supporter,” says Robinson.

Whether looking to correlate data by cohort or determine trends, Robinson notes it has become much easier to do so. “It used to be that information about learners’ geography, specialty, discipline, etc. were handwritten. If we wanted to drill down to that level of detail, we’d have to determine who was going to figure it all out,” she says. “Now technology can parse out that data by user, we’re able to tell more of a story and show segmentation all the time.”

Critical liaison between pharma and physicians

In the highly regulated healthcare industry, Medical Affairs departments get insights directly from doctors through scientific exchange, advisory boards, congress participation, etc. However, CME providers are required by accrediting bodies to collect data every time they interact with physicians. This affords them unique opportunities to provide data and value to Medical Affairs External Education and Field Medical teams.

“That’s a significant value we provide,” adds Robinson. “We go beyond education to provide insights they can immediately use. For example, our polling during a CME program could show that 50 percent of the doctors aren’t going to prescribe a certain drug because they aren’t comfortable doing so, or because the drug isn’t on the formularies. We can ask direct questions like, ‘Why aren’t you using this?’ This nets information you wouldn’t get just by looking at changes in ‘scripts.”

Robinson anticipates a growing focus on qualitative data, as well as a desire to look longitudinally to see what can be learned across multiple programs, particularly those on the same topic. “There is a lot of value in being able to look back in time on things that affected learner behavior,” she adds.

As the role of CME provider continues to evolve, Robinson anticipates increased need for data demonstrating the impact of these programs. “At the end of the day, the most compelling outcomes stories are going to come out a combination of quantitative and qualitative data, requiring robust collaborations between CME providers, technology companies and Medical Affairs teams within pharma. If our events are going to improve clinical practice in ways that benefit patients, it’s on us to ask the right questions and our data providers to allow us the capability to do that,” says Robinson.