Skip to content


  • Add this article to your LinkedIn page
  • Add this article to your Twitter feed
  • Add this article to your Facebook page
  • Email this article
  • View or print a PDF of this page
  • Share further
  • Add this article to your Pinterest board
  • Add this article to your Google page
  • Share this article on Reddit
  • Share this article on StumbleUpon
  • Bookmark this page

Dark Analytics: Shining a torch for pharma

In a FirstWord FutureViews report on dark analytics and pharmaceuticals, comments from PA Consulting’s dark data team – represented by Dr Shaibal Roy, Martin Knoebel, and Michael Carver – are included.

On appraising pharma’s maturity in dark analytics, PA’s dark data team comments:

“The implications of dark data [for pharma] are profound. There are so many obvious decisions made by commercial leaders are based on traditional ways of working or long-term assumptions due to the fast pace of the industry, especially in the commercial cycle. However, the implications are not so obvious that we see thoughtful, analytical approaches that thoroughly challenge assumptions.”

On dark analytics to maximise the value of the deep dark web, the team says:

“There are two types of social media data that pharma can mine. First, there’s data in the form of narrative content from social media channels, which can be especially useful to understand unprompted conversations. Secondly, there is the data about the content – the metadata. There are different types of social media as well – there is open social media, closed social media, and secret social media. Closed refers to social media requiring registration and verification of identity, such as physician communities. Secret refers to social media wherein only individuals with membership of that community know about and can access the platform, and its search results You cannot treat these social media types equally.”

“Rather than thinking about what sort of data we have or what we can do if we had buckets of data, in our opinion, it is more important to think about that our biggest problems are, what we can do about them, and how we can holistically use the data available to help us solve the problem. So, for us, starting by asking about the types of social media data available can be very limiting. We prefer focusing on what insights pharma can mine from data types that may or may not include social media data.”

“Social media data can be used to understand how the conversation is going about living with a certain illness, for instance, psoriasis. Without structuring how you might analyse this precisely, you can spend a lot of time not learning very much at all. Therefore, we encourage structured approaches and discourage researching everything that has ever been published about psoriasis, or even code everything that has been said in the last seven days to come out with some sort of score. We ask our clients, if they could evidence or prove something associated with living with psoriasis, what would they do about it? How would their approach change? Some of our clients are not prepared for such a discussion because their expectations may be that social media will reveal something stunning and new. Sometimes there is a preference to see something like a map of the world with a hotspot over a country, and the hope deep down is that such a map might relate to a clinical trial location opportunity for a failing or future trial. To build from this expectation and facilitate commercial insights relating to an illness or a wellness community, we typically go further than simply using a specialist social media listening tool – we will work out where the overlaps are between communities talking about living with psoriasis with communities talking about skin products and the known list of usual clinical trial sites. With this type of convergent thinking, you can very quickly you end up in an unexpectedly useful place.”

On dark analytics technology, tools and techniques, the team comments:

“We actually discourage asking about what type of technology would be needed to effectively embed analytics into pharma’s business model. It’s a bit like presenting at innovation conferences and people ask which technology they should be more scared of: AI or machine learning? When we have a ‘what technology is needed’ type of question, we end up making it difficult for leaders to engage. It is our strong philosophy that technology is simply an enabler for executing analytics that provide insights and drive decisions and actions that result in outcomes.”

“Historically, pharma is used to making decisions in a certain way, which is not consistently evidence-based across the whole organisation. There might be pockets in the company that are extremely analytical, data-driven and experienced in data. Then there might be other areas where data and analytical best practices have not really come through yet. The disconnect between these two areas, especially at the leadership level, can lead to a culture where the potential benefits of sharing analytical approaches is not recognised. So, for us, rather than asking about what type of technology is needed, a better question would be how does the culture of the organisation need to change to really enable best practice uses of data? Much more than the technology, what is more important is the culture of operational leadership and how key decisions are made.”

“We encourage pharma to think deeply about what it means to scrape a website. For example, let’s consider the deep web as an open source available to access and use. The key question is why a pharma company would want to scrape the deep web at all. Why do you need access to the deep web? Just because it is possible, does not mean we should do it. At times, it may be more sensible to go to a supply chain partner, share your concerns and challenges, and work in collaboration with stakeholders there, instead of investigating internet sources that will have a range of terms of use and privacy policies that could easily present restrictive limitations. Basically, we need to know why we are doing things with deep web data in the first place and whether that is the right path to take. If we must use social media data, for example, then we need to ensure we do so in a purposeful way and not collect data in case it is useful.”

On the utility of dark data analytics for insight generation in the real world, the team says:

“If you think about a congress that happened in the early 2000s, there would have been enthusiastic and tech-enabled people using their desktop computers to access private discussion forums on their physician community website to talk about the conference they attended. In the years that followed, those physician communities started to create micro-sites around congresses, working with potential sponsors to effectively develop a virtual experience of attending that conference. These micro-sites have also become a great way to reuse content and access videos from conferences in a way that was unthinkable in the early 2000s. Across all of these interactions during the conference, including the micro-site, data is collected. There will be data on who is accessing a video, when, for how long, what links people go to afterwards, and who their connections are. You can use the information about the network of people talking about that conference, attending that conference, and then receiving links about that conference in different channels. For example, you might like to understand specific areas of knowledge development to better support a group of 5,000 oncologists involved in a congress; so, you can keep in touch with them over the next five years and start to piece all the conference data together in a purposeful way.”

“Pharma’s expectations tend to be bottom-up. We have been asked to “tell us everything you can” about people living with a certain condition. And what we see all the time is screen grabs from social media listening tools, citing personal accounts saying something about living with an illness. That is not the way we work. If you need a bottom-up approach to understand an illness, you are much better off speaking to a specialist about what living with a condition is like.”

“Acceleration of clinical trials is one of the obvious use cases for advanced analytics using social media data. There is also a growing sector of specialists and CROs [contract research organisations] that use social media in some way. Our observation is that over time most social media use has been to develop engagement campaigns or direct Twitter ad spend to encourage people to get involved in a clinical trial. We believe there is much greater opportunity here. For example, understanding academic and patient communities in different places to develop trial sites is something that could really help with the acceleration of clinical trials of the future. For example, could you use an open online global dementia community to develop a network of dementia clinical sites around the world for future clinical trials? This might not solve a problem right now, but it will help to develop more effective clinical trials a decade from now. Clinical trial development becomes more strategic with purposeful use of dark data.”

On identifying the overcoming barriers to dark analytics, the team comments:

“The challenge is not only in using various types of data; it is the inherent risk that comes with data and metadata types as well. If you are gathering open source data, there is always an immediate question about whether the information is sensitive or personal. Open source information can still be sensitive or personally identifiable. We have heard open source data referred to as ‘fair game’ by social media analytics professionals trying to work with pharma – this is not a helpful way to think of your risks. So, if you must analyse data that introduces risks, you can sensibly limit risk by limiting the data that you access. Starting with access only to meta data about the content of interest is another way to lower the risks in initial exploratory efforts.”

“The critical challenge is cultural. We need leaders in pharma companies to get into the habit of relying on the data instead of simply their own long-term position and assumptions. Leaders also need to embrace changes in how they manage their peers and colleagues and anticipate what might happen in response. For example, when leaders start to make analytically-driven decisions and find this new way of working is not working exactly as they had hoped, leaders may assume the new way is falling down. Colleagues will quickly return to the old way of making decisions, without actually looking for the evidence or attempting to learn lessons to try again. So, the culture and leadership in pharma have to change so that people learn to rely on insights and analytics to make decisions.”

“We believe social media in general has fallen into the trap of over-promising and under-delivering, which has resulted in a lack of confidence to invest time and funds needed to make the most of social media. This means that the people leading clinical trials for pharma companies do not have any expectation of social media to reduce their operating costs. So, we have created this environment where it has been difficult to make social technology-oriented thinking part of a normal pharma role. Social media has become just a shiny thing, rather than a fundamental driver of business value. We need to start using dark data as part of the normal way of working in pharma, perhaps starting with advanced analytics using social media to reduce R&D opex.”

“It might be that there are more people in pharma with job titles related to social media listening and analytics, but is there really a lot of meaningful discussion? The conversation about social media listening in pharma is still limited to simple listening and not really analytics. Social media listening is a skill set. The tools that you might use to manage social media data are the same tools that you would use to manage any data set. Effectively listening and analysing social media starts with knowing why you are listening, otherwise you’ll spend hours cruising through different blog postings and you will feel completely out of touch with your scope.”

“Pharma leaders need to worry less about data analysis and more about what it is that they have always wanted to know. They must focus more energy on defining the question that they need answered, rather than assuming what technology or analysts can or cannot do. We use the phrase ‘commissioner of insight’ to explain the mindset leaders need to adopt to make the most of dark data. The necessary skill is to be an effective commissioner of insight. To do that, you must have a clear understanding of the outcome that you need. You must be capable of understanding the commercial and patient-oriented outcomes that you seek. You must be capable of relating those outcomes to actions that you can currently take and then make changes to the operating model, if needed. You must be capable of understanding a data story and relating that to a position you must take. You must be capable of making a decision that is influenced by data. Therefore, you must be capable of rejecting your gut instinct in favour of data and analysis.”

“The future rewards of overcoming challenges in dark analytics are the same potential future rewards as with any evidence-based approach, which include being profoundly more effective in terms of directing marketing effectiveness, driving sales models, and engaging with patients and the public.”

“So much can be achieved with your dark data in a few days with guidance from external stakeholders and commercial partners. You do not need to hire the commercial leader from a creative agency that you have been working with for a long time to get on top of dark analytics – although it is very understandable that you will turn to the partners you trust most. Why not invite completely new thinking instead – if you learn to ask the right questions you can easily engage the right organisation – or crowd – to answer those questions in days or weeks rather than months.”

On navigating the future of dark data analytics, the team comments:

“In the future, if we don’t make changes, social media analytics might be reviewed as an epic missed opportunity. Future pharma leaders will review social media analytics as something they cannot believe we did not use more of. We all have the sense this opportunity is being missed. That is why firms like us are enthusiastic about opportunities to put our insights out there regarding social media data utility, in the context of the extraordinary opportunity presented by the use of your dark data. But the moment to really exploit social media data with a great sense of wonder and innovation may have already passed because we know cross-sector best practices. Our shared challenge is to spread these ways of working. So, the future of social media analytics for pharma is in the extension of known best practices from a few departments or companies and then turning them into routine and shared usual practices across multiple business units.”

“Rather than worrying about the expanding digital universe as a problem, pharma can prepare for the future by thinking about the digital universe as an opportunity for greater commercial value from more dark data. This requires pharma to embrace exploratory approaches and commission the right insights, which means having a crystal-clear understanding of what you need to know. What is more challenging here is to prepare for the expanding digital universe while thinking about revenue and patient impact opportunities. Instead of being seduced by a really big number (e.g. the reach of a tweet), you must think about how to make the most of, and light up, the dark data universe that is emerging. You need to use the dramatic pace and scale of global dark data innovation to stimulate and inspire your own approaches to make the most of your data and metadata.”

Contact the business intelligence and analytics team


By using this website, you accept the use of cookies. For more information on how to manage cookies, please read our privacy policy.