Skip to content


  • Add this article to your LinkedIn page
  • Add this article to your Twitter feed
  • Add this article to your Facebook page
  • Email this article
  • View or print a PDF of this page
  • Share further
  • Add this article to your Pinterest board
  • Add this article to your Google page
  • Share this article on Reddit
  • Share this article on StumbleUpon
  • Bookmark this page

Shining a light on dark data

This article was first published in The MJ

Austerity has forced authorities to prioritise the services they offer and to target the groups who can benefit most from the allocation of diminishing resources. All that has to be done alongside reducing overall demand and signposting where self-service options and third party support can be used before the council has to intervene. To do this effectively and to make the right strategic and evidence-based decisions requires information and insights.

Councils have an advantage in that they already hold vast amounts of information on their local population, the local area and the services available. The trouble is that while this information is often easily accessible it is hardly ever used for anything other than the original purpose, usually routine reporting. Yet re-using that data and looking at it in a different way can generate new insights that can drive real improvements. 

This approach is known as “dark data” analytics and is already being used by the private sector to inform commercial decisions to reduce their operational expenditure and drive value from their assets. For example, applying simple, structured analytics to existing data recently allowed a global pharmaceutical company to secure new insights into the productivity of its sales workforce. It then used this to focus sales personnel on selling a smaller set of products as the data had showed this was more successful than trying to sell a wider portfolio of products. By using data to refocus the team they saved $10m.

In the UK utilities sector, regional water companies such as Yorkshire Water have adopted dark data approaches to provide a new evidence base for their transformation plans. A side effect of analysing dark data in this way is that leadership teams start to focus on the data together, and that means they drive each other to take more evidence-based decisions.

Dark data approaches work most powerfully with problems where leaders can sense that there must be something hidden in the data that has not been revealed before. They might have a hunch or rule-of-thumb that they want to explore. For example World Vision, the international children’s charity, had an established set of assumptions about how best to increase donations, but they were not delivering results. They used their dark data to examine hidden patterns in the behaviour of their loyal supporter base and found they were disproportionately targeting a single demographic group using a single channel, and missing out on valuable donations from new sponsors.

Local government is at the early stages of adopting dark data approaches. One authority has investigated how to increase the effectiveness of foster carer recruitment by applying dark data analytics to openly available social media data. This aims to provide evidence about specific local communities that are not currently targeted by existing communications and marketing efforts.

Other opportunities could include analysing housing repairs data, for example comparing information on broken doors with social care data on alcohol abuse to identify possible domestic violence issues. Combining policing and local authority data could be used to develop new insights, such as using neighbourhood policing statistics alongside children’s services data to identify young people at risk of joining gangs or not attending school.

There are two key principles that need to be adopted for successful dark data analysis. The first is to understand that effective use of dark data starts with frontline staff, senior officers, and elected members simply asking each other about the hunches they have always wanted to prove or disprove, or the gut-based decisions that need to be converted to evidence-based decisions. What is important is to start with these questions, rather than immediately looking at what data is available. While beginning with the data can bring information, it does not always help to support a change or decision. However, by starting with a decision and then looking for the evidence, scarce resources can be focussed on solving the priority issues first.

The second principle is not to worry about whether it is ‘big data’ or ‘small data’ that should be analysed. It’s not the size or format of the dataset that matters but the technique and logic that are used to answer the key question. One of the advantages of the tools used in analysing dark data is that do not need perfect or comprehensive datasets to add value to strategic decision-making.

Whilst dark data cannot address all the pressures faced by authorities, if deployed carefully and with the right support in place, it can play a valuable role in managing scarce resources.

Contact the author

Contact the innovation team


By using this website, you accept the use of cookies. For more information on how to manage cookies, please read our privacy policy.