Skip to content


  • Add this article to your LinkedIn page
  • Add this article to your Twitter feed
  • Add this article to your Facebook page
  • Email this article
  • View or print a PDF of this page
  • Share further
  • Add this article to your Pinterest board
  • Add this article to your Google page
  • Share this article on Reddit
  • Share this article on StumbleUpon
  • Bookmark this page

Big data should focus on realising customer, business benefits

By Stephen Kerr


Massachusetts Institute of Technology, The Economist and others have made a clear correlation between business performance and big data: If you do big data, you will be more successful.

The fresh aspects of big data lie in the low-cost services and products that process gigabytes of data in seconds. A few years ago, this level of performance was limited to companies with deep pockets. Big data is the new wave.

Big data is as relevant to utilities as it is to any company—possibly more so.

Utilities have a rich data set of near-real-time metered consumption data with other real-time and reference data sources.

Big data offers gold to utilities: understanding how to improve their service and asset management processes and offering customers the means to manage their energy costs.

Utilities, however, are necessarily conservative entities whose income is tightly regulated; the means to recover investment must be clear before spending rate-of-return regulated cash.

Big data exists to provide nuggets of gold, but a utility's big data strategy should not be simply panning for gold. Benefits don't arise from insight; action also is required. Creating insight is an important big data strategy, but its focus should be on realising the potential benefit with the following:

  • Targeting realistic benefits that improve operational costs and customers' consumption behaviour
  • Establishing the means to exploit the nuggets of gold found in advance
  • Preparing for weaknesses in data management practices
  • Using management controls and constraining technology choices to keep capital and operational costs low.

Target realistic benefits

The potential benefits from utility data are many, but often they are unrealistic. When starting with big data, identify realistic benefits. Some utility processes and systems are too risky to change, need significant work or are not worth the effort. While enumerating a benefits list and before claiming a potential benefit and attempting to measure it, ensure a realisation method exists. Figure 1 illustrates some benefits big data can enable for a utility and its customers with filters on how the benefit will be realised and whether the benefit justifies the cost.

Utilities differ from one another. Figure 1 illustrates the thinking required but is not an exhaustive list of potential benefits. Once that thinking has been applied, shorten the list and populate it with realistic targets that cover operational improvements and customer benefits.

Establish the means to exploit nuggets of gold found in advance

Big data allows one to jump in and start looking. Accessible data can be analysed quickly and integrated with new data sets to reveal relationships and events that affect customers and business.

For a utility, it doesn't retain a large data analytic function that can scour a massive, multidimensional data space; it needs to focus on a realistic outcome. Figure 1 provides the first filter for this, but the data analysis that follows for the candidates also must keep true to the achievable benefits objective.

Figure 2 is a simulated data set that illustrates the benefit of improving home insulation in a way a consumer would readily understand how to respond. This is a valuable presentation of the data when customers are trying to eliminate anomalous loads and reduce overall load. Figure 3 shows the same information but the plot of consumption is not compared with another data set, so the conclusions are not as easily reached and provide less consumer value.

Similarly, when the audience is the utility's operations group, data must be presented in a form from which conclusions can be drawn and acted upon.

Figure 4 shows data from the same synthetic data set used in Figures 2 and 3. In this case, additional asset data has been integrated so the locations of transformers that regularly exceed a nominal loading threshold within a single circuit are easily identified. These kinds of insights can be created quickly from utility data, but each has a goal in showing some facet of a business process or customer behaviour that can be acted upon.

The big data strategy must not assume the audience will see the light and act on it. The analysis must consider these factors so the end product is familiar, easily consumed by the target audience, and can be acted upon within the context of the business process or consumer behaviour.

Prepare for weaknesses in data management practices

Using big data tools to discover nuggets of gold is the fun, rewarding part, and it can be achieved easily when data sets are well-managed and easily integrated. Utility data is not always so; it is common to find poor data quality that doesn't conform and is difficult to integrate with other data.

Many utilities recognise they can be better data stewards and have embarked to improve their data management practices. These are generally long-term initiatives to make a utility's whole data set interoperable; customer, GIS, network, asset, outage, mobile, advanced metering infrastructure and supervisory control and data acquisition are important to a utility's operation. These initiatives are intent on improving all dimensions of data management practices.

Companies without data controls find difficulty in exploiting it with big data. A big data strategy should focus on the analysis and exploitation phases and depends on good-quality data's arising as the products of other capabilities. Discovered weaknesses in data management may be handled tactically as in any data initiative, but big data is not the best place to address all of an organisation's data management capabilities. Without those constraints, the scope and cost of a big data initiative will burgeon and become uncontrollable.

Use management controls, constrain tech choices to keep big data capital, operational costs low

Big data offers implementation routes. Many vendors offer services and products whose costs vary significantly. At one end, cloud-based services offer an implementation route that requires nearly no capital investment, notably Google's BigQuery.

At the other end of the cost scale, data-processing products allow the same storage and processing capacities behind a utility's own firewall.

A big data strategy must account for these major differences and accordingly constrain technology choices and implement usage controls.

NOTE: All customer, network, geographic and consumption data in this article are synthetic, and Google BigQuery and Cloud Storage services were used to store, analyse and present it.

Stephen Kerr is an IT strategy expert at PA Consulting Group

Contact the energy and utilities team

By using this website, you accept the use of cookies. For more information on how to manage cookies, please read our privacy policy.