Drowning in big data? This thing called Hadoop could be a life preserver

Drowning in big data? This thing called Hadoop could be a life preserver

The electric industry is poised to become a cross-industry thought leader in handling, analyzing, and making strategic use of big data. But first the industry has to master the data. By big data, Im referring to terabytes and petabytes of information; the type of data that will have your IT team screaming about lack of storage and causing bottlenecks in the IT network. At its core, big data presents two fundamental challenges:  how to store and work with volumes of data sizes and structures and how to turn this data into a strategic advantage.

One answer to these challenges can be found in a funny word: Hadoop. Hadoop, in a nutshell, is a platform that can effectively store and provide computational capabilities over the biggest of big data.  It is a cost-effective way of reducing analytical challenges into small, bite-sized chunksbasically a divide and conquer technology. Simply put, Hadoop is a distributed and scalable open-source platform for handling big data.

Hadoop runs on clusters of interconnected computers known as nodes that work together to solve the same problem. For example, with Hadoop, utilities can leverage massive volumes of machine-generated data from smart meters to detect patterns and make recommendations to customers. The end result is a more efficient energy grid and improved customer relationships. By understanding technologies like Hadoop, utilities can get the most of all data in a cost-effective fashion and using data warehouses, a costly method, only for the highest-value data.  Additionally, Hadoop can provide a centralized data hub across organizations, reducing silos of information.

You may be asking yourself, so what? Why should I care about big data and Hadoop? Recently, SAS was working with a customer who wanted to use smart grid interval data to develop recommendations for demand response program participation. The customer was unable to deal with the volume, variety and velocity of data available, including  15-minute load information for millions of customers, hourly weather information over many years, and customer comments collected in a customer information system (CIS).  All of this data is needed for a utility to be able to analyze real-time petabytes of information collected from electrical smart meters and convert the data into patterns of behavior that can be utilized to optimize power systems and predict future customer behavior.

In this case, the customer elected to use Hadoop for the massive storage of historical information from smart meter readings and then created queries for the data needed to analyze the potential for participation in demand response events.  Being able to cost-effectively store years of information and seamlessly query and apply advanced analytics, the utility could segment their customer base and actively pursue the best candidates to most likely participate in demand response activities to meet overall energy goals.

Focusing on solving current demand response problems with historical methods does not work very well with existing database systems since customer smart meter data is generally stored separately from system information and customer service comments.  From an analytical perspective, the more detailed information available, a better predictive model can be developed. With customers usage habits changing over time, a Hadoop data structure combined with advanced analytics gives utilities the ability to make recommendations to customers who are most likely to participate in demand response programs as well as generate other information that could be used to develop behavior-oriented rate structures or other customer programs.

Most utilities currently do not have information systems robust enough to capture, store, or analyze information like real-time smart grid data. Traditional siloed databases were never intended for the volume, variety, or velocity of data that is available these days. With Hadoop, no data is too big. And in todays hyper-connected world where more and more data is being created every day, Hadoops advantages mean that businesses and organizations can now find value in data that was recently considered too big.  It is ideal for data from sources such as social media, documents, graphs, and anything that cant easily fit into rows and columns.

Utilities can learn from the technology giants like Yahoo!, Facebook, Twitter, and Amazon, plus industries like banking and retail, who have already adopted Hadoop for their big data needs.   

All and all, it is a very exciting time to be in the utility industry. With the explosion of the Internet of Things, a network of everyday objects from smart meters to iPhones, can share information and complete tasks such as controlling your thermostat or participating in a demand response event.  It is anticipated that there will be over fifty billion devices connected to the Internet by 2020 with many having a direct impact on the utility and its operations The companies that learn to convert this data into a strategic resource will have a competitive advantage in the future in terms of running a cost-effective grid, improving customer service and offerings, and becoming a trusted energy advisor. Being able to combine system, customer, and third-party information while using all of the data available is a game changer.  Additionally, with Hadoop being open source, changes are made nearly daily as best practices from across all industries are funneled into improving the platform. It will be exciting to watch as the newest convergence of technologies and big data meet through smart grids, smart generation and smart cities, all helping us to master and uncover the value of big data.

Source: intelligentutility

SMART GRID Bulletin March 2018

View all SMART GRID Bulletins click here

Enter your email-id to subscribe to the