You are living in the big data generation. The sources of digital data used for analytics are growing exponentially. Millions, billions, and more data points. This is not data that can be fit in a spreadsheet. Some companies are struggling to keep up and make sense of this overwhelming amount of information. Like the industrial revolution, this is the information revolution.

What makes big data possible? Data collection and data storage. The digital storage of information is the ability to record things in an easily accessible digital format. Consider the history of written language: rock pillars, clay tablets, papyrus scrolls, the printing press. All of these were innovations in the way that information is stored and communicated. The invention of the computer launched humanity into a whole new level of information storage and we are seeing this trend continue today. As storage becomes cheaper and cloud storage services proliferate, the ability to store data is no longer a significant constraint. Big data relies on the storage of massive amounts of data and the ability to access it quickly and easily.

The scale of big data is much greater than our everyday familiarity. We are comfortable working with files in the range of kilobytes, megabytes, and gigabytes. These are size categories based on the unit of the “byte” which is a sequence of eight 0/1 bits, such as 0010101. A kilobyte is more than a thousand bytes, a megabyte is more than a million bytes, and gigabyte is more than a billion bytes. We often refer to our cell plans in the number of “gigs of data.” But big data operates at scales even larger than gigabytes. A terabyte is more than a trillion bytes and a petabyte is more than a thousand terabytes. These is storage at a scale much greater than the common megabyte.

In the world of business, we are not just talking about data on companies. We are talking about data on individuals, behaviors, and transactions. Data is now collected in almost every aspect of life. Every time we browse the internet, data is being collected. Every time we use our phone, data is being collected. 

The main source is big data is the internet. An individual browsing the internet records large amounts of information including pages visited, time on page, browsing path, etc. Social media is an even stronger indicator of individual behavior. This is why data from social media can be so valuable. Every view, every click – these are data points that can be used to refine the business offering.

Corporations are now collecting data on all of their transactions. It is extremely powerful to know who bought which product, when they bought it, and possibly why they bought it. This transactional data can help a company understand and serve its customers in a much more effective way. Consider the transition from non-traceable cash to credit cards as a means of payment. This identifies the buyer and allows for analysis of repeat transactions. Loyalty cards also serve the purpose of identifying the buyer at the point of sale.

Publicly available data is also going digital. It used to be that public records were stored in paper form in some government filing cabinet deep in the archives. This type of information is being digitized and becoming more readily accessible. Corporate filings of financial statements were once accessed by paper, but this type of information is now readily available on the internet or through various data platforms.

Sensors are part of the future of big data. Sensors collect data on anything from motion, temperature, video, etc. This is an example of how data collection is growing beyond the internet. In the internet of things, almost any information can be digitized and used. Erik Brynjolfsson, Director of MIT’s Center for Digital Business says, “We’re rapidly entering a world where everything can be monitored and measured…But the big problem is going to be the ability of humans to use, analyze and make sense of the data.”

Working with big data is not easy. Big data is both large and complex, which makes it difficult to process using on-hand data management tools or traditional data processing applications. Big data will quickly overwhelm a laptop or spreadsheet. To address this, the technology and analysis of business analytics is changing. Technology is advancing beyond the spreadsheet and analysis is moving beyond traditional methods for working with numbers.

Corporations are asking, “What should we do with big data?” It is an asset of unknown value. Most companies now recognize that their data is valuable, but they are unsure about how to tap into this value. With the overwhelmingly large amounts of data, it can be difficult to separate what is relevant from what is irrelevant. This means that it is easy to lose focus and get lost in an exploration of the data. The key is when, how, and where to apply analytics. Just like with smaller data, the goal of working with big data is to develop actionable insight.