Getting Started With Data Lakes in a Smarter Way

The data lake is a place to put all the structured, semi-structured, and unstructured data organizations might want to gather, store, analyze and use for insights and action.

Traditionally, data has been a distributed resource across the organization (internal data) and the ecosystem in which it operates (external data). It’s difficult to combine the right data if it’s spread out in different places in and out of the cloud to successfully complete a big data project.

Data lakes and analytics are some of the logical concepts and realities that come from (big) data lakes, among others. Several years ago, James Dixon, the CTO of Pentaho, introduced data lakes as a better choice for big data applications compared to data marts or data warehouses.

A data lake can be compared to a large body of water in a more natural state, as opposed to a data mart, which is like a store of bottled water. It is constructed of a stream of data filled with one or more sources. Users are welcome to examine the lake, test the waters, or take samples.

The main purpose of data lakes is to be used for analytics and action

In fact, a data lake looks a lot like a lake, without the swans. But it’s not exactly a lake. It is clear, however, that a big data lake is, in essence, a data storage repository that contains vast amounts of raw data.

There is no traditional data management approach that can handle big data or big data analytics (or that doesn’t cost a lot). We are essentially looking for correlations between different data sets using big data analytics so that we can bring our business outcomes together. Moreover, if the data sets are in different locations, it will be virtually impossible to link the two.

This might involve combining data from different sources to gain insights about customers (for example, traffic data, weather data, data regarding customers not directly related to our business) to improve the customer experience, offer new services, or simply sell more.

Data ingestion for bottom-up analytics

All of this is relevant to a data lake, right? That’s because data lakes are one of two data management methods for analytics.

In this case, we’re talking about the data lake, which is the bottom-up alternative to data warehousing (top-down). In order to better understand this, let’s explore the image of a real lake. Lakes do not just fill up like that. Water carries down rivers and streams.

Using big data lakes for big data analytics and solving the data silo problem

The same thing happens in a data lake. No matter the source or structure, data is ingested, also known as the ingestion of information. In order to achieve our goal, we gather and use the relevant data analytics.

It is quite common to find your data in quite a few formats: structured data (say, data taken from a traditional relational database or spreadsheet: rows and columns), unstructured data, log data, XML, machine-to-machine, IoT data, etc.

From a context perspective, they also involve many types of data, such as customer data, data from OSIsoft applications, sales data, etc. (entered via APIs into the data lake). It also follows that we increasingly have access to external data (sources) for the purposes of achieving our goals.

Analyzing, visualizing, and taking action with data lakes

Data is stored and fed into the data lake, as it makes/could make sense, while it is also continuously coming into the lake, via APIs, and other methods, via batch processes, and from all sorts of applications and systems.

A second big part of the equation (the first being ingestion) is the storage dimension. Consequently, there are no silos in a big data lake approach. As a result, we are now ready to start the exciting work of big data analytics.

Using artificial intelligence we can detect patterns within a data set seemingly unrelated to one another (such as purchasing behavior and weather patterns), between customer data from one source and customer data from another, between traffic data and air pollution data, etc. It’s simple and straightforward. Where can these patterns be used? Big data is an asset in almost every way, and there are countless real-life examples of how it can be used to your advantage.

Analyzing is obviously not enough. Having analyzed something, you must envision, understand, and act on it. According to the data lake infographic below, as EMC describes how data lakes operate: data would flow out, which was analyzed, resulting inaction which led to new insights.

Why do we need data lakes? Benefits of data lakes

Traditionally, there have been two information management approaches in analytics. How come data lakes (bottom-up analytics) are popular for data analysis?

Various reasons contribute to this. A data lake is not just some bottom-up chaos; it has numerous technologies, protocols, and so forth involved in order to be classed as a lake. As an example, consider the water that enters the lake through the streams: there are filters in place before the water enters the lake.

Risks and challenges facing data lakes

We will not get too technical here, as there are many benefits of big data lakes. Likewise, there are challenges, risks, and benefits to consider as usual.

As one example, if a data lake is not properly designed in a strategic way with the necessary goals in mind and cleaning, data swamps can result. Additionally, organizations move away from traditional data lakes towards goal-based, business-driven ones because of this.

The approach to creating a data lake must be driven by an organized strategy and business-driven approach. The data volumes have become more important in recent years due to the rising data volume and the belief that end-to-end, all data has potential value.

Wrap up

Data lakes are increasingly popular with companies, as the inherent advantages of easily accessible information streams and the lower cost of storing it compared to traditional warehouses are attractive to them.

By leveraging our data lake consulting, we help you understand and utilize your Big Data for better insight. By creating an architecture framework for your data, we provide you with a roadmap for understanding your business needs.