Lambda Architecture Analysis

CHAPTER 1

INRODUCTION

1.1 INTRODUCTION TO LAMBDA ARCHITECTURE

Lambda Architecture is a design principle in BigData systems where dealing with throughput and latency in real-time is the most important. This mixture of batch processing with real-time streaming process provides the benefits of both the approaches, thus making the system having the precomputed views which enables high throughput and fresh calculations are done on online data to provide end result most accurate with high throughput, decent accuracy and low latency. The lambda architecture is inspired by the rise in bigdata architectures striving for accuracy as well as speed. Architecture contains three layers : i. Batch Processing Layer with precomputed views ii. Real-time

(2008) H-store: a high-performance, distributed main memory transaction processing system. The H-Store system is a highly distributed, row-store-based relational database that runs on a cluster on shared-nothing, main memory executor nodes.OLTP applications make calls to the H-Store system to repeatedly execute pre-defined stored procedures. Each pro-cedure is identified by a unique name and consists of struc-tured control code intermixed with parameterized SQL com-mands.

Jeffrey Cohen et al. (2009) MAD Skills: New Analysis Practices for Big Data. Magnetic, Agile, Deep (MAD) data analysis as a radical departure from traditional Enterprise Data Warehouses and Business Intelligence. We present our design philosophy, techniques andexperience providing MAD analytics for one of the world’s largest advertising networks at Fox Interactive Media, using the Greenplum parallel database system.

Seeger, Marc (21 September 2009). "Key-Value Stores: a practical overview" . Marc Seeger. Retrieved 1 January 2012. Key-value stores provide a high-performance alternative to relational database systems with respect to storing and accessing data. This paper provides a short overview of some of the currently available key-value stores and their interface to the Ruby programming

One just need to append all tweets in the HDFS and periodically run some simple process that aggregates them by hour date. The dataset looks like this :

There is plenty of literature and examples on the Internet on how to do such simple tasks with (from lower to higher level): Pangool, Cascading, Pig or Hive. The idea is that the output of the batch layer should look like above: a tabulated text file with hashtags counts. Because you save all the tweets in the HDFS, you can run a batch process that calculates many other things, and which recalculates everything from scratch every time. You have complete freedom and fault-tolerance here

4.2.2 The serving layer

A serving layer database only requires batch updates and random reads. Most notably, it does not need to support random writes. This is a very important point because random writes cause most of the complexity in databases. By not supporting random writes, serving layer databases can be very simple. That simplicity makes them robust, predictable, easy to configure, and easy to operate. ElephantDB, a serving layer database, is only a few thousand lines of