Lambda Architecture

12713 2018-10-23 10:30

Why lambda architecture?

To solve three problems introduced by big data

  1. Accuracy (好)
  2. Latency (快)
  3. Throughput (多)

e.g. problems with scaling a pageview service in a traditional way

  1. You start with a traditional relational database.
  2. Then adding a pub-sub queue.
  3. Then scaling by horizontal partitioning or sharding
  4. Fault-tolerance issues begin
  5. Data corruption happens

The key point is that X-axis dimension alone of the AKF scale cube is not good enough. We should introduce Y-axis / functional decomposition as well. Lambda architecture tells us how to do it for a data system.

What is lambda architecture?

If we define a data system as

Query = function(all data)

Then a lambda architecture is

Lambda Architecture

batch view = function(all data at the batching job's execution time)
realtime view = function(realtime view, new data)

query = function(batch view. realtime view)

Lambda architecture = CQRS (batch layer + serving layer) + speed layer

Lambda Architecture for big data systems

If you find this article helpful

follow me on Github :)

Download App

Learn startup engineering anywhere, anytime