Home > Design Patterns > High Velocity Realtime Processing

High Velocity Realtime Processing (Buhler, Erl, Khattak)

How can high velocity data be processed as it arrives without any delay?

High Velocity Realtime Processing

Problem

Storing large amounts of data, arriving at fast pace, as a dataset and processing it in a batch manner incurs processing latency, causing a delay before analysis results become available.

Solution

The incoming stream of data is processed as it arrives without being stored to a disk-based storage technology.

Application

A system is set up that ingests incoming events, stores them in a memory-backed storage device and then processes them instantaneously.

An event data transfer engine is used to continuously capture streams of data. The incoming stream of data can be pre-processed while in transit before it gets stored to a memory-based storage device. A realtime processing engine, such as Spark, is then used to process the data stored in the memory-based storage device.

The High Velocity Realtime Processing pattern is applied together with the Streaming Source and Realtime Access Storage patterns due to the requirements of the availability of current data and low latency data access, respectively.

Although enabling realtime analyses, the application of this pattern results in a complicated and costly data processing solution because of the inclusion of the event data transfer engine (complexity) and memory-based storage device (expensive).

High Velocity Realtime Processing: Data is constantly acquired from the streaming source and is kept in memory, from where it is processed instantaneously as each datum arrives. Using memory as a storage medium helps remove latency when compared to using a disk as a storage medium.

Data is constantly acquired from the streaming source and is kept in memory, from where it is processed instantaneously as each datum arrives. Using memory as a storage medium helps remove latency when compared to using a disk as a storage medium.

  1. A sensor provides pressure readings every 10 seconds.
  2. The readings are saved to a memory-based storage device.
  3. (a,b,c) A realtime processing engine is used to instantly process the individual reading.
  4. (a,b,c) An operational dashboard is updated each time an individual reading is processed.
  5. The entire process of processing each reading takes a very short time to complete, resulting in up-to-date information.