Home > Design Patterns > Automatic Data Replication and Reconstruction

Automatic Data Replication and Reconstruction (Buhler, Erl, Khattak)

How can large amounts of data be stored in a fault tolerant manner such that the data remains available in the face of hardware failures?

Automatic Data Replication and Reconstruction

Problem

Big Data storage technologies generally use commodity hardware that normally carry a higher chance of failure when compared with enterprise-grade hardware. Consequently, the probability of data loss increases.

Solution

Multiple copies of data are kept, and any copies lost due to hardware failures are automatically reconstructed.

Application

A Big Data storage technology is used that implements automatic replication of data such that the same dataset is stored on more than one machines and further provides the automatic reconstruction of lost copies of data.

Mechanisms

Automatic Data Replication and Reconstruction: The dataset that needs to be stored is automatically copied across multiple machines in a cluster. This way, if a machine becomes unavailable, data can still be accessed from a different machine. Furthermore, functionality is added to automatically reconstruct a copy of data that was lost due to machine failure. Apart from providing fault-tolerance and high availability, the application of the Automatic Data Replication and Reconstruction pattern further provides high performance data access and enables scaling-out.

The dataset that needs to be stored is automatically copied across multiple machines in a cluster. This way, if a machine becomes unavailable, data can still be accessed from a different machine. Furthermore, functionality is added to automatically reconstruct a copy of data that was lost due to machine failure. Apart from providing fault-tolerance and high availability, the application of the Automatic Data Replication and Reconstruction pattern further provides high performance data access and enables scaling-out.

  1. A dataset is saved using a distributed file system.
  2. (a,b,c,d) The distributed file system automatically creates four copies of the dataset and saves the copies to Nodes A, B, C and D of a cluster.
  3. Due to a hardware failure, Node A becomes unavailable.
  4. When a user tries to read the dataset, the operation succeeds as the user is seamlessly directed to any of the available Nodes B, C or D.