Home > Design Patterns > High Volume Hierarchical Storage

High Volume Hierarchical Storage (Buhler, Erl, Khattak)

How can large amounts of non-relational data that conforms to a nested structure be stored in a scalable manner so that the data retains its internal structure and sub-sections of a data unit can be accessed?

High Volume Hierarchical Storage

Problem

Storing very large amounts of semi-structured data using traditional database technologies so that individual elements of a data unit can be referenced without performing post-query processing is either not possible because of the inability to parse the structure of the data unit or suffers from scalability issues.

Solution

Semi-structured data is stored in a nested form based on a clustered storage technique.

Application

A NoSQL-based Big Data storage technology is used, which stores each data unit as a nested document where not only the complete data unit be accessed via a unique key but also individual sections of the data unit can be accessed.

Mechanisms

A document NoSQL database is used to store nested data. Such a database generally uses a hierarchical format, such as JSON, as an internal data storage format. Apart from providing key based access, such a storage device is also capable of selecting sub-units of a data unit along with full create, read, update and delete (CRUD) functionality. API-based access is generally provided by the document NoSQL database for performing the CRUD operations.

High Volume Hierarchical Storage: A database based on NoSQL technology is used that is capable of storing data in a hierarchical format and understanding the internal structure of the data. Saving data based on a nested structure further enables relational-like storage such that the related child table records can be embedded inside the parent table record.

A database based on NoSQL technology is used that is capable of storing data in a hierarchical format and understanding the internal structure of the data. Saving data based on a nested structure further enables relational-like storage such that the related child table records can be embedded inside the parent table record.

  1. An XML invoice is saved in a document NoSQL database, which retains the internal structure of the invoice
  2. The user only requires the purchased items to be returned from the database.
  3. The database successfully returns only the list of the purchased items from the invoice.