Fan-out Ingress (Buhler, Erl, Khattak)
How can streaming data arriving from a single source be simultaneously forwarded to multiple destinations without any delay?
An event data transfer engine with fan-out capability is used to capture an incoming stream of data. The event data transfer engine is configured with multiple sinks. Once configured, the incoming stream of data is automatically replicated and then forwarded to the configured sinks.
Functionality is added to capture the incoming stream in realtime, create inflight copies of the data stream and forward each stream to the configured data sink. Some basic level of data cleansing is performed before the stream is replicated. The application of the Fan-out Ingress pattern not only removes data provisioning latency but also provides a scalable realtime ingestion functionality.
In the diagram, a data stream needs to be forwarded to an external system, a processing engine and a storage device. The stream is collected and forwarded as follows:
- The stream is captured by an event data transfer engine.
- The event data transfer engine creates three copies of the stream as it arrives.
- (a,b,c) The copied stream is then forwarded to the external system, processing engine and storage device as a continuous stream.
- This results in a low data provisioning latency in terms of the time it takes the data to reach the sink from the source.