QuantumSI News:

QSI's Product Page

Products

MCB Reliable Data Collection, Mediation, Buffering and Distribution

MCB, Reliable Data Collection, Mediation, Buffering, and Distribution

_here_

QuantumSI has slashed the cost of deploying and maintaining multiple data collection systems with it's revolutionary simple Multi-Cast Buffer (MCB) system. This mediation system is composed of a high reliability UNIX daemon application set that provides an elegant solution to sharing critical data feeds among mission critical processes that must access a single shared data source or feed.

This exciting technology enables applications to be deployed and that share the same data feed, more importantly does so while increasing reliability and both centralizing and minimizing management tasks across all enterprise applications.

The MCB is typically deployed in mission critical roles in either a Telecommunications or a Manufacturing environment where data loss can not be tolerated - yet there are multiple consumers (automated or manual) for the critical data stream.

The overall objectives that have been meet by Quantum's generic multi-cast buffer (MCB) system are: 1) isolate and minimize the critical system components; 2) avoid data losses by any non-critical process failure and/or scheduled restart; and 3) avoid non-reversible data transformations. [ PDF ]

Benefits

Increase Reliability Eliminate or minimize critical single points of failure in either machines or processes and thus localize the critical machine and/or processes to a highly reliable infrastructure. This design point increases the overall reliability of the Enterprise computing infrastructure.
Limit/Minimize
Critical Infrastructure
Implement applications and procedures which fall into the non-critical category that may fail for any reason foreseen and unforeseen (scheduled or unscheduled) yet recover without data loss. By allowing a history of data to be re-requested (or automatically replayed) vast areas of the Enterprise IT infrastructure that are now 7/24 critical can be moved into a lower and less critical tier of computing reliability requirements.
Increase Recovery
Capabilities
Store data be in an original unaltered format (or alternatively) have the ability to quickly process and regenerate the original raw input format. In the event of a downstream application logic flaw, the MCB allows the same data to be requested and processed after implementing a correction to the Enterprise business rules that are being applied to the data.
Simple to Use The MCB system can be easily configured and deployed via self installing distribution packages.
Reliable and Proven Deployed and working in over 1000 application instances in over 60 sites both domestic and international settings.
Internet/Intranet compatible The same data that is streamed to mission critical applications if specific access is authorized can be viewed by simple tools like Telnet from within the Enterprise Intranet or even the Internet.
Flexible Allows automatic "chat" scripting and complex login and logout sequences into production equipment to maintain the proper data reporting modes in the event of system maintenance and or failure/recover of the monitored device..
Benign Allows support personnel to temporarily override data streaming and perform maintenance via a shared I/O source. Intelligently manages the "login" token and in the event of normal disconnect and/or inactivity will automatically reestablish the correct data reporting mode.
Secure Utilizes Radius and LDAP authentication via plug-in modules to support highly secure data environments.

MCB Deisgn Concept

The design philosophy of the MCB dictates that all critical data sources be loss free (with respect to a reasonable service window from 1-2 hours to 1-12 days) this period is dependent upon the amount of allocated disk space for a particular MCB and the needs of your organizations recovery and/or backup windows.

The second objective of the MCB requires that all non-critical processes or readers of any such critical data have built in mechanisms to startup, recover and synchronize from a robustly maintained check point when reading the data. Alternatively a reader can be set up specifically to recover in a known fashion Any downstream application which restarts can request a specific MCB to stream data in specialized recovery modes - for example send up to the last ten (10) minutes in 3X real-time and then transition into a real-time emission domain. If this is unfeasible Quantum provides a helper application which manages the check pointing for a "dumb" process.

The last objective implies that any data which is buffered at the critical source, or archived from the point of initial buffering, never be subjected to a non-reversible transformation. This requirement is critical if a flaw in a downstream processing rule is uncovered, perhaps in the event of an upstream system or software upgrade, your organization now has a window of recovery that wasn't previously available.

The below diagram illustrates the use of design objectives one and two above. The only critical part of the processing stream is the MCB system. An executable instance of the same generic MCB system is run to collect and store N days worth of data from each data source. Only the host machine and the MCB systems need to be absolutely reliable. The reliability requirements of the remaining downstream infrastructure for processing the data are greatly relaxed.


_here_

Isolate and minimize critical components

All other processing on the non-critical machines can be built to recover from failures without data loss and any such failures do not impact other processes which access the same data source. Thus the MCB cleaning removes depandancies between procesisng stages and eliminates data "ownership" issues.

Supported Data Input Interfaces

The MCB system directly supports the following input interfaces:

The MCB system directly supports the following output interfaces:

In either interface domain the TCP/IP connectivity allows data to be accessed from any machine in the Enterprise LAN or WAN. In addition either raw (e.g. binary) or text (ASCII) input feeds are supported by the MCB system.

Operational Overview

Replication of a limited data resource

First the MCB system solve the distribution problem of a limited data resource feed. The MCB effectively:

Historical reporting

Second the MCB system allows requests to be dynamically negotiated. A dynamically negotiated request might involve a "loss free" archiving or reporting system. In this case a process would ask the MCB to provide all data from a known "checkpoint" and then provide data in a real-time fashion.

This type of system is targeted to support "loss free" historical (e.g. non real-time) analysis of the various system data feeds. In particular the needs of data completeness by systems such as logging, paging and other applications are easily meet.


Real time analysis

Third the MCB system allows requests to be statically setup in a non-negotiable fashion. This style of connection would support the needs of the production OSI/NetExpert feed.

As a third party application such as OSI's NetExpert can not request a missing segment of data from the MCB system. Thus a "helper process" shall manage the check pointing and provide a canned recovery procedure. It is currently envisioned that the "helper process" should try to replay the last 10 minutes of data and then automatically transition into supplying a real time to NetExpert.

Although not "loss free" this recovery method via a "helper process" meets the needs of supporting a high reliability real-time feed in the event of short planned or unplanned "application" system restarts.


Scenario Replays

Forth multiple MCB system support the ability to replay segments of buffered data in real time to recreate event scenarios. This ability facilitates the following:

Functional Overview

Data Store

The basic feature provided by an instance of the MCB system is the ability to map virtual FIFOs onto a physical FIFO. Once such possible mapping is shown in the diagram below. Given two (or more) processes each locally maintaining an indicator of the last item read a virtual FIFO is formed. The processes can access any time window on demand as long as the earliest requested item is within the currently buffered space of the physical FIFO.


_here_

Each process maintains its own persistent marker and thus consumes data at its own rate provided that it 1) on average consumes data faster than it arrives and 2) is never shut down for a period in which the sliding window of data in the physical FIFO scrolls past the persistent marker.

The data in the underlying physical FIFO is stored in a first in first out fashion and is eventually discarded or archived. Note the archive process is simply another reader process.

No interaction or side effects occur between the N readers, as these are complete independant processes reading from an immutable first in first out datastore.


Data Access

The core system supports access via a TCP/IP host/socket interface which is compatible with 1) any serial port or serial terminal server 2) the OSI NetExpert gateway system and 3) any program which can be executed.

The major advantages of using a socket based interface in the MCB system are as follows:

Status and Control

The core system supports access via a Java based status and control daemon that functions across the LAN, WAN or even across the Internet. This GUI allows either operators and privileged administrators to perform many tasks including but not limited to:

Screen shots of the MCB Control/Status GUI

A screen shot of the main window of the MCB Control/Status GUI,which can be used to manage hundreds of distributed MCBs from a central location, is shown below:

Several screen shots of the configuration sub-window of the MCB Control/Status GUI are shown below:



 

Copyright 1995-2019 Quantum Systems Integrators, Inc. All Rights Reserved. 

Select an Area: