SoftWell: Navigator bus

Product Concept

The black box model was chosen as the logical model for combining systems. Its essence is that the systems included in the complex do not know anything about the internal architecture and database structures of other systems.

Taking into account the requirement to combine an arbitrary number of previously undefined systems from various suppliers into a single complex, and considering that there is no need to perform combined transactions within the complex, information exchange in asynchronous mode is implemented.

MSMQ (Microsoft Message Queuing) is selected as the transport environment.

Each of the systems during the initial registration in the complex informs the other a list of information that it is ready to supply for them on request. In addition, systems can publish information structures that can be transferred by other systems for use (additions, modifications, use in calculations, etc.).

To store such information and provide it to all systems, the complex has a Dispatcher. All systems publish publicly available information on the manager and receive descriptions of other systems from it. The manager can send notification messages to the systems about certain events.

Considering that the complex should include both systems developed under the contract, as well as inherited (developed earlier) and systems that can be developed by other suppliers, the interaction between systems through MSMQ is assigned to the so-called adapters, the requirements for which and their interfaces are developed within the framework of the project.

A different adapter is being developed for each system. Functions assigned to adapters: Interaction with the bus manager (system registration, obtaining a description of other systems, obtaining the information necessary to start an interaction session in accordance with the protocol, etc.) Converting received requests from other systems into a format that is understandable to your system. Generate output sets (according to the received requests) and convert them into an intermediate exchange format that is understandable to all systems. Generating notification messages about events that occurred in their system and converting them into an intermediate exchange format that is understandable to all systems for transmission to systems that subscribe to receive such information. Generate confirmations of receipt of information (if such a requirement was contained in the document) in an exchange format that is understandable to all systems. Maintain a delivery record of information requiring confirmation (in accordance with the request received) and a delivery mark (based on confirmation documents). Processing information transmitted by other systems for use within the system (if supported and described in the manager). Processing notification messages from the manager. Preparation of data for use in other systems when certain events occur in its system for transfer to systems that subscribe to receive such information. Transfer information about the current state of your system and your system to the bus manager for centralized accumulation and storage.

Considering that the protocol of interaction with MSMQ is the same for all adapters and documents moving in a complex should be understood by all systems (intermediate format), an object was created within the framework of the project that implements interaction with MSMQ. Interfaces of this object are known to adapters of all systems and xdr files have been developed that describe the structure of inter-system exchange documents (intermediate format) and the structure of information about systems registered on the bus manager. However, system adapters can independently implement the mechanism of working with MSMQ. A mandatory requirement is the ability to understand and prepare documents in an intermediate format.

To ensure system stability, it is possible to distribute queues and adapters to different computers. In order to increase the stability of the system and reduce the likelihood of information loss, a number of structural and systemic measures have been taken. In addition, in order to ensure urgent exchange of information and in case of any problems with the delivery of information through MSMQ, direct access interfaces to them (bypassing MSMQ) are provided for the adapters, for which all the necessary information is also stored on the bus manager. However, even in this case, the exchange is carried out through documents in an intermediate format.

In addition to XML format, adapters can support other, so-called binary formats. In this case, the format must be uniquely described or, together with the adapter, an object intended to work with this format should be supplied. Preparation by a document in binary format should be an alternative preparation of documents in XML format, and not replace it, i.e. the adapter can additionally be able to prepare and process documents in binary format. The number of binary formats supported by the system is not limited and is determined only by the system developers. When a new binary format appears, the system must transmit the relevant information to the bus manager to enable the corresponding library to be registered with the new format in other systems.

Thus, with this technology, it is possible to combine an arbitrary number of arbitrary systems (including inherited ones), for which it is enough to develop an adapter in accordance with the requirements set forth in this document.

Basic Principles

• Simultaneous or sequential implementation of systems to be combined;
• Already implemented and combined systems do not know anything about the systems that will appear after them;
• The need to ensure both synchronous and asynchronous exchange between systems
• All information that the system can provide to other systems is combined into objects - named logically completed entities (for example: transactions, securities, counterparties, etc.);
• Descriptions of all objects of each of the systems should be stored in a single storage to ensure equal access to them both existing systems and developers of new systems;
• A call to the system for information shall be formulated in terms of this system;
• Guaranteed delivery of information;
• The need to combine, including already implemented systems;
• Systems do not know anything about the internal architecture and structure of databases (and the fact of their presence) of other systems;
• The list of systems to be combined is not defined in advance;
• There is no need to perform combined transactions (follows from the sequential appearance of systems).

General architecture

Each system must register with Bus Navigator. To be registered

• general information about the system and
• description of system objects.

Interaction with the Dispatcher

The main tasks of interaction are system registration, obtaining a description of other systems, obtaining the information necessary to start an interaction session in accordance with the protocol, and others:

• Converting received requests from other systems into a format that is understandable to your system.
• Generate output sets (according to the received requests) and convert them into an intermediate exchange format that is understandable to all systems.
• Generating notification messages about events that occurred in their system and converting them into an intermediate exchange format that is understandable to all systems for transmission to systems that subscribe to receive such information.
• Generate confirmations of receipt of information (if such a requirement was contained in the document) in an exchange format that is understandable to all systems.
• Maintain a delivery record of information requiring confirmation (in accordance with the request received) and a delivery mark (based on confirmation documents).
• Processing information transmitted by other systems for use within the system.
• Processing notification messages from the manager.
• Preparation of data for use in other systems when certain events occur in its system for transfer to systems that subscribe to receive such information.

• Transfer information about the current state of your system and your system to the bus manager for centralized accumulation and storage.

Adapters

Adapters of all systems are executed as DCOM applications. This provides the possibility of their remote launch (including other adapters) to ensure the ability to quickly respond to requests.

Synchronous exchange mode is implemented using a remote call to the methods of interfaces of adapter objects (a list of such methods and their parameters are defined and mandatory for all adapters).

The exchange of information, as well as the description of systems, is carried out by XML documents. The schemas of these documents are part of the Bus Navigator.

Thus, the adapter interacts both with Bus Navigator, receiving from it and supplying the necessary information to it, and with its application, receiving from it requirements for requests for certain information from other systems and supplying it to its application. The procedure for transferring information from other systems received from Bus Navigator to its system is determined during the implementation of the adapter and no additional requirements are imposed on it. The system itself interacts only with the adapter, for it all other sources of information and there is an adapter. An example of system construction is given in the following diagram:

(Click on the image to view the diagram in full size.)

We emphasize that all information exchange between systems is carried out through a bus (Bus Navigator).

The bus is the queue manager (MSMQ in this case) and the Manager. The manager is implemented in the form of a DCOM application and stores in its database the information necessary to ensure intersystem interaction about all systems. This information includes:

• Unique identifier of the system.
• Version of the system description.
• System description date.
• Natural language system name.
• Description of the system for understanding by developers of other systems.
• System adapter GUID.
• The name of the computer with the system adapter.
• The name of the system input queue.
• The name of the system output queue.
• Computer name with system queues
• Description of system objects (tables, procedures, functions, etc.) containing or creating information that the system is ready to publish or accept for use in its business logic.

Each object contains the following description:

• Name of the published system object.
• The type of the published system object (table, COM method, stored procedure, function, script, abstract object, i.e. an object that does not have an exact correspondence in the space of methods and properties of the system, but describes some general concepts of the system)
• The return type of the object (for ObjectType = Method, Function, or Script).
• The name of the object is in natural language.
• Description of the object for understanding by developers of other systems.
• Object properties (table fields, method parameters, procedures, functions, etc.). Each property of an object is described by the following attributes:
• The name of the object property.
• Whether property values are unique within the system (only for table fields and abstract objects).
• Object property type according to urn: schemas-microsoft-com: datatypes.
• Whether the property is an input or output parameter (only for method, procedure, function, and script fields).
• Unique property ID of the object in the dispatcher database (assigned by the dispatcher during registration).
• The name of the property is in natural language.
• Description of the property of the object for understanding by developers of other systems

General description of adapters

The structure of the system adapters and their interaction with other components of the system and the complex as a whole can be represented by the example of the VaR adapter in the form of the following diagram:

(Click on the image to view the diagram in full size.)

The modules actually belong to the adapter

• 1.2.1 (Business Logic Module) and
• 1.2.2 (Bus Navigator interaction logic module).

The business logic module is unique for each application and may not exist in general (for example, when implementing a two-level client-server model). The module of interaction logic with Shina actually implements the exchange of information. Its functions include:

• Obtaining from the Manager the necessary information for organizing exchange between systems (names of queues, computers with queues and adapters, GUIDs of adapters).
• Formation of documents placed in queues in accordance with the accepted scheme.

Receiving documents from other systems, analyzing them, performing the necessary actions both in accordance with the adopted exchange protocol and in accordance with the internal logic of the system, converting (if necessary) the data received from other systems into an internal format.

Transport module

The transport module is an embedded COM object and provides interaction with MSMQ, for which it provides the adapter with an interface.

Document Structure

Title. Includes:

• Document number;
• The system that prepared the document;
• Document type (standard or binary);
• Action described by the document:
• Request;
• Response to the request;
• Notification of an event in the system (change, delete, add information)
• receipt (confirmation of receipt of information);
• notification of an event in the system with simultaneous transmission of information (modified, deleted, new);
• service information;
• confirmation of receipt of information in the data notification document.
• Urgency;
• Transactionality;
• Other service information.

Document body

May be empty or include the following information:

• Objects (requested or sent in response to a request or with a notification message);
• Conditions for querying objects (in documents requesting information).