general.tape-socket Definition/General


Command general.tape-socket Definition/General
Applicable release versions: AP/Unix, AP 6.1
Category General (155)
Description defines a tape system across a network. This section is an introduction to the functionality of the tape over a network sub-system.

Documentation Structure

The tape system over a network documentation is divided into the following entries in the Advanced Pick Reference Manual:

- "tape socket, General": (this section) General introduction about the tape over a network, its capabilities and the fundamental notions.

- "network save/restore, General": How to use the tape over a network utility to do simultaneously a full save on one system, and a full restore on another.

- "tape-socket, TCL": Detailed reference of the TCL command "tape-socket".

Overview :

The objective is to establish a 'pipe' between two systems through which tape blocks can be transferred: On the sender's side, the tape process (eg T-DUMP) writes the tape blocks into a 'pipe', and, on the receiver's side, the associated process (eg T-LOAD), reads the blocks and processes them, as shown in the following figure:

+---------------+ +---------------+ - - - -
| +---+ | | +---+ |
| | T | | | | T | | Pick
| +---+ | | +---+ | VM
| | | | | |
+---|-----------+ +-----------|---+ - - - -
| | | | | |
| +-<-========|<-[netw]<-|========-<-+ | Unix
| pipe | | pipe |
+---------------+ +---------------+ - - - -

However, it is not possible to send data directly from a pipe across a network. This is solved by creating SERVERS on both the sender and receiver. These servers work closely together to establish a communication path and maintain it. This principle is represented by the following figure:

+---------------+ +---------------+ - - - -
| +---+ +---+ | | +---+ +---+ |
| | T | | P |-|<-[netw]<-|-| P | | T | | Pick
| +---+ +---+ | | +---+ +---+ | VM
| | | | | | | |
+---|-------|---+ +---|-------|---+ - - - -
| | | | | | | |
| +-<-=-<-+ | | +-<-=-<-+ | Unix
| pipe | | pipe |
+---------------+ +---------------+ - - - -

The principle is to have phantom processes on both the sending and the receiving systems establishing a network communication, and exchanging data with the tape processors through a Unix pipe. On the receiver's side, 'T' is the process reading from tape, doing a T-LOAD, for example, and 'P' is the phantom process handling the network reception.
On the sender's side, 'T' is the process writing to tape, doing the T-DUMP, and 'P' is the phantom process handling the network emission.
On each machine, a Unix pipe is set up to allow data exchange between the phantom process and the tape process. This pipe must be set up in the Pick configuration file of each machine as a 'network' device (type 'c').

Optionally, the system can be linked to the transaction log system, to automatically create a 'hot backup' system where all updates to a main machines are sent over the network to a backup system, which is thus a mirror image of the main system, ready to take over the application in case of failure of the main system.

The network is accessed through the BSD 'socket' library.

Servers :

The phantom processes on each system act as a server for the other system. In the following, the phantom process will be called "input server" on the receiver's side and "output server" on the sender's side. On each system, the servers are identified by a "server name". The System Administrator defines all working parameters at install time, and whether the servers should be linked automatically to the transaction logger. On most configurations, there will be only one server. Therefore, the System Administrator would use the 'default' server name, which is used in the menus.

Features :

- Automatic startup of Servers.
- Automatic reconnection of Servers in case of network failures.
- Synchronization of clocks when Servers are started. The sender system executes a remote "set-date" and "set-time" on the remote system with its own time and date, to ensure they are in sync. This can be prevented using the (S) option of the "tape-socket" TCL command.
- Optional tracing of all network traffic.
- Optional automatic 'link' to the Transaction Log sub-system to have an automatic startup of both systems.
- Optional interface to the transaction logger to regularly test that the transaction logger is operating properly.
- Optional notification to a designated Pick user of network and transaction logger incidents.
- Remote commands can be sent from the Output Server to the remote Input Server. This allows some degree of remote system administration of the Input Server.
- Automatic installation of the software the first time the command is used.
- Simple menu interface for management of the default Server and TCL interface for use in macros.
- Short, rotating, log file, for the most recent error messages, and separate permanent log for long term monitoring.

Automatic Startup :

Once a Server has been set up (see the section "Server Setup" below), a simple TCL command starts it. This command can be inserted in the 'user-coldstart' macro. If the Server has been linked with the Transaction Logger (see below), and if the Server is the Output Server, the Transaction Logger is also started, or re-started, automatically as a phantom process. If the Input Server is linked to the Transaction Logger, then a transaction restore is started automatically on the terminal doing the 'start' command, in which case this command, if inserted in the 'user-coldstart' macro, should be the very last command.

Automatic Connection of Servers :

By default, the Servers are trying to communicate with each other, so that the connection is established automatically. In case of network problem, the Output Server tries every 5 seconds to re-establish the communication. When the communication is broken, and cannot be re-established in less than 5 seconds, a message is sent to the System Administrator. After this, another message is sent when the communication is re-established. There is normally never a need to stop the Input Server. The Output Server can then be stopped and restarted.

Linking to the Transaction Logger Sub-System :

This feature allows to completely control the Transaction Logger (sender) and the Transaction Restore (receiver) by the tape-socket Servers, in a 'hot backup' configuration. Once the Input Server has been started on the backup system, all operations can be done from the main system: starting the Output Server starts the Transaction logger; stopping the Output Server detaches the tape from the transaction logger, and sends a message to the input server, so that the Transaction restore can be restarted automatically. The Transaction Log Test Polling feature allows detecting transaction logger incidents, by notifying the System Administrator that the test item has been received in time by the backup system (see more on this below). Before starting or stopping the Transaction Logger, the Output Server makes several controls, to be sure the Transaction Logger is ready. If not, the process will be cleaned up, and, if required, a new Unix process will be spawned.
It is important, once the Servers have been linked to the Transaction Log Sub-System, to start/stop the transaction logger and/or restore EXCLUSIVELY through the tape-socket server start/stop commands (or through the menus). Failure to do so, will generate error messages to the System Administrator who will have to do some manual intervention to correct the situation. The 'txlog' Transaction log menu can still be used to obtain the status of the transaction logger, but NOT to start, stop or detach the tape. If it necessary to release the transaction log queue, and, to do so, use the txlog menu, stop the Output Server FIRST, else, not only data will be lost, but the transaction restore will de-synchronized and will have to be restarted manually.

Transaction Log Test Polling :

Periodically, for example every 10 minutes, a test item 'test' is written in a test file "dm,ts.log,test', created automatically. This item contains unique information, and a time/date stamp. If the Transaction Logger and Restore are working properly, this test item is transferred into the same file, on the backup system where the arrival time and data are recorded in the item. Before updating the item, however, the Output Server sends a message to the Input Server to check whether the PREVIOUS test item arrived correctly. If so, everything is fine, and the Output Server is able to calculate an approximate transfer time (which includes not only the actual transfer time, but also the time the test item stayed in the transaction log queue). If the item did not arrive, a message is sent to the System Administrator. Note that, especially if the polling period is short, it is possible for the test item to still be in the queue, and, therefore, make it appear that it was not transferred. If the queue is really large, it is possible that it may take a long time before the test time progresses to the top of the queue before being transmitted.

TCL Command :

The "tape-socket" TCL command is used to create the input or output server and control their activity. See the section "tape-socket, TCL" for more information on the command.

System Setup :

The details of the system setup may vary depending on the application (hot backup, network save/restore, ...). The following are just general guide lines. See the appropriate Advanced Pick Reference Manual documentation.

- Establish a network between the two systems. This network must support TCP/IP (eg Ethernet, Token Ring, etc...). The System Administrator must set the network names of both systems, even though only the receiver's host name is used.

- Determine a free TCP/IP port number. Use the "netstat -a" Unix command to see what is currently in use. A value like 2000 or 3000 is usually safe.

- Create Unix pipes on both system, using "mknod", and make sure they have appropriate permissions. Put these pipes as 'pseudo-tapes' in the Pick configuration files of both systems.

- Setup the Servers on both systems (see the section "Server Setup" below).

- Start the Input and Output Servers.

Server Setup :

The "tape-socket" menu has a "Server Setup" option which asks for the necessary running parameters. Some basic elements have to be defined by the System Administrator for the system to operate. The "Server Setup" option asks the following elements (if values are already defined, they are displayed between brackets, and typing <return> leave them unchanged):

Server Name
Using the "other server" submenu, it is possible to manipulate more than one server. A server name is any alphanumeric string ('.' are not allowed), of up to 8 characters. The default server name is 'tserver'.

Server Type
Enter 'in' for the Input Server and 'out' for the Output Server. This value is required.

Remote HOST name
Enter the network name of the system on which the Input Server will reside. This parameter is required for the Output Server setup, to know on what piece of hardware the Input Server will be started. The host name must be declared in the Unix file "/etc/hosts".

TCP/IP port number
Select a valid (>1024, <32767), unused TCP port number. Use the "netstat -a" Unix command to find out which ports are currently used on the system. Both Servers MUST agree on this value, else they will not be able to communicate. This value is required.

Currently, only "inet" (Internet Protocol) is supported. This value is required.

Unix pipe name
Select a valid Unix pipe name (eg /dev/tape). If it does not exist, it must be created, and added in the Pick configuration file, if not already there, using the TCL command "config tape", for example. This value is required. See the example in the "tape-socket, TCL" documentation.

Number of trace messages to keep
Log entries are kept in small circular buffer that can be displayed with the "Status" command. Select a number from 4 to 16 for example. All messages (except periodic 'OK' messages) are also kept in the Permanent Log.

Pick user to notify in case of errors
Enter one or more Pick user ids, separated by commas, the system will send a message to, in case of error. If left empty, then 'dm' or 'sysprog' will be notified. In case of repetitive errors, these messages can become a nuisance, and suppressed later. However, it is STRONGLY advised to select at least one user who would always be logged on to the system so that errors will not go unnoticed. The whole system normally operates without much human intervention, and if notification is disabled, or if the user to notify is not logged on, communication can be interrupted for a very long time before an error is corrected. Errors are also logged into a file. To select all users on the system, enter an asterisk (*). To send a message to a line number, whether it is logged on or not, use an exclamation mark followed by the line number in decimal (eg !0). To disable the notification, enter 'off'.

Transaction Log test polling period
Enter the value in seconds, or in HH:MM:SS of the period with which the Output Server will test the operation of the Transaction Logger. If the Transaction Logger is not used, enter 0 to disable the polling. Refer to the section "Transaction Log Test Polling" above for a detailed discussion of this mechanism. An appropriate value would be between 00:01:00 (1 mn) to 01:00:00 (1 hr). A smaller period could create false alarms by long delays due to the size of the queue. A longer period means that an incident would be unnoticed for a long time. The rule of thumb is that if the normal data update rate is small, then the test period should be small too. In case of mass transfers, like when doing a "touch" (see the TCL command "touch") of an entire data base, which would generate an enormous amount of data transfer, it is advised to temporarily increase the polling period to a large value (say 1 hr), so that the Output Server does not 'panic' at not seeing its updates of the test item go through the network.

Start Transaction logger
This parameter determines whether the Server is to be 'linked' to the Transaction Log Sub-System, to set a 'hot backup' configuration. Type 'on' to establish the link, and 'off' to disable it. When 'on', the Servers will attempt to control the Transaction logger and restore. When selected, be sure to understand the relationship of the two sub-systems as discussed in the section "Linking to the Transaction Logger Sub-System" above. The most important rule is to control the transaction logger EXCLUSIVELY through the "tape-socket" menus or commands, to make sure the Servers are aware of the states of the Transaction Log Sub-System.

Log (DL) files or ALL
This parameter is required only for the Output Server if it is 'linked' to the Transaction Logger. It determines the options with which the Transaction Logger will be started when the Output Server is itself started. Enter 'DL' to log only the files for which the attribute 1 of the D-pointer has been changed to have a 'L' code (see the TCL command "set-dptr" for one way of doing this). Enter 'ALL' to log all updates to all files. Be careful with this second option, since updates to system files, like the 'abs', 'users', etc.. will also be logged.

Transaction Log queue period
This parameter is required only for the Output Server if it is 'linked' to the Transaction Logger. It determines, in seconds, the time after which an update is actually transmitted to the remote system. Enter 0 for the default value (currently 30 seconds). When dealing with a fast network, a typical value would be 2 or 3 seconds. This parameter means that an update to a file would stay on the master system for 2 or 3 seconds, for example, before being sent to the remote system. In case of system crash, only these updates would be lost. On AP versions prior to 6.1, always enter 0.
Related transaction.logger