7 News Support Utilities

7.1 Exchange of Items via NNTP Over DECnet or TCP

The utilities described in this section are used for exchanging items with other sites using NNTP protocols over high-speed DECnet or TCP links. These utilities are designed for transferring batches of items as part of incoming and outgoing newsfeeds. Retrieval or posting of individual items by a remote client is done via the local NNTP server, which is described elsewhere in this manual.

7.1.1 NNTP_Xfer

NNTP_Xfer is a utility used to retrieve news items from a remote NNTP server to the local system using the NNTP transfer protocols over a DECnet or TCP connection. Specifically, the local system sends NEWNEWS messages to the remote NNTP server, requesting that system to transmit the message identifier strings of items recently received at that site. The local system may then respond with a subsequent sequence of ARTICLE requests, to retrieve those items not held locally. This utility essentially makes the local system the active partner in the transfer. Because it is less efficient than NNTP_Xmit, and because the NEWNEWS command can impose a heavy load on the remote NNTP server, this method of news transfer is not recommended unless it is impossible for the remote site to run NNTP_Xmit to feed new items directly to you.

7.1.1.1 Installation

NNTP_Xfer.Exe is created in the News_Dist directory when NewsBuild is run. You should copy this image to a location, such as News_Manager, where it is accessible to the News maintenance job which will retrieve items from upstream sites, but is not accessible to other users. This job should invoke NNTP_Xfer using a foreign command definition, such as
$ NNTP_Xfer == "$News_Manager:NNTP_Xfer"

7.1.1.2 Command Syntax

NNTP_Xfer can be invoked with a DCL-like syntax, using qualifiers to pass information to the image, with a U*ix-like syntax involving simple command line switches, or with an abbreviated syntax requiring three parameters. You may use either syntax, but you may not mix the two within the same command. In either case, NNTP_Xfer should be invoked via a foreign command definition; it will parse its own command line. The interpretation of all parameters is case-insensitive.

Sample DCL-like syntax:
$ NNTP_Xfer/Node=remhost/Protocol=transport/File=grpfile - [/Groups=grouplist][/DayOffset=days][/HourOffset=hours] - [/Task=decnet-task][/Stream][/LogFile=log-file][/Debug]

Sample U*ix-like syntax:
$ NNTP_Xfer -n remhost -p transport [-g grouplist] - [-d days] [-h hours] [-t decnet-task] [-s] - [-l log-file] [-o]

Sample abbreviated syntax:

$ NNTP_Xfer remhost transport log-file
If the first command line argument does not begin with '-' or '/', NNTP_Xfer assumes this abbreviated syntax is being used, and will interpret the command line arguments accordingly. As a consequence, the following is an illegal command
$ NNTP_Xfer cicero decnet News_Manager:Cicero_Xfer.Log -g misc.legal.*
(The correct syntax would be
$ NNTP_Xfer -n cicero -t decnet - -l News_Manager:Cicero_Xfer.Log -g misc.legal.*
in this case.)

/Node=remhost -- the name of the remote host. This is the actual address to be used in establishing the connection, not the name that site uses in news item headers. For DECnet connections this should be the DECnet nodename of the remote node, or its node number in the format area*1024+node (area*1024 is optional if the server is in the same area as your system). For TCP connections it should be the FQDN of the remote host, or its IP address in dotted decimal form.
[U*ix-like syntax: -n remhost Abbreviated syntax: first parameter]

/Protocol=transport -- the transport protocol to be used when connecting to the remote node. This value should be 'TCP', or the name of the TCP transport you specified when building News, if you want to establish a TCP connection to the remote NNTP server; NNTP_Xfer uses the standard NNTP port (119). If any other value is specified, NNTP_Xfer will attempt to establish a DECnet connection.
[U*ix-like syntax: -p transport Abbreviated syntax: second parameter]

/File=grpfile -- file specification of a file containing the list of newsgroup patterns to be passed to the remote server. Grpfile should be a text file containing newsgroup patterns separated by commas or record boundaries; a pattern cannot span a record boundary. All records beginning with the character '#' are considered to be comments, and are ignored. The list of newsgroup patterns read from the file is treated just as if it had been entered using the /Groups qualifier (except that files cannot be nested using the '@' for indirection).

If both /File and /Groups are specified, the /File qualifier is ignored. If neither /Groups nor /File is specified, then all newsgroups on the remote system are searched.
[U*ix-like syntax: -f grpfile. If both -g and -f are present on the command line, or if either is specified multiple times, only the last occurrence is used.]

/Groups=grouplist -- a comma-separated list of newsgroup patterns specifying the newsgroups about which information is to be obtained. If the DCL qualifier /Groups was used, and first character of a newsgroup pattern is '@', the remainder of the pattern is treated as a file specification, as though it had been specified using the /File qualifier. The text of grouplist is converted to lower case, and all occurrences of '_' are replaced by '!'. The list is then passed to the remote NNTP server as the first parameter to a NEWNEWS command, and the server interprets the patterns. According to the NNTP standard, the following rules should be applied to each pattern:

all newsgroups whose names match a pattern in grouplist should be searched for new items, as specified by the other parameters to the NEWNEWS command.
the character '*' is a wildcard which matches any number of characters in a newsgroup name
when being compared to newsgroup names, each pattern has the characters '.*' appended to it.
if the first character in a pattern is '!', then any newsgroups whose names match that pattern should not be processed as part of the NEWNEWS search. Many servers will follow the same rule used to interpret patterns in News.Sys entries: a more specific match takes precedence over a less specific one. Thus the pattern comp,!comp.os,comp.os.vms would include all comp.* newsgroups except the comp.os.* groups, but would include comp.os.vms.

The total length of the NEWNEWS command cannot exceed 512 characters, so grouplist should not be longer than 487 characters.

If /Groups is not specified, or if no grouplist value is given, all newsgroups on the remote system are searched. If /Groups is specified, the /File qualifier is ignored.
[U*ix-like syntax: -g grouplist. If -g is specified, grouplist must be specified. The '@' character does not receive the same special interpretation as it does following the DCL-like /Groups qualifier; use the -f switch (described under the /File qualifier) to process files containing lists of newsgroup patterns. If both -g and -f are present on the command line, or if either is specified multiple times, only the last occurrence is used.]

/DayOffset=days -- NNTP_Xfer maintains a record of the time at which it last contacted a system, and as part of the current NEWNEWS command, requests items which have been added on the remote system within days days before the last contact. Thus, if days=0, NNTP_Xfer requests items added on the remote site after the time of the last contact, while if days=1, NNTP_Xfer requests all items appearing at the remote site since one day before the last contact. If the /DayOffset qualifier is not specified, days defaults to 0.
[U*ix-like syntax: -d days]

/HourOffset=hours -- This qualifier functions similarly to the /Days qualifier, except that hours specifies the number of hours before the last contact to offset the time specified in the NEWNEWS command. This can be particularly useful since NNTP_Xfer records the time of the last contact in local time, while the NEWNEWS command requires that time be specified as GMT. Therefore, if your local timezone is two hours ahead of GMT, you should specify an hours value of 2 in order for the NEWNEWS command to properly correct its recorded time to GMT. If the /HourOffset qualifier is not specified, hours defaults to 0.
[U*ix-like syntax: -h hours]

/Task=decnet-task -- the name of the DECnet object through which the NNTP server on the remote system is contacted. If the /Protocol value specifies a DECnet connection, NNTP_Xfer determines the remote task name as follows:

if /Task is present, decnet-task is used. If decnet-task does not contain the character '=', the string "TASK=" is prepended to it. Decnet-task is converted to uppercase before being used.
if /Task is not present, an attempt is made to translate the logical name News_remhost_Task, and if this succeeds, the translation is used.
if neither of the above cases is true, a default name of "TASK=NNTP" is used.

You should ask the remote site for the NNTP server's task name when making arrangements for a news feed, and pass the appropriate string to NNTP_Xfer either on the command line or via logical name. In particular, the names "TASK=NNTP" or "0=NNTP" are common on VMS systems, and "NNTP=" is common in Ultrix systems.

If /Protocol specifies a TCP connection, the /Task qualifier is ignored.
[U*ix-like syntax: -t decnet-task If the U*ix-like syntax is used, decnet-task is not checked for the '=' character, and "TASK=" is not added.]

/Stream -- specifies that stream input calls should be used over a DECnet connection, rather than record-based input calls. This is normally necessary only when connecting to a NNTP server running on an Ultrix system. If /Protocol specifies a TCP connection, the /Stream qualifier is ignored.
[U*ix-like syntax: -s]

/LogFile=log-file -- file specification of a file to which a one-line summary of the current session will be appended. This summary will list the time, remhost, grouplist, the number of message IDs returned by the remote system, and the number of items actually transferred.
[U*ix-like syntax: -l log-file Abbreviated syntax: third parameter]

/Debug -- causes status messages tracking the progress of the session to be output to the terminal. In addition, all traffic over the connection to the remote server is echoed to the terminal.
[U*ix-like syntax: -o]

In addition, if NNTP_Xfer is invoked with the U*ix-like command line switch -he, a listing of the U*ix-like switches, with brief explanations, will be printed, and NNTP_Xfer will exit. There is no DCL-like equivalent for this option.

7.1.1.3 Transfer Session Summary

After parsing the command line, NNTP_Xfer checks to see whether there are any unprocessed message-ID files left over from previous sessions which terminated abnormally. NNTP_Xfer checks for old message-ID files of the same type as the current session, so, for example, if the current session is a selective search, then NNTP_Xfer will only check for message-ID files from prior selective searches of this remote site. If any old message-ID files are found, NNTP_Xfer will process them as it does any message-ID file (see below).

NNTP_Xfer then attempts to determine the time of the last connection to remhost, by looking for the file News_Manager:NNTP_remhost.LastCall. (When remhost appears in any file names, all occurrences of '.' in the host name are converted to '-'.). This file is normally written by NNTP_Xfer at the end of a NEWNEWS inquiry, and contains a hexadecimal number specifying the number of seconds elapsed between midnight January 1, 1970 and the last NEWNEWS inquiry. If NNTP_Xfer cannot read this file, it uses midnight January 1, 1970 as the time of the last inquiry. This time is then adjusted using the offsets specified by the /DayOffset and /HourOffset qualifiers. The adjusted time and grouplist are used to construct a NEWNEWS command, which is sent to the remote server. The list of message-IDs received from the server in response to the NEWNEWS request is written to a message-ID file in the News_Manager directory. The message-ID file is named as follows:

if all newsgroups on the remote server are being searched, then the file name will be NNTP_remhost.IDs, where remhost is the name of the remote site, with all '.' characters converted to '-'.
if a selective search is being performed (i.e. a grouplist other than '*' was specified), then the name will be NNTP_remhost_Selective.IDs, with remhost constructed as described above.

If the entire list is received without error, then the LastCall file is updated to reflect the time of the current connection.

Once the entire list of message-IDs has been received, NNTP_Xfer checks each message ID against the item index file and the history file on the local system. If the ID is not found on the local system, NNTP_Xfer requests the text of the item from the remote site, and stores it in a batch file in the News_Manager directory, named in a manner similar to the message-ID files, except that the file type is '.Batch' instead of '.IDs'. Once all of the message IDs are processed successfully, the message-ID file is deleted. The batch files should then be added to the local News database using the Add File command in News.Exe.

7.1.1.4 Compilation constants

The following C preprocessor constants, defined in NNTP_Xfer.C, may be useful in tailoring the behavior of NNTP_Xfer to your system:

MAX_BATCH_SIZE -- maximum size of a batch file (in bytes). Once a batch file reaches this size, NNTP_Xfer closes it and creates a new version. The distribution code sets this constant to 250000.
MAX_RESTART_ATTEMPT -- NNTP_Xfer will stay connected for considerable periods, particularly if there are low speed links between two systems, and the link may be broken by the underlying TCP software. NNTP_Xfer will attempt to restart a broken connection, which often is sufficient. To stop these restart attempts continuing indefinitely, NNTP_Xfer will only restart a link a certain number of times, as specified by this constant. The distribution code sets its value to 20.
CLIENT_TIMER -- the maximum time which the local system will wait for remote data before terminating the link on a timeout. The distribution value is 250 (seconds).
RESP_TIMER -- The maximum time to wait when "pinging" the remote system to check that it is still alive. The distribution value is 30 (seconds).

7.1.1.5 Examples

The following command will cause NNTP_Xfer to contact the NNTP server on wombat.bush.au via TCP, and search all newsgroups on the remote system for message IDs appearing since the time of the last non-selective NNTP_Xfer connection to wombat.bush.au:
$ NNTP_Xfer/Node=wombat.bush.au/Protocol=TCP - /Logfile=News_Manager:NNTPXfer.Log
The message IDs will be stored in News_Manager:NNTP_wombat-bush-au.IDs, and items retrieved from the remote server will be stored in News_Manager:NNTP_wombat-bush-au.Batch. When the session is complete, a summary will be appended to News_Manager:NNTPXfer.Log.

The following command will cause NNTP_Xfer to contact the NNTP server on the remote Ultrix system EMU, and search all newsgroups named animals.* for message IDs appearing on the remote system since one day before the last selective NNTP_Xfer connection to EMU. The connection will be over DECnet using stream I/O calls, with "NNTP=" specified as the task string in the connection request.
$ NNTP_Xfer -n emu -p decnet -s -t "NNTP=" - -g animals -d 1 -l News_Manager:NNTPXfer.Log
The message IDs will be stored in News_Manager:NNTP_emu_Selective.IDs, and items retrieved from the remote server will be stored in News_Manager:NNTP_emu_Selective.Batch. When the session is complete, a summary will be appended to News_Manager:NNTPXfer.Log.

The following command will cause NNTP_Xfer to contact the NNTP server on the remote system KOALA, and search all newsgroups for message IDs appearing on the remote system since the last non-selective NNTP_Xfer connection to KOALA. The connection will be over DECnet using the default task string "TASK=NNTP".
$ NNTP_Xfer KOALA DECnet News_Manager:NNTPXfer.Log
The message IDs will be stored in News_Manager:NNTP_koala.IDs, and items retrieved from the remote server will be stored in News_Manager:NNTP_koala.Batch. When the session is complete, a summary will be appended to News_Manager:NNTPXfer.Log.

7.1.2 NNTP_Xmit

NNTP_Xmit is a utility used to send news items from the local system to a remote system using the NNTP transfer protocols over a DECnet or TCP link to the remote NNTP server. The actual protocol used is the local system sending IHAVE messages to the remote system notifying that system of locally held message identifier strings, to which the remote system will either respond with a "please send me" string to receive the actual text of the relevant item, or a rejection indicating that it already has the item. This utility essentially makes the local system the active partner in the transfer.

The list of message identifier strings is set up by News itself when processing incoming news batches via the Add File command or local postings via the Post command. In either case the News.Sys file is consulted, and if the item is to be forwarded to a remote site whose News.Sys flags field is N or NX, then the message ID is added to the spool file for that site.

The spool files are, by default, organized as sequential text files with one message ID string per line. In most cases this should be adequate, but where the news traffic level is high, or the remote system is connected with a low bandwidth line, there may be contention for this file between News.Exe (writing new message IDs to the file) and NNTP_Xmit.Exe (reading message IDs from the file during a transfer session). In such cases the NX flag should be used in the flags field of the News.Sys entry for that site, so that a shared access RMS indexed file is used to store the message ID strings.

7.1.2.1 Installation

NNTP_Xmit.Exe is created in the News_Dist directory when NewsBuild is run. You should copy this image to a location, such as News_Manager, where it is accessible to the News maintenance job which will forward items to downstream sites via NNTP over DECnet or TCP, but is not accessible to other users. This job should invoke NNTP_Xmit using a foreign command definition, such as
$ NNTP_Xmit == "$News_Manager:NNTP_Xmit"

7.1.2.2 Command Syntax

NNTP_Xmit can be invoked with either a DCL-like syntax, using qualifiers to pass information to the image, or a U*ix-like syntax, involving simple command line switches. You may use either syntax, but you may not mix the two within the same command. In either case, NNTP_Xmit should be invoked via a foreign command definition; it will parse its own command line. The interpretation of all parameters is case-insensitive.

Before invoking NNTP_Xmit, the process should insure that it has read access to all News item files in the local database. You can do this by running the job which invokes NNTP_Xmit under the same UIC as the owner of the News_Device directory tree (usually the News manager), or by granting the process SysPrv or ReadAll privilege, or by Installing the image with SysPrv or ReadAll. NNTP_Xmit does not turn off these privileges when not using them.

In addition, you should make sure NNTP_Xmit.Exe has SysNam privilege, if you want it do define the logical name NNTP_More_To_Offer when it exceeds the maximum number of message IDs you have specified for a single session, and there are more message IDs remaining in the remote system's spool file. You can do this by running NNTP_Xmit.Exe in a process which has SysNam, or by Installing the image with SysNam. NNTP_Xmit does not turn off SysNam when it is not using it.

Finally, if you plan to use the News resource lock mechanism (i.e. NewsShutDown.Exe) when you need exclusive access to the News database, NNTP_Xmit.Exe must have access to SysLck privilege. You can do this by running NNTP_Xmit.Exe in a process which has SysLck, or by Installing the image with SysLck.

Sample DCL-like syntax:

$ NNTP_Xmit/Node=remhost/Protocol=transport
/IDFile=id-file[/Task=decnet-task][/Stream][/IndexID]
[/LogFile=log-file][/DebugFile=dbg-file]

Sample U*ix-like syntax:

$ NNTP_Xmit remhost transport id-file
[-t decnet-task] [-s] [-x] [log-file] [dbg-file] [-h]

/Node=remhost -- the name of the remote host. This is the actual address to be used in establishing the connection, not the name that site uses in news item headers. For DECnet connections, this should be the DECnet nodename of the remote node, or its node number in the format area*1024+node (area*1024 is optional if the remote node is in the same area as your system). For TCP connections it should be the FQDN of the remote host, or its IP address in dotted decimal form.
[U*ix-like syntax: first parameter not beginning with -- or immediately following -t.]

/Protocol=transport -- the transport protocol to be used when connecting to the remote node. This value should be 'TCP', or the name of the TCP transport you specified when building News, if you want to establish a TCP connection to the remote NNTP server; NNTP_Xmit uses the standard NNTP port (119). If any other value is specified, NNTP_Xmit will attempt to establish a DECnet connection.
[U*ix-like syntax: second parameter not beginning with -- or immediately following -t.]

/IDFile=id-file -- file specification of the spool file to be used in offering items to the downstream site. This is the file specified in the spool-file field of the remote site's News.Sys entry.
[ U*ix-like syntax: third parameter not beginning with -- or immediately following -t.]

/Task=decnet-task -- name of the NNTP server DECnet task on the remote node. This parameter is used only for DECnet connections, and is ignored for TCP connections. If the task name is not specified on the command line, NNTP_Xmit will attempt to translate the logical name News_remhost_Task, and, if successful, will use this as the task name. If this fails, a default task name of "TASK=NNTP" is used. The task name is converted to uppercase.

You should determine the DECnet task name of the remote NNTP server when you set up your newsfeed to the downstream site. In particular, the names "TASK=NNTP" or "0=NNTP" are common on VMS systems, and "NNTP=" is common in Ultrix systems.
[U*ix-like syntax: -t decnet-task]

/Stream -- indicates that during a DECnet connection stream input routines should be used instead of record-based input routines. This is necessary when connecting via DECnet to a NNTP server running under Ultrix. This parameter is ignored for TCP connections.
[U*ix-like syntax: -s]

/IndexID -- indicates that the spool file is an RMS indexed file, rather than a sequential text file (see discussion above).
[U*ix-like syntax: -x]

/LogFile=log-file -- file specification of a file to which will be appended a one line summary of the transfer session on completion. This summary will list the number of identifiers offered to the remote system, the number of items accepted and rejected, and the number of local lookup failures.
[U*ix-like syntax: fourth parameter not beginning with -- or immediately following -t .]

/DebugFile=dbg-file -- the file specification of a file to which will be written a full traffic log of the transfer session. This file will expand quite rapidly when processing a large batch of identifiers, so the parameter should be used only in a controlled testing environment.
[U*ix-like syntax: fifth parameter not beginning with -- or immediately following -t .]

The U*ix-like command line switch -h causes NNTP_Xmit to print a summary of the command syntax, as does invoking NNTP_Xmit with no parameters. There is no DCL qualifier which has this function.

7.1.2.3 Transfer Session Summary

NNTP_Xmit will establish a connection to the NNTP server on the remote system, and then commence a sequence of IHAVE messages to the remote server. The remote server will respond with either a request to send the full item or a rejection.

This process will continue until

the input identifier file is fully read, or
NNTP_Xmit is forced to exit when another process locks the News database, or
the remote or local system terminates the transport link, or
more than NNTP_MAX_OFFER identifiers are offered to the remote system. If this occurs, NNTP_Xmit will define the logical name NNTP_More_To_Offer in executive mode in the system logical name table, with the translation "YES", before exiting.

If the spool file is a sequential text file, it is opened in read mode, and a second work file named NNTP_pid.Tmp is opened in the same directory in write mode. If an item cannot be locally read, then the text will be written to this temp file for a subsequent retry. If the transfer is terminated before complete reading of the input file, all unoffered message identifier strings are then written into this work file. On exit the original input file is deleted, and the work file is renamed to the original input file name.

If the spool file is an indexed file, NNTP_Xmit makes a sequential pass through the file. All offers (i.e. remote acceptance or remote rejection) are followed by deletion of the corresponding record from the identifier file.

7.1.2.4 Compilation constants

The following C preprocessor constants, defined in NNTP_Xmit.C, may be useful in tailoring the behavior of NNTP_Xmit to your system:

NNTP_MAX_OFFER -- the maximum number of message IDs which can be offered to the remote system in a single session. Once NNTP_Xmit has offered this many identifiers, it closes the connection and quits. Before quitting, it attempts to define the logical name NNTP_More_To_Offer (qv). (This will succeed only if you have provided NNTP_Xmit.Exe with the necessary privilege.) The distributed copy of NNTP_Xmit.C uses the value 5000 for this constant.
CLIENT_TIMER -- the maximum time which the local system will wait for remote data before terminating the link on a timeout. The distribution value is 250 (seconds).
RESP_TIMER -- The maximum time to wait when "pinging" the remote system to check that it is still alive. The distribution value is 30 (seconds).

7.1.2.5 Examples

The following command will contact the NNTP server on the remote system wombat.bush.au using TCP, and will offer the items whose message IDs appear in the file News_Manager_Dev:[IHave_Wombat]Wombat.Ids. A summary of the session will be appended to the file News_Manager:NNTPXmit.Log.

$ NNTP_Xmit/Node=wombat.bush.au/Protocol=TCP -
_$ /IDFile=News_Manager_Dev:[IHave_Wombat]Wombat.Ids -
_$ /LogFile=News_Manager:NNTPXmit.Log

The News.Sys entry for wombat.bush.au would be something like:

wombat:comp,misc,news,rec,soc,talk,vmsnet,au,to.wombat:N:\
News_Manager_Dev/[IHave_Wombat]Wombat.Ids:world,net,au

The following command will contact the NNTP server on the remote Ultrix system EMU using DECnet, and will offer the items whose message IDs appear in the file News_Manager_Dev:[IHave_Emu]Emu.Ids. A summary of the session will be appended to the file News_Manager:NNTPXmit.Log.

$ NNTP_Xmit EMU DECnet -s -t "NNTP=" -
_$  News_Manager_Dev:[IHave_Emu]Emu.Ids -
_$  News_Manager:NNTPXmit.Log

The News.Sys entry for EMU would be something like:

emu:*,!alt,!control,!junk,!to,to.emu:N:\
News_Manager_Dev/[IHave_Emu]Emu.Ids:*

The following command will contact the NNTP server on the remote VMS system KOALA using DECnet, and will offer the items whose message IDs appear in the RMS indexed file News_Manager_Dev:[IHave_Koala]Koala.Ids. A summary of the session will be appended to the file News_Manager:NNTPXmit.Log, and a detailed record of the session will be written to the file News_Manager:Koala_NNTPXmit.Test.

$ NNTP_Xmit/Node=KOALA/Protocol=DECnet -
_$ /IDFile=News_Manager_Dev:[IHave_Koala]Koala.Ids/IndexID -
_$ /LogFile=News_Manager:NNTPXmit.Log -
_$ /Debugfile=News_Manager:Koala_NNTPXmit.Test

The News.Sys entry for KOALA would be something like:

koala:aarnet,vmsnet,to.koala:NX:\
News_Manager_Dev/[IHave_Koala]Koala.Ids:world,net,au,aarnet

previous: 7 News Support Utilities
next: 7.2 Testing NNTP connections over DECnet or TCP/IP networks
Table of Contents