5 Setting Up News as a Local Reader/Server

5.12 Arranging to exchange news with other sites

Once you have decided which newsgroups you will carry on your system, you will need to make arrangements to exchange with other sites items in any groups not limited to your system. At first, you will probably want to function as an 'end node', that is, you will receive incoming news feeds, but will not establish any outgoing feeds, except as necessary to pass items posted by local users back to your feed sites. Once your setup is stable and you have a better idea of its impact on your system's performance, you may wish to provide outgoing feeds to other sites. We encourage as many sites as possible to do this, since distribution of news is based largely on cooperation among participating sites, and you'll benefit the net as a whole, as well as your standing within the community, by providing a service as well as receiving it.

There are several places you might look for sites willing to provide you with a news feed. If you are bringing up a system on an existing network within your organization, it's possible that someone else on your network already has news running, and can feed you local and institutional hierarchies, as well as netwide groups. In the long run, this will benefit them as well, if you are capable of operating another NNTP server on your network to distribute the load generated by NNTP client newsreaders. If this isn't possible, you can try asking nearby sites (e.g. large university or corporate sites) to supply you with news -- local colleagues may be able to point you in the right direction, or you can search out potential sites by consulting UUCP maps for your area. If you're looking for a feed for one of the smaller professional hierarchies, you'll often find information on obtaining feeds in the FAQ of the appropriate admin group, and can even post asking for a feed. Finally, some sites offer commercial newsfeeds, charging fees based on the volume of news transferred. You may end up using a combination of these approaches, with some sites providing local or regional groups, while others provide worldwide hierarchies.

If you're lucky enough to be able to choose among several feed sites, there are a number of factors you should take into account. First, consider the cost of obtaining a feed from a given site, not only in direct fees, but also in such indirect ways as line charges for UUCP calls. Second, make sure the sites you connect to are stable, as users will not appreciate it if your news feed is inconsistent or frequently disrupted. Thirdly, since you may be exchanging a lot of data, try to link up with sites with which you have high bandwidth connections, in order to minimize the impact of news traffic on your communications. Finally, if possible, you may want to spread the load out by taking partial feeds from several sites (e.g. Usenet from one site, alt.* from another, and regional hierarchies from a third); this is a bit more of a headache to set up, but may lead to the best performance and the most goodwill in the long run.

5.12.1 Obtaining the information necessary to set up an exchange

When making arrangements with a particular site, you should be sure to exchange the following information:

the list of newsgroups you will exchange
the name that each site places in Path: headers to indicate that an item has travelled through that site
the format in which items are to be transferred. Options supported by News include:
- individual items -- each item is sent as a separate file. When adding items received by mail to the local database, News will attempt to strip off mail headers added during the transfer.
- news batches (also called "rnews format") -- multiple items are concatenated into a batch file, which is transferred as a unit.
- "N blocking" -- this option can be used with either individual items or news batches. The character 'N' is placed in the first column of each line in the file; the contents are otherwise identical to a file generated without this option. This was originally designed to circumvent problems with mailers which mangled files containing blank lines, and is rarely used anymore.
- compressed batches (also called "compressed rnews format") -- news batches are generated as usual, then compressed before transfer, using a tool such as LZComp. This is the preferred format for exchange among most UUCP sites.
- NNTP NEWNEWS protocol -- the receiving site queries the sending site for the message IDs of all items received since the last exchange, and then requests the items it does not already have. This is less efficient than a direct IHAVE setup, and is therefore rarely used.
- NNTP IHAVE protocol -- the sending site maintains a list of new items received since the last exchange with the receiving site, and exchange of these items only is negotiated during the next NNTP session. This is the preferred method for NNTP exchanges over direct connections.
the method by which news is to be transferred. Commonly used options include:
- UUCP -- news batches are transferred during UUCP calls, with each site processing incoming batches and generating outgoing batches between calls.
- DECnet copy or ftp -- news batches generated independently at each site are copied to known locations at the other site, where the local News maintenance jobs look for new batches to process.
- mail -- individual items or batches are mailed between sites and processed locally. This method of transfer is rarely used anymore, as the volume of news has grown beyond the capacity of most mail systems to handle efficiently, and there are usually other methods better adapted to the high volume available.
- direct NNTP connection -- the two sites establish a connection over DECnet or TCP/IP links, and negotiate transfer using the NEWNEWS or IHAVE (preferred) protocols. This method is practically limited to high speed links, as the volume of data will overwhelm slower connections.
the destination locations and login or proxy login information needed if news batches will be copied directly from one site to another.
the addresses and TCP port numbers (default is 119) or DECnet object definitions for servers to be used for direct NNTP connections.
the contact person at each site for news administration.

Remember that even if you plan to function as an 'end node' for now, you will still need to transfer items posted by users at your site back to your feed site for further distribution, so arrangements for bidirectional transfer must be made. Once this information is exchanged, each site can update its configuration accordingly. You should try a few small test transfers to insure that your configuration is correct before starting high volume transfers, and then check the logs of your maintenance jobs (and NNTP server, if you're using direct NNTP connections) carefully for a few weeks, in case any hidden bugs show up as the feed is broken in.

5.12.2 Checklist for integrating a new upstream site into News processing

Once you have made arrangements for an upstream site to feed you news items, you should be sure you have completed the following tasks:

5.12.2.1 Set up appropriate network connections

If the transfer is to be by NNTP over TCP or DECnet, you should make sure that the relevant NNTP server is running, and that you have edited the file News_Root:NNTP_Access.News to allow the upstream site transfer access to your site. In addition, if the transfer is over DECnet, you should make sure that the user identification information which will be used by the upstream site is valid.

If the transfer is by direct file copy, make sure that the user identification information which will be used by the upstream site is valid.

If the transfer is by UUCP, modify your UUCP schedule, if necessary, to contact the upstream site at appropriate intervals.

If the transfer is by mail, make sure that the address to which the items will be mailed is valid.

5.12.2.2 Set up the spool area for incoming files

Make sure that the incoming connection will leave the news items in the area you expect, that sufficient space is free to handle the incoming feed, and the your NewsAdd job will have access to the spool area.

5.12.2.3 Include the new items in your News add processing

Make sure that your NewsAdd job checks for incoming files, and performs any necessary preprocessing (e.g. uncompressing files, extracting mail). This may happen automatically, as when an additional site is integrated into a set of existing feeds using NNTP over TCP, or may require explicit action in your NewsAdd job, as when a mailing list is gatewayed into News. Finally, make sure that you perform any necessary cleanup (e.g. deleting intermediate files).

5.12.2.4 Test the incoming feed

Arrange for the upstream site to feed you a few items, and follow them through the entire processing cycle until they're incorporated into the local News database. If this succeeds, run the feed for a few days with a small number of items to insure that the regular processing cycle handles the feed appropriately, and that any bugs which do crop up don't cause a large volume of news items to back up. Once you're satisfied that this is stable, you can step up transfer to its working volume, and monitor it for a few days to make sure all is well (e.g. by examining the NNTP server and NewsAdd job log files).

This represents a relatively conservative approach, and you may be able to skip some of the intermediate steps if you're simply integrating a new site into a processing environment which is already working well.

5.12.3 Checklist for integrating a new downstream site into News processing

Once you have made arrangements to feed news items to a downstream site, you should be sure you have completed the following tasks:

5.12.3.1 Add the News.Sys entry for the new site

Make sure the site field of the entry is the name that site puts into Path: headers of news items. Look over the feed pattern for both your site and the downstream site, and determine whether you should include a skip_sites field. Make sure that the subscriptions and distributions fields specify all the newsgroups and distributions you want to transfer (including the to.site group for that site, if you are using one), and don't specify any you don't want to transfer. Set up the flags and spool_file fields appropriately, and insure that the spool file specified can be created (directory exists, ownership and privileges allow News.Exe access, and sufficient unused diskquota exists).

5.12.3.2 Set up appropriate network connections

If the transfer is by direct file copy, make sure that the user identification information which you will use to connect to the downstream site is valid.

If the transfer is by UUCP, modify your UUCP schedule, if necessary, to contact the downstream site at appropriate intervals.

If the transfer is by mail, make sure that the address to which the items will be mailed is valid.

If the transfer is to be by NNTP over TCP or DECnet, insure that you can connect to the receiving site's NNTP server, and that you are allowed to transfer items to that site.

5.12.3.3 Set up transfer of items to the downstream site

Items are spooled for transfer to a downstream site whenever local users post into a newsgroup which you are feeding to that site, or whenever Add File processing adds items to a newsgroup you are feeding to that site. However, you still need to actually transfer the spooled items to the receiving site. This is usually done as part of a regular News maintenance job (either a separate job or as part of the NewsAdd job). You should insure that this job runs frequently enough to insure timely transfer of new items to the downstream site.

If the transfer is via mail, UUCP, or file copy, this job should actually mail the items, or copy them to the appropriate location at the receiving site, and then delete the local spool files.

If the transfer is via NNTP over DECnet or TCP/IP, you should invoke the NNTP_Xmit utility to transfer the items using the NNTP IHAVE protocol. NNTP_Xmit will automatically delete the local spool file when all the items it contains have been successfully processed.

If the 'downstream site' does not actually exist (e.g. you are generating batch files containing the contents of groups you want to archive), you should perform whatever processing of the spool files you need.

5.12.3.4 Test the outgoing feed

First, try to transfer a small spool file by hand, to insure that the transfer pathway is properly configured, and that the downstream site handles incoming items appropriately. Then, try running a small feed for a few days (e.g. by altering the subscriptions field of the downstream site's News.Sys entry to include only a few newsgroups), and make sure that the periodic News maintenance jobs behave appropriately. When you are satisfied with this, widen the feed to its full volume, and follow processing for a few days to insure that all is well (e.g. by examining the NNTP_Xmit log files).

previous: 5.11.4 Obtaining exclusive access to the News database
next: 6 Setting up News in Mixed Local/Served Configuration
Table of Contents