5 Setting Up News as a Local Reader/Server

5.11 Setting up the maintenance jobs

In order to keep your News database up to date, you will need to remove expired items, and add items received from upstream feed sites to the local database (postings by local users are automatically added at the time of posting). In addition, if you provide newsfeeds to other sites, you will need to actually transfer the items enqueued by News for delivery to each downstream site. These tasks are usually accomplished via periodic batch jobs running with a frequency determined by the speed with which you want to propagate items and the load these jobs place on your system. Each of these tasks is discussed in greater detail in the following sections.

5.11.1 Adding incoming items to the local News database

Items transferred to your site by upstream feed sites or received from mailing lists which you have gatewayed into News are added to the local database using the Add File command in News. You should consult the section in the manual which describes this command in detail to determine which qualifiers are appropriate for your site. For instance, the /Delete qualifier is a useful way to clean up after items are added, and is used routinely at most sites. Conversely, it is not a good idea to use certain qualifiers routinely: Also, if you plan to feed items to downstream sites which you don't carry locally, and you use a message ID queue file (News.Sys flags field value 'N'), you should not use the /NoJunk qualifier, since this will not allow any local copy of the item to be created, and NNTP_Xmit will not be able to retrieve the item text when it processes the message ID queue file.

Depending on the method by which the items were transferred, some preprocessing may be necessary to generate files suitable for input to News. If you are adding files from different sources, it is usually most efficient to do any required preprocessing first, and then add all of the resulting files in a single News session. This is usually done from a batch job, which is often called the 'NewsAdd job'. Examples of DCL procedures used to accomplish this are found in the News_Dist ZIP file (they all have NewsAdd in their names). Depending on the type of incoming news feeds you have and the load on your system, you may wish to run this job as often as once every thirty minutes or so, or as rarely as once a day. Remember that local users will not be able to read incoming items, nor will you be able to feed them to downstream sites, until you have added them to the local database.

5.11.1.1 Batches received by your NNTP server

All versions of the NNTP server provided with News store incoming batches in the News_Manager directory as files named NNTP_{long_unique_string}.Batch, which can be handed off directly to News, using a command like
NEWS> Add File/Delete News_Manager:NNTP_*.Batch
If the NNTP server was interrupted for some reason in the middle of receiving a batch, it may leave behind the items it received in a file named NNTP_{long_unique_string}.Incoming_Batch, so you may want to add a command like to your NewsAdd job 
$ Set NoOn
$ Rename/Log/Before=Today News_Manager:NNTP_*.Incoming_Batch .Batch
$ Set On

before you invoke News to execute the Add File command.

5.11.1.2 Batches received by direct file transfer

These batches may or may not require preprocessing, depending on the arrangements you have made with the transferring site. In general, News expects the file passed as a parameter to the Add File command to contain a series of items either in 'rnews' format, or as a succession of mail messages. Most sites are able to provide batches in one of these formats, so you shouldn't have to convert the actual data. However, it is fairly common, especially among UUCP sites, to compress the batch files in order to minimize the bandwidth needed for transfer. The most common method of doing this is via the U*ix compress(1) tool, or its VMS equivalent, LZComp. Files received in this format must be uncompressed using the appropriate utility. For instance, files compressed using LZComp can be uncompressed using the command
$ LZDcmp compressedfile uncompressedfile
while those compressed using U*ix compress(1) are uncompressed using
$ LZDcmp -X 2 compressedfile uncompressedfile
In both cases, you must obtain the LZDcmp program and define LZDcmp as a foreign command to invoke the image ,i.e.
$ LZDcmp == "$dev:[dir]LZDcmp.Exe" (Note: If you don't already have LZComp and LZDcmp, you can get copies at most anonymous ftp sites carrying VMS software. For example, try ftp.spc.edu, in the directory [.macro32.savesets].)

Once this is done, you can hand the uncompressed file off to News, using a command like
NEWS> Add File/Delete uncompressedfile
where uncompressedfile is the name of the uncompressed batch file.

5.11.1.3 Items received via mail

News is able to process items received via mail, and will attempt to remove header lines added during the mail transfer as it adds the item to the News database. This option is intended principally to allow you to gateway mailing lists into News. If at all possible, you should set up your news feeds based on file transfer or NNTP, since these are more efficient and less error-prone than mail transfer.

In order to gateway a mailing list into News, you should subscribe to the mailing list from an account to which the NewsAdd job has access. Then, from the NewsAdd job, you execute the following commands: 

$ Mail
Set Folder Newmail
Extract/All News_Manager:News_Mail.Batch
Delete/All
Exit

in order to extract the items into a text file. Note that if you are gatewaying a number of mailing lists, and each list places a unique string in the From: header, you can subscribe to multiple lists from a single account, and sort them using the VMS Mail command Select, e.g. 
$ Mail
Set Folder Newmail
Select/From="LIST1-Tag"
Extract/All News_Manager:News_List1.Batch
Delete/All
Select/From="LIST2-Tag"
Extract/All News_Manager:News_List2.Batch
Delete/All
. . .
Exit

If you're not able to distinguish among the lists using Select, you'll have to subscribe to them from different accounts, and use some other method to extract the items into files (e.g. DELIVER). If nothing else, you can do something like the following in your NewsAdd job: 
$ list = "LIST1-L"  ! mailing list
$! Note: listuser account must have write access to News_Manager
$ listuser = "LIST1ACCT" ! account you have subscribed to the list
$ fnam = "Sys$Scratch:" + list + "_News.Com"
$ Open/Write cmdfil &fnam
$ Write cmdfil "$ Delete/NoLog/NoConfirm 'F$Environmane(""Procedure"")'" 
$ Write cmdfil "$ Mail" 
$ Write cmdfil "Set Folder Newmail"
$ Write cmdfil "Extract/All News_Manager:News_''list'.Batch"
$ Write cmdfil "Delete/All"
$ Write cmdfil "Exit"
$ Write cmdfil "$ Exit"
$ Close cmdfil
$ Submit &fnam /User=&listuser  ! Requires CmKrnl priv
$ Synchronize/Entry=&$Entry  ! Wait for the job to finish; >= VMS 5.4

or, alternatively: 
$ list = "LIST1-L"  ! mailing list
$! mail file for account which you have subscribed to list
$! (account running NewsAdd job must have access to this file)
$ listuser_mailfile = "$Users:[List1Acct]Mail"
$ fnam = "Sys$Scratch:" + list + "_News.Com"
$ Open/Write cmdfil &fnam
$ Write cmdfil "$ Delete/NoLog/NoConfirm 'F$Environment(""Procedure"")'" 
$ Write cmdfil "$ Mail" 
$ Write cmdfil "Set File ''listuser_mailfile'"
$ Write cmdfil "Set Folder Newmail"
$ Write cmdfil "Extract/All News_Manager:News_''list'.Batch"
$ Write cmdfil "Delete/All"
$ Write cmdfil "Exit"
$ Write cmdfil "$ Exit"
$ Close cmdfil
$ _@&fnam
$ Synchronize/Entry=&$Entry  ! Wait for the job to finish; >= VMS 5.4

Once you have extracted the messages into a file, you can add them to News. Note that, unless the extracted messages are carrying news headers, you must tell News to generate a Newsgroups: header and a Message-ID: header when you add the items to News, by using appropriate qualifiers to the Add File command:
NEWS> Add File/Delete/GenID/Newsgroup="local.list1" -
News_Manager:News_List1.Batch
NEWS> Add File/Delete/GenID/Newsgroup="local.list2" -
News_Manager:News_List2.Batch

5.11.2 Transferring enqueued items to downstream sites

When News adds an item to the local database, whether as part of Add File processing or when a local user posts an item, it also enqueues it for transfer to other sites according to the entries in your News.Sys file. You will then need to actually transfer the enqueued items to the appropriate sites. Again, this is usually accomplished from a batch job; since transferring is usually a natural sequel to adding new items, this is often done from the NewsAdd job.

If the enqueued items are in single item (News.Sys flags field value 'M') or 'rnews' batch (News.Sys flags field value 'B') format, you just need to send them off to the other site via UUCP, DECnet copy, ftp, mail, or whatever means you've arranged with the other site. (Again, you should try to avoid feeding items by mail, and should group items into 'rnews' batches if possible, in order to transfer them efficiently and avoid damage done by unfriendly mailers.)

If you have enqueued message IDs rather than items (News.Sys flags field value 'N'), you can use the resulting queue file as input to the NNTP_Xmit utility, to conduct a transfer session with the downstream site using the NNTP IHAVE protocol. Read the section on NNTP_Xmit in this manual for details of the syntax used for invoking this program. Note that since NNTP_Xmit retrieves the actual item text from the News database, you should run it before any enqueued items have expired and been removed from the local database. This is often a concern if you feed newsgroups to downstream sites which you don't carry locally. These items will be placed in the newsgroup 'junk' on your system, and it is customary to set the retention time for this group to a short period (e.g. 1 day), in order to keep it from growing too large, so the items won't be around for long. The sample file NNTP.Com in the News_Dist ZIP file demonstrates one way of organizing the NNTP queue files for different downstream sites. Any scheme will work, as long as your batch job knows where to look for queue files for each site.

If you use different batch jobs to add items to the local database and to transfer items to downstream sites, you should run the transfer job after each NewsAdd job, if possible. Running it more frequently is of limited value, since it will only transfer items posted by local users since the last NewsAdd run. Running it less frequently will result in longer lag times before downstream sites receive, and can make use of, new items.

If you are functioning as an 'end node' (i.e. you don't feed any downstream sites), you will still need to run this job to transfer local postings back to your feed site. Since this typically involves a small number of items, you should run this job fairly frequently, so that postings from your site are propagated to the net with reasonable speed.

5.11.3 Removing expired items from the News database

Once newsitems reach their expiration date, they must be removed from the local database in order to keep it from growing indefinitely. This is accomplished via the Skim command in News, which can also be used to check the integrity of the database, and remove files that have become 'orphaned' due to errors in news processing. This task is usually performed in a batch job often called the NewsSkim job. You should read the section of this manual which describes the Skim command in detail in order to familiarize yourself with the options available when using this command.

Since item expiration is determined by date only, it is not useful to run the NewsSkim job more often than daily. If you have a large News database, you should run a basic NewsSkim job every day to expire items, using a command like
NEWS> Skim/Items/Served/NoDirectories/NoFileCheck/NoHistory
Since this can take a while, you may wish to run it during low-usage hours. Once every several days, you should run a complete Skim pass over the database to clear out any deleted newsgroup directories and 'orphaned' files, and to remove old entries from the history file, using a command like
NEWS> Skim/Items/Served/Directories/FileCheck/History
This will consume significant CPU and I/O resources, so it should be run during a quiet time on your system.

In addition to pruning the item database, it is usually worthwhile to periodically optimize the index files News uses to keep track of current newsgroups and items, as well as items deleted recently from the local database. This is done as follows:

Since the FDL files were generated and optimized based on the state of the index files before the skim, the resulting files will be preallocated to contain additional entries up to the size of the files before the skim.

Note that Analyze/RMS and Convert require exclusive access to the index files, so you will need to insure that no other processes have these files open when performing these tasks. This is described in detail in the next section. In addition, if your index files are large (as is typical if you're carrying many groups), optimizing the index files may take some time, and will require enough free space in News_Root to accommodate the new versions of each file created by Convert. Be sure you have enough space available, and remember to purge the old versions of the index files after Convert is finished.

5.11.4 Obtaining exclusive access to the News database

On some occasions, it is necessary to insure that one process has exclusive access to the News database. Most often, this is necessary when tuning the index files, as described above, since Convert requires exclusive access to the files it is optimizing. This may also be necessary when you are copying portions of the News database, or performing other operations during which the data could be lost if another process added or deleted items. News provides two mechanisms for doing this: In either of these cases, any other images which are forced to exit will return an exit status of SS$_DEADLOCK (decimal 3594). You can check for this return status in the DCL procedures which drive your add and skim jobs, and repeat the run if necessary after NewsShutDown releases the database lock or News_Stop is deassigned.

Note that in order to use NewsShutdown, *all* processes running News images (i.e. News.Exe, NNTP server, NNTP_Xfer, or NNTP_Xmit) must have SysLck enabled. Each News image checks this at startup, and if SysLck is not enabled, it will not participate in the News locking mechanism. It will not try to turn SysLck on before checking, nor will it turn SysLck off when it's finished using it.

previous: 5.10.6 Setting up the Wollongong TCP NNTP server
next: 5.12 Arranging to exchange news with other sites
Table of Contents