5 Setting Up News as a Local Reader/Server

5.1 Setting up disk space for News use

In order to maintain a local news database, you will need to set aside sufficient disk space for the News images and support files, for the database itself, and for the temporary files News creates during various maintenance tasks. If you plan to carry a large number of newsgroups, this can require quite a lot of space, as well as some attention to tuning in order to prevent News maintenance jobs from bogging down. As you decide how to allocate space, it may be helpful to consider the following five areas.

5.1.1 News images and support files

These files are usually grouped in one directory, referenced by the logical name News_Manager. This area is also often used as the root of a directory tree, referenced by the logical name News_Manager_Dev, for spool files holding articles which have arrived from feed sites, but have not yet been added to the local News database, and items awaiting transport to downstream sites. Depending on the size of your newsfeeds, this area can take up as little as a few thousand disk blocks or as much as a few hundred thousand.

5.1.2 Index files for the local news database

The three ISAM files News.Groups, News.Items, and History.V60 are used by News to keep track of items in the local database. They are all located in the directory referenced by the logical name News_Root. In addition, if you are using NNTP message ID caching, the section file News.Cache is located here as well. The index files can grow quite large on a system carrying a full newsfeed (in particular, News.Items can be over 200,000 blocks in size), and performance is improved considerably if fragmentation of these files is kept to a minimum. For this reason, you may wish to place News_Root on a volume which has adequate space and which is not used to store large numbers of small files (e.g. the news item database), since this tends to increase fragmentation. In addition, you should periodically optimize the index files, as described later in the setup procedure. Finally, you can improve efficiency by using global buffers to cache the indexes of these files. (See the appendix RMS Tuning of the News Index Files for more information.)

5.1.3 Item files

Text files and directories containing the actual items in the News database are stored in a tree rooted by the logical name News_Device. These files, while small, may number into the hundreds of thousands, and so account for the majority of space taken up by News, requiring as much as 2 Gbytes of disk space for a full newsfeed. In addition, since most item files are small, and they are constantly being created and deleted as items are added and expired, the volume on which News_Device resides is prone to fragmentation. For these reasons, you should consider the following steps to prepare the volume which will contain News_Device:

In order to keep fragmentation from adversely affecting other files, try to locate News_Device on a dedicated disk, or at least one on which there is relatively little creation or extension of other files going on. If the size of your newsfeed doesn't justify dedicating an entire spindle to News, consider using a package like VDDRIVER (available on some of the more popular VMS ftp sites) to create a "virtual disk" on which to place News_Device. This will isolate your item files within the container file of the virtual disk, and so will prevent them from fragmenting the rest of the spindle.
Be sure to preallocate sufficient headers for the item files on the volume which will contain News_Device. You can estimate the number of item files which will fit on a volume by dividing its size (in blocks) by the figure
((clusters_per_file * cluster_size_in_blocks) + 1),
where
clusters_per_file is the number of clusters taken up by an average item file (this will vary depending on which newsgroups you plan to carry; if you don't have an estimate from other sources, you can use an average size of 5 blocks as an initial rule of thumb, and divide that by the cluster size of News_Device to obtain the number of clusters used by the an average item file).
You can preallocate headers on the volume by initializing it with the DCL command
$ Initialize/System/NoErase/NoHigh/Headers=n/Maximum_Files=n dev: label
where n is the estimated maximum number of files obtained above. If the device is already in use, and you are running VMS 5.5-2 or later, you can perform an image backup of the device, reinitialize it using the above command, and restore the image backup using the /NoInit qualifier to Backup. If you are running an earlier version of VMS you'll need to either obtain a fixed copy of Backup from DEC, or copy all files off the device, reinitialize it, and copy the files back, since versions of Backup prior to VMS 5.5-2 erroneously truncate IndexF.Sys even when the /NoInit qualifier is specified, and so remove any benefit you have gained by preallocating headers.

While not strictly required, these steps are strongly recommended if you intend to carry a large number of newsgroups. If you don't make these preparations, you may well run out of file headers on News_Device, causing News to crash, and forcing you to take these steps in a hurry at some inconvenient time. Verbum sapienti satis est.

5.1.4 Spool files

Incoming and outgoing items for exchange with other news sites are stored in spool files pending their addition to the local News database or transfer to the receiving site, respectively. The types of files and space requirements vary depending on the size of incoming and outgoing newsfeeds, and on the transport method used.

5.1.4.1 NNTP over TCP/IP or DECnet

Incoming files received by the NNTP server are placed in the News_Manager directory, unless you change the hard-coded value in NNTP_Server.C, so if you are expecting a large amount of incoming NNTP traffic, you should make sure you have enough space available on the volume which contains News_Manager. Since these NNTP batch files are larger than item files, and tend to be of a relatively constant size, they do not in themselves pose as much of a risk for fragmentation as the item files do, but it is still a good idea not to place them on a volume where many other files are being created. Outgoing NNTP traffic is handled by NNTP_Xmit using message ID list files, which are much smaller than batch files, and so do not require much space for even large newsfeeds. You specify their location when you set up the feed for a given site, and their location is more a matter of convenience than performance; a few approaches to organizing these files are discussed later.

5.1.4.2 Other methods of transport

Incoming and outgoing batches transferred by other means (e.g. file copy, UUCP) can be placed wherever you have space for them. Like incoming NNTP batches, these files can become rather large, depending on the number and size of your newsfeeds, but are fairly short lived, and remain at a fairly constant level over time. Once your feeds have been active for a few weeks, you should have a good idea how much space will be required for your spool areas, and can plan accordingly.

5.1.5 Scratch files

News uses temporary scratch files to hold items being processed (e.g. added to the local database, received by the NNTP server). These files are small and transient, so they don't impose any real burden on disk space. However, I/O to these files can become a limiting factor in processing large numbers of items, so you may wish to consider using a RAM disk for these files to speed up tasks like adding incoming batches to the local database. You can control the placement of these scratch files via the logical names News_Scratch (for files used by News.Exe) and NNTP_Scratch (for files used by the NNTP server).

previous: 5 Setting Up News as a Local Reader/Server
next: 5.2 Generating the required files
Table of Contents