7 News Support Utilities

7.3 Copying News index files

These utilities allow you to recover as much as possible of damaged News index files, or to copy intact index files using locking to prevent other processes from modifying the files you're copying. In general, you won't need to use these programs unless your index files have been scrambled in some way.

It's pretty rare that News index files become corrupted in ways that aren't fixed routinely on the next full Skim pass. As this is written, I haven't heard of a group index file dying (except with the disk it resides on) yet. The item index file is a bit less bulletproof; in particular, some sites report that duplicate key values occasionally creep into their file, causing an RMS error when records with that key value are manipulated. A lot of people would love to know how this happens (and woe betide the bug then), but it's rare enough that it's been difficult to track it down.

However, the world being what it is, bad things do happen occasionally to good files, so News is distributed with these utilities, to allow you to recover as much information as possible. You'll generally find out about a damaged index file pretty quickly, as News will begin to report errors, particularly during Add File and Skim passes, since these actually modify the index files. If you do develop problems with an index file, it's a good idea to shut down all News activity as soon as possible, so as not to compound the problem. The sections of this manual describing NewsShutDown and the logical name News_Stop describe ways to do this. In addition, it's critical that you not perform a Skim pass before you've done whatever rebuilding you plan on, since Skim is designed to clear up discrepancies in the News database, mostly by deleting "orphaned" items, and you can lose valid newsitems if you skim when the index files are scrambled.

If your item index file has become corrupted, you should first use NewItmFile to recover as much of the damaged item index file as possible. NewItmFile will tell you how far it got before it ran into an error. Then, use Rebuild_Item_File to reconstruct item index records for the rest of the items in your database. This process can take quite some time, if you have a large database. If there are particular groups you don't consider worth the effort of rebuilding, you can directly delete the item files in those groups before running Rebuild_Item_File. News stores the item files on News_Device, in a directory with the same path name as the group name, so the item files for the group news.software.anu-news are News_Device:[News.Software.ANU-News]*.Itm;*.

If your group index file is damaged, you should first use NewGrpFile to recover as much of the old file as possible. Then, from within News, re-create all newsgroups which are still missing, either by hand or using a checkgroups message. After this is done, perform a Skim/Newsgroups on any groups you had to re-create, so that the item counts in that group's index record will be accurate. Finally, you'll have to manually reset restricted newsgroups, using the Set Newsgroup/Restrict command. The old membership information is still there, and will be used; you just have to tell News to look for it.

In either case, if you decide that the amount of work involved in recovering parts of the existing database is too great, you're always free to quickly recover what information you can with NewGrpFile or NewItmFile, and then let the next full Skim sort out any remaining inconsistencies, or even to write off your entire current database and start fresh.

7.3.1 NewGrpFile

NewGrpFile generates a new version of the newsgroup index file News_Root:News.Groups by copying the existing version record by record, until an error is encountered. It can be used to partially recover a damaged News.Groups file, and so minimize the number of groups which must be recreated manually, in the event that this file becomes corrupted. How much of the existing file is recovered depends on where in the file the first bad record is encountered. Any records not recovered must be created manually using the News commands Create Newsgroup and Set Newsgroup. Be careful not to execute the News command Skim before the newsgroup index file is fully rebuilt, or else valid item files and directories may be deleted.

In order to use NewGrpFile, you must have SysLck privilege, and read and write access to the News_Root directory.

7.3.1.1 Installation

NewGrpFile.Exe is created in the News_Dist directory by NewsBuild.Com. You should leave it in a location, such as News_Manager, where it is available to you if you need to recover the newsgroup index file.

7.3.1.2 Command syntax

NewGrpFile is invoked using the DCL Run command; it takes no parameters or qualifiers.

7.3.1.3 Session summary

When NewGrpFile is invoked, it attempts to acquire an exclusive lock on the News resource, in order to prevent other processes from modifying the newsgroup index file as it is being copied. It will wait up to 20 minutes for the lock to be granted, and if it is unsuccessful, it will print an error message and proceed to copy the file while allowing shared access.

Once the lock is acquired or the request has timed out, NewGrpFile copies the existing News_Root:News.Groups record by record to News_Root:News.Groups_Wrk. The work file is created with an initial size that is 87% of the size of the existing News.Groups, and is contiguous; if this is not possible, a message is printed, and the file is created contiguous-best-try. Its extension size is 6% of the size of the existing News.Groups. If no errors are encountered, the work file is renamed to News.Groups, and the old copy of News.Groups is deleted. Be careful of this -- if NewGrpFile was unable to obtain an exclusive lock on the News resource, another process may still have the old copy of News.Groups open, in which case any changes this process makes will affect only in the now-deleted version of this file, and will not be reflected in the new version.

If any errors are encountered, copying stops immediately, and a message is printed which notes the RMS status code for the error. The work file is not renamed, and the original News.Groups is left unchanged.

7.3.1.4 Example

The following results represent a successful run of NewGrpFile against a small newsgroup index file. Note that the News resource lock was not obtained, so the news manager must insure that no other processes have the old (now deleted) copy of the newsgroup index file open.


$ Run [News_Dist]NewGrpFile
Unable to aquire Exclusive Access to the News System:
operation canceled
Continuing . . .

	NEWSGROUP INDEX (#0)
	bionet.molbio.genbank.updates (#6)
	control (#7)
	junk (#3)
	univ.announce (#28)
	wombat.announce (#10)
	wombat.genbank.updates (#11)
	wombat.misc (#9)
	wombat.old_sys_welcome (#8)
End of file reached, 9 records read, 9 records written
Renaming newly created NEWS_ROOT:NEWS.GROUPS_WRK to NEWS.GROUPS

7.3.2 NewItmFile

NewItmFile functions like NewGrpFile, except that it operates on the newsitem index file News_Root:News.Items. Similarly to NewGrpFile, NewItmFile can be used to try to recover part of a damaged item index file. Any part of the file which is not recovered must be generated manually by retrieving the individual item files from the appropriate directories under News_Device, and re-adding them using the News command Add File/Retry. Be careful not to execute the News command Skim/File before the newsitem index file is fully rebuilt, or else valid item files may be deleted.

If you are unable to recover a portion of the index file using NewItmFile, you may wish to use Rebuild_Item_File, which creates new item index file records from scratch, instead. Beware that running either of these programs may require quite some time (up to many hours), if there are many items in your News database.

In order to use NewItmFile, you must have SysLck privilege, and read and write access to the News_Root directory.

7.3.2.1 Installation

NewItmFile.Exe is created in the News_Dist directory by NewsBuild.Com. You should leave it in a location, such as News_Manager, where it is available to you if you need to recover the newsitem index file.

7.3.2.2 Command syntax

NewItmFile is invoked using the DCL Run command; it takes no parameters or qualifiers.

7.3.2.3 Session summary

When NewItmFile is invoked, it attempts to acquire an exclusive lock on the News resource, in order to prevent other processes from modifying the newsgroup index file as it is being copied. It will wait up to 20 minutes for the lock to be granted, and if it is unsuccessful, it will print an error message and proceed to copy the file while allowing shared access.

Once the lock is acquired or the request has timed out, NewItmFile copies the existing News_Root:News.Items record by record to News_Root:News.Items_Wrk. The work file is created with an initial size that is 75% of the size of the existing News.Items, and is contiguous; if this is not possible, a message is printed, and the file is created contiguous-best-try. Its extension size is 12% of the size of the existing News.Items. A status message is issued for each group copied completely, and when processing groups with large numbers of items, a running count is displayed. If no errors are encountered, the work file is renamed to News.Items, and the old copy of News.Items is deleted. Be careful of this - if NewItmFile was unable to obtain an exclusive lock on the News resource, another process may still have the old copy of News.Items open, in which case any changes this process makes will affect only the now-deleted version of this file, and will not be reflected in the new version.

7.3.2.4 Example

The following shows a successful run of NewItmFile on a system with a small News database:

$ Run [News_Dist]NewItmFile
Unable to aquire Exclusive Access to the News System:
operation canceled
Continuing . . .

      Newsgroup: junk, Count: 1
      Newsgroup: control, Count: 1
      Newsgroup: wombat.old_sys_welcome, Count: 1
      Newsgroup: wombat.misc, Count: 4
      Newsgroup: wombat.announce, Count: 1
      Newsgroup: wombat.genbank.updates, Count: 1344
End of file reached, 1375 records read, 1375 records written
Renaming newly created NEWS_ROOT:NEWS.ITEMS_WRK to NEWS.ITEMS

7.3.3 Rebuild_Item_File

Rebuild_Item_File allows you to reconstruct part or all of the News item index file, in case your current copy has been damaged. It differs from NewItmFile in that it does not try to recover data from the corrupted item index file, but uses the item text files to accumulate as much data as possible, and builds a new index file from scratch. In general, it's best to try recovering as much of a damaged item index file as possible with NewItmFile first, since it's much faster than Rebuild_Item_File. If you don't get back what you need, you can then fall back to Rebuild_Item_File.

If your item index file has been damaged, and you're considering rebuilding it, keep these facts in mind:

Rebuild_Item_File reads the item text files to obtain information necessary to reconstruct the item index file records. This means the item text files need to be around when Rebuild_Item_File runs. However, the Skim/File command in News deletes any item text files whose item index file record is missing. Therefore, it's crucial that you do not run Skim/Files before rebuilding the index file.
Since it's reading lots of item text files, Rebuild_Item_File can take a *long* time to run (many hours to a few days, depending on how many newsgroups you carry, system load, I/O rate on News_Device, etc.), during which time News cannot be used, you cannot add new items, and so forth. In light of this, you may decide it's easiest to just write off the lost item files and start fresh. I'm not saying that Rebuild_Item_File is a bad idea -- it isn't, but know what you're getting into.

In order to run Rebuild_Item_File, you must have read access to the item text files (usually on News_Device), and write access to the item index file into which new records will be placed (usually in News_Root).

Since you wouldn't be running Rebuild_Item_File if there weren't a problem with the index files, you'll want to keep the other News images from running until the item index file is rebuilt. This can be done via the News_Stop logical name, or by running Rebuild_Item_File under NewsShutDown. Both of these approaches are described in more detail in previous sections of this manual.

7.3.3.1 Installation

You can build Rebuild_Item_File.Exe with the following commands:

$ Set Default [News_Build]
$ CC [-.News_Src]Rebuild_Item_File.C
$ Link/Exec=[-.News_Dist] Rebuild_Item_File.Obj, -
   NewsMsg.Obj, [-.News_Src]Options_C_Link.Opt/Option
$ Delete Rebuild_Item_File.Obj;

You may keep this image in any convenient location (e.g. News_Manager).

7.3.3.2 Command Syntax

Rebuild_Item_File should be invoked via a foreign command definition; it will obtain the necessary parameters from its command line. The syntax for invocation is
$ Rebuild_Item_File [indexfile] [itmfil_pattern]

indexfile -- The file specification of the item index file. If this parameter is omitted, it defaults to News_Root:News.Items.

itmfil_pattern -- A wildcard file specification which matches all item text files you want Rebuild_Item_File to process. If this parameter is omitted, it defaults to News_Device:[000000...]*.Itm;* (i.e. all the item text files in the local News database).

7.3.3.3 Session Summary

When Rebuild_Item_File starts up, it tries to open the group index file News_Root:News.Groups, and exits if unsuccessful. It then opens the item index file specified by indexfile, creating a new file if one does not already exist.

Once the index files are open, Rebuild_Item_File begins processing files which match itmfil_pattern. For each file, it first checks that the directory in which the file is found matches the name of a newsgroup in the group index file. If so, it then looks for that item's entry in indexfile, and, if it finds the entry, does nothing. If there is no index entry, it build a new record as follows:

The itm_from field is filled in using the From: header of the item. If this header is missing, the itm_from field is left empty.
The itm_title field is filled in using the Subject: header of the item. If this header is missing, the itm_title field is left empty.
The itm_postdate field is filled in using the Date: header of the item. If this header is missing, the current date and time are used.
The itm_lines field is filled in using the Lines: header of the item. If this header is missing, the itm_lines field is left empty, and the record is marked to show that this field is invalid.
The itm_id field is filled in using the Message-ID: header of the item. If this header is missing, a message ID of <grpnum:itmnum> is used instead, where grpnum is the group index number of the newsgroup name constructed from item file's directory, and itmnum is the item number read from the item file's name.
The itm_recvdate field is set to the creation date of the item file, or, if that cannot be obtained, to the current date and time.
The remaining fields in the item index record are zeroed, except for the LINESVALID flag, which is set or reset depending on whether an accurate line count was obtained.

Rebuild_Item_File then attempts to write the new record into the item index file. If successful, it continues on to the next item text file. If it fails because a duplicate key is detected (indicating that another item has the same message ID as this item), it prints an error message, discards the new record, and continues. If any other error occurs, it is signalled immediately, and Rebuild_Item_File continues or exits based on the severity of the error.

Rebuild_Item_File displays a running count of item files processed, and when it is done, displays a message showing the total number of items processed.

7.3.3.4 Example

The following example uses Rebuild_Item_File to create records for all items in the local.* hierarchy in the alternate index file News_Manager:Local.Items. (This has little practical use, but I don't intend to run Rebuild_Item_File against my whole database if I don't need to!).

$ Rebuild_Item_File News_Manager:Local.Items -
                      News_Device:[Local.Test]*.Itm;*
Scanning Item files: news_device:[local.test]*.itm;*:

Added 50 news articles.

previous: 7.2.2.4 Example
next: 7.4 Miscellaneous maintenance utilities
Table of Contents