Tuesday RCF had a major failure of the nfs server rnsf03 taht serves Brahms and Phobos user area.
As per the message (inclose) that have restored files. This morning when things were supposed to be back to normal a severe problem showed up. It turned up that several files had NOT be restored. This was the case for at least one brahms user (me) and several phobos users. To help restoring the system without having to destroy the work done today
a copy of a complete directories as found on HPSS backup (complete by oct 9 incremental till Oct12) will be placed in a seperate directory tree, and you can use this to determine if all files are there. I will keep you informrmed once it is available for inspection (and copy).
Clearly RCF did not to well in restoring files - first the areas had wrong protection - secondly files backed up were not restored.
RNFS03 experienced a hardware fault early yesterday
evening. The faulty hardware was replace, but other
problems (system load related) extended the length
of the outage. RAID disk configuration and hence
data was lost in the file systems served by
rnfs03. We believe that the load issues have been
resolved and we are currently in the process of
restoring the lost data from backup. Due to the
size of the file systems restoration of the
data is taking a non-trivial amount of time.
Since the file systems on rnfs03 are being restored
from backup, existing NFS file handles for open files
will become stale. Due to the number of processes
that are affected, a system reboot is the fastest
and cleanest way to restore normal operations
on the shared systems, rcf.rhic.bnl.gov, rmds03,
and rsun00, is reboot will probably be necessary.
(rcf.rhic.bnl.gov will definitely be rebooted.)
A announcement will be made when the file systems
have been restored, and when the shared systems
will be rebooted. Current estimates place this
at about 5:00PM EST.
Shigeki Misawa (misawa@bnl.gov)
-- This message forwarded from the RCF announcements page. Recent messages are available at: http://www.rhic.bnl.gov/RCF/Announcements/announce.html-------------------------------------------------------- Flemming Videbaek Physics Department Brookhaven national Lab
tlf: 516-344-4106 fax: 516-344-1334
email: videbaek@bnl.gov
This archive was generated by hypermail 2b29 : Tue Feb 01 2000 - 20:35:19 EST