(OT) Mailing list server problems

[Bill Campbell]

We had a fairly large snow storm with power outage over the
weekend while my wife and I were in Arizona.  The power went out
in the wee hours of 5 February after our nightly backup to our
in-house Synology Network Attached Storage (NAS).

The power came back up sometime on Tuesday, although some of
our systems systems require hands-on to bring them up after a
power failure, and my brother couldn't get here due to the snow
so things had to wait until Carolyn and I got home Wednesday
morning.  It took me a while to get to the servers as we found a
tree had fallen across the driveway, landing on another tree on
our yard which broke its fall so it didn't do too much damage
when it hit the garage roof and Carolyn's car.  I had to get a
chain saw so I could at least uncover her car so she could go to
work on Thursday.

The server for our Mailman mailing lists runs in a VMware virtual
machine under the old free VMware server on a CentOS Linux box.
That machine hadn't come up clean, and I had to fiddle a bit to
get things running again.  It wasn't until yesterday afternoon
that I figured out that the root file system had been corrupted,
and mostly disappeared after I booted off a Knoppix CD and ran a
full 'fsck -f -y' on the root file system with everything in the
/lost+found directory, largely useless.

At that point, I removed the /lost+found stuff, ran another fsck,
then tried to copy the root file system using rsync from the NAS
box.  This failed as it said the file system was full, even
though the 8GB file system should hold less than 5GB.  I tried
again using find and cpio, with the same result.

When this failed, I restored the VM from a fairly ancient
snapshot, used rsync to bring it current from the most recent
backups from early Sunday morning, and things seem OK now.

The upshot as far as the FilePro list is that messages posted
between the time I got things up initially Wednesday afternoon,
and late yesterday are gone.

Bill
-- 
INTERNET:   bill at celestial.com  Bill Campbell; Celestial Software LLC
URL: http://www.celestial.com/  PO Box 820; 6641 E. Mercer Way
Voice:          (206) 236-1676  Mercer Island, WA 98040-0820
Fax:            (206) 232-9186  Skype: jwccsllc (206) 855-5792

Talk is cheap...except when Congress does it. -- Anon