Home

Debian/Ubuntu Tips & Tricks

Debuntu.org: .deb packages, Unix/Linux Tutorials and Articles.

sponsors


User login

Poll

Syndicate

Syndicate content


Tips


Home » category » HowTo

Debian-Ubuntu weakness to filesystem corruption recovery

I really love Debian based distribution, namely, Debian itself and Ubuntu, but there is something wrong... The weakest part is located in /var/lib/dpkg/.

Well, yesterday my /var partition got corrupted and because of that I had to reinstall my whole system. This could (and should) have been avoided. But I had no choice left.

Let's read more and you will see why I had to reinstall the whole system and how it could have being avoid.

I usually set up my system with a :

  • / partition (1.5G) hosting all that is necessary to run a linux system (/bin and /sbin)

  • /usr (4G), this is where all extra distribution packages will be stored

  • /home (10G). Where all my personnals data, computing work and programs are stored. By using a separate /home partition you can safely install a new Unix system and keep all your bookmarks, user specific conf ...

  • /tmp (actually I'd be better off aggregating / and /tmp partition)

  • /var : the place where services are stocking their data, I will go through it later on.

The /var directory is a really precious one when running a server, this is the place where mysql keep his datas, postfix (and other mailing soft) his mails.

Debian based system creates a repertory named /var/lib/dpkg. This is the place where the system is going to keep it's precious datas.
A good overview of How-to maintain a Debian System explain that the list of installed packages is located in /var/lib/dpkg/status file and that This file must not be corrupted, or otherwise your system is HOSED as this is the main database for the package manager software: dpkg. Hopefully, because Unix system are often used for production, a back up of this file is made, named status-old.

But because this file is too important for the system, you better take double security measure in order to avoid a disaster. That is the reason why a cron job save this file on a daily basis, along with /etc/password, /etc/group, /etc/shadow.

Well great then! Even though you delete /var/lib/dpkg, you are still out of the mess... Well, not really. I doubt that this backup is made in order to help people inadvertenly erasing /var/lib/dpkg. I guess this is rather to provide necessary file to recover your system.

But then, what happens if the whole /var get corrupted?

YOU JUST GET F****D !!!

Yep, that is just like keeping your second keyring inside your house, if you lose the first keyring, you don't give a damn about second keyring that you will find in the place you can't access, aren't you?

I believe that having a separate partition for /var is a save way to protect your system, basically, if I get troubles on any other partition than /, I should be able to at least get a running system and being able to set back up the rest (even though precious datas might have disappeared).

CONCLUSION:

Because of this bad experience I had, I'm not going to keep debian making backups in the same place. /var/backups is a pretty small directory, therefore I'm going to keep the backups in /root and change in /etc/cron.daily/standard:

bak=/var/backups

by

bak=/root/backups

I don't know what you guys think of this, but I would be really glad to get your comment and see if any other alternatives could be done.

The best would be to at least be able to rebuild the /var/lib/dpkg and /var/cache/apt trees so the system can run without troubles.

Submitted by chantra on Fri, 2006-05-12 16:32.

If you're not making off-mach

Submitted by Cef (not verified) on Sat, 2006-05-13 00:46.

If you're not making off-machine backups, you should. At LEAST of the contents of the /var/backups/ directory. Even if it's a shell script that requires user interaction that you run after any changes to the system and then scp's the files somewhere. It's better than nothing. I mean really, what would you have done if the hard drive had died? Or if the machine had gone up in smoke? You'd be just as hosed.

And rather than changing the location, how about copying to multiple places? Redundancy is a wonderful thing. If you are going to change the location, how about using a symlink instead of editing scripts, so that anything that might make backups there in the future will go to the right place. The added benefit is that any changes that happen to scripts (eg: updates and security problems) will not immediately break your changes - less to maintain and less change of problems.

Also, unless you are running something custom in the way of packages that aren't source-able anymore, you don't need to backup /var/cache/apt either - it's just a cache and will be rebuilt when you "apt-get update" with a valid sources.list file. You should backup your sources.list though, something that isn't done in the usual cron backup!


I agree with the symlink opti

Submitted by chantra (not verified) on Sat, 2006-05-13 09:41.

I agree with the symlink option :).
But I do find a shame that dpkg can't get back up by itself. If dpkg is broken, then you can say goodbye to your system.
I believe dpkg should at least be able to start a new database, for instance, if dpkg files are broken, dpkg should warn you, try to use its backups and if it can't ask if you want to create a new database, even though lots of package will be reinstalled, at least you can run your old system.


You can create a new dpkg dat

Submitted by hackel (not verified) on Thu, 2006-05-25 16:48.

You can create a new dpkg database easily. A simple search would have revelaed this script:
debian-package-database-rebuild which would get your system going again.

I do agree that this could be made simpler, e.g. a dpkg option to automatically parse a backed up /var/lib/dpkg/status and rebuild the DB would be ideal, but in a pinch this script works!