LVM crash

During installation of a new scsi disk in a ProLiant 370 server with
RHEL4, LVM manager crashed (hung) and the configuration was corrupted.
The machine cannot boot, and gets kernel panic.

Configuration:

/dev/sda1: mounted as /boot non lvm

/dev/sda2: included in VolGroup00 (ext3),

/dev/sdb1: included in VolGroup00 (ext3)

/dev/sdc1: should have been included in VolGroup00 (the new disk)
/dev/sdc3: util partition on the disk or something from the vendor.

/ (root) was mountet on VolGroup00, except /boot

Printouts fra lvm (bootet fra RHEL4 rescue cd)

Output from lvscan and vgscan:

lvm>lvscan
lvm>vgscan

Couldn't find device with uid m51fn4 .... oWH4t1
[repeat a couple of times]
Volume group "VolGroup00" not found

lvm>pvscan

Couldn't find device with uid m51fn4 .... oWH4t1
PV /dev/sda2 VG VolGroup00 lvm2 [33,81 GB / 128MB free]
PV /dev/sdb1 VG VolGroup00 lvm2 [33,91 GB / 0 free]
PV unknown device VG VolGroup00 lvm2 [0 / 0 free]

lvm>

If only the lvm is corrupt, and the partitions and data are ok, I
suppose it's possible to recover from this. But how? I do not want to
experiment with this server, and I'm not very fluent in lvm configuration...

--
jon martin solaas

Jon Martin Solaas wrote:
> During installation of a new scsi disk in a ProLiant 370 server with
> RHEL4, LVM manager crashed (hung) and the configuration was corrupted.
> The machine cannot boot, and gets kernel panic.
>
> Configuration:
>
> /dev/sda1: mounted as /boot non lvm
>
> /dev/sda2: included in VolGroup00 (ext3),
>
> /dev/sdb1: included in VolGroup00 (ext3)
>
> /dev/sdc1: should have been included in VolGroup00 (the new disk)
> /dev/sdc3: util partition on the disk or something from the vendor.
>
> / (root) was mountet on VolGroup00, except /boot
>
> Printouts fra lvm (bootet fra RHEL4 rescue cd)
>
> Output from lvscan and vgscan:
>
> lvm>lvscan
> lvm>vgscan
>
> Couldn't find device with uid m51fn4 .... oWH4t1
> [repeat a couple of times]
> Volume group "VolGroup00" not found
>
> lvm>pvscan
>
> Couldn't find device with uid m51fn4 .... oWH4t1
> PV /dev/sda2 VG VolGroup00 lvm2 [33,81 GB / 128MB free]
> PV /dev/sdb1 VG VolGroup00 lvm2 [33,91 GB / 0 free]
> PV unknown device VG VolGroup00 lvm2 [0 / 0 free]
>
> lvm>
>
> If only the lvm is corrupt, and the partitions and data are ok, I
> suppose it's possible to recover from this. But how? I do not want to
> experiment with this server, and I'm not very fluent in lvm
> configuration...


If the system boots up, it is probable that /dev/sda is accessible.

If the UID is for /dev/sdb, it seems that your system does not
find the disk at all. Is the disk still in the system?

If yes, is it possible that the new disk (/dev/sdc?) is attempting
to use the same SCSI bus ID as the previous /dev/sdb. Check the
jumpering in the new disk.

--

Tauno Voipio
tauno voipio (at) iki fi

Tauno Voipio wrote:
> Jon Martin Solaas wrote:
>> During installation of a new scsi disk in a ProLiant 370 server with
>> RHEL4, LVM manager crashed (hung) and the configuration was corrupted.
>> The machine cannot boot, and gets kernel panic.
>>
>> Configuration:
>>
>> /dev/sda1: mounted as /boot non lvm
>>
>> /dev/sda2: included in VolGroup00 (ext3),
>>
>> /dev/sdb1: included in VolGroup00 (ext3)
>>
>> /dev/sdc1: should have been included in VolGroup00 (the new disk)
>> /dev/sdc3: util partition on the disk or something from the vendor.
>>
>> / (root) was mountet on VolGroup00, except /boot
>>
>> Printouts fra lvm (bootet fra RHEL4 rescue cd)
>>
>> Output from lvscan and vgscan:
>>
>> lvm>lvscan
>> lvm>vgscan
>>
>> Couldn't find device with uid m51fn4 .... oWH4t1
>> [repeat a couple of times]
>> Volume group "VolGroup00" not found
>>
>> lvm>pvscan
>>
>> Couldn't find device with uid m51fn4 .... oWH4t1
>> PV /dev/sda2 VG VolGroup00 lvm2 [33,81 GB / 128MB free]
>> PV /dev/sdb1 VG VolGroup00 lvm2 [33,91 GB / 0 free]
>> PV unknown device VG VolGroup00 lvm2 [0 / 0 free]
>>
>> lvm>
>>
>> If only the lvm is corrupt, and the partitions and data are ok, I
>> suppose it's possible to recover from this. But how? I do not want to
>> experiment with this server, and I'm not very fluent in lvm
>> configuration...
>
>
> If the system boots up, it is probable that /dev/sda is accessible.

It does not boot, ref the kernel panic mentioned above.

>
> If the UID is for /dev/sdb, it seems that your system does not
> find the disk at all. Is the disk still in the system?

To me it seems sdb is just fine, ref the pvscan output. And, yes, the
disk is very much in the system.

>
> If yes, is it possible that the new disk (/dev/sdc?) is attempting
> to use the same SCSI bus ID as the previous /dev/sdb. Check the
> jumpering in the new disk.

I think the disks are basically ok, before the shit hit the fan, the
system was up and running off /dev/sda and /dev/sdb, and I could use
fdisk to make a partition on /dev/sdc.

--
jon martin solaas

On Sat, 18 Feb 2006 19:40:55 +0100, Jon Martin Solaas <jon.martin.solaas@jahoo.nei> wrote:


>Tauno Voipio wrote:
>> Jon Martin Solaas wrote:
>>> During installation of a new scsi disk in a ProLiant 370 server with
>>> RHEL4, LVM manager crashed (hung) and the configuration was corrupted.
>>> The machine cannot boot, and gets kernel panic.
>>>
>>> Configuration:
>>>
>>> /dev/sda1: mounted as /boot non lvm
>>>
>>> /dev/sda2: included in VolGroup00 (ext3),
>>>
>>> /dev/sdb1: included in VolGroup00 (ext3)
>>>
>>> /dev/sdc1: should have been included in VolGroup00 (the new disk)
>>> /dev/sdc3: util partition on the disk or something from the vendor.
>>>
>>> / (root) was mountet on VolGroup00, except /boot
>>>
>>> Printouts fra lvm (bootet fra RHEL4 rescue cd)
>>>
>>> Output from lvscan and vgscan:
>>>
>>> lvm>lvscan
>>> lvm>vgscan
>>>
>>> Couldn't find device with uid m51fn4 .... oWH4t1
>>> [repeat a couple of times]
>>> Volume group "VolGroup00" not found
>>>
>>> lvm>pvscan
>>>
>>> Couldn't find device with uid m51fn4 .... oWH4t1
>>> PV /dev/sda2 VG VolGroup00 lvm2 [33,81 GB / 128MB free]
>>> PV /dev/sdb1 VG VolGroup00 lvm2 [33,91 GB / 0 free]
>>> PV unknown device VG VolGroup00 lvm2 [0 / 0 free]
>>>
>>> lvm>
>>>
>>> If only the lvm is corrupt, and the partitions and data are ok, I
>>> suppose it's possible to recover from this. But how? I do not want to
>>> experiment with this server, and I'm not very fluent in lvm
>>> configuration...
>>
>>
>> If the system boots up, it is probable that /dev/sda is accessible.

>It does not boot, ref the kernel panic mentioned above.

You can't do anything until you can boot the system. Have you tried
booting from your distro's install CD?

Do a google search for "rebuild linux LVM". I've done similar searches for
rebuilding raid volumes. You should find everything you need in the various
LVM how-to out there.

>
> You can't do anything until you can boot the system. Have you tried
> booting from your distro's install CD?

Yes, the printouts from pvscan, lvscan and vgscan are run off the rescue
cd. So my hypothesis is that I can recreate the configuration for lvm
using the tools on the rescue cd and get the machine up again.

>
> Do a google search for "rebuild linux LVM". I've done similar searches for
> rebuilding raid volumes. You should find everything you need in the various
> LVM how-to out there.

Didn't really find what I needed in the lvm-howto, but I guess the
answer is out there. I've tried a bunch of goole-searchdes, thankyou,
but not the exact one you've suggested. Maybe it's the magic one :-)

Thanx,

--
jon martin solaas

Jon Martin Solaas wrote:

>
> Didn't really find what I needed in the lvm-howto, but I guess the
> answer is out there. I've tried a bunch of goole-searchdes, thankyou,
> but not the exact one you've suggested. Maybe it's the magic one :-)

One of the more interesting things revealed by searching for "recover
linux LVM" was the suggestion to use lvimport:

--------------
http://groups.google.com/group/comp....a378b4765d6ce5



Try booting from CD-ROM and use vgimport with the -f flag to get
your LVM back:


vgimport -f -v data /dev/hda1 /dev/hda5 /dev/hda7


When booting from disk next time, save the configuration with
vgcfgbackup.
--------------

In my case it'd be

vgimport -f -v VolGroup00 /dev/sda2 /dev/sdb1

Does anyone know if this is likely to work? What does this command
actually do when run off the rescue cd? Where is actually the
configuration stored when I shutdown from the rescue disk and boots off
the harddisks again? Will the fact that the root filesystem is on the
VolGroup00 cause any problems?

--
jon martin solaas

On Sun, 19 Feb 2006 21:07:01 +0100, Jon Martin Solaas wrote:

> Jon Martin Solaas wrote:
>
>>
>> Didn't really find what I needed in the lvm-howto, but I guess the
>> answer is out there. I've tried a bunch of goole-searchdes, thankyou,
>> but not the exact one you've suggested. Maybe it's the magic one :-)
>
> One of the more interesting things revealed by searching for "recover
> linux LVM" was the suggestion to use lvimport:
>
> --------------
> http://groups.google.com/group/comp....a378b4765d6ce5
>
>
>
> Try booting from CD-ROM and use vgimport with the -f flag to get
> your LVM back:
>
>
> vgimport -f -v data /dev/hda1 /dev/hda5 /dev/hda7
>
>
> When booting from disk next time, save the configuration with
> vgcfgbackup.
> --------------
>
> In my case it'd be
>
> vgimport -f -v VolGroup00 /dev/sda2 /dev/sdb1
>
> Does anyone know if this is likely to work? What does this command
> actually do when run off the rescue cd? Where is actually the
> configuration stored when I shutdown from the rescue disk and boots off
> the harddisks again? Will the fact that the root filesystem is on the
> VolGroup00 cause any problems?

Good luck Jon, a knotty problem indeed. LVM is wonderful for its dynamic
configuration capability but definitely introduces an additional layer of
abstraction into your system.

Look in /etc/lvm for the configuration settings if you're able to access
that, and preserve/back them up before performing the
import function.

Keep us posted on your progress and hoped for success.

Frank

frank wrote:

>
> Good luck Jon, a knotty problem indeed. LVM is wonderful for its dynamic
> configuration capability but definitely introduces an additional layer of
> abstraction into your system.
>
> Look in /etc/lvm for the configuration settings if you're able to access
> that, and preserve/back them up before performing the
> import function.
>
> Keep us posted on your progress and hoped for success.

Well, I'll keep you posted, not much posted on this topic until now ...

I was also considering using vgexport, but I'm not really sure what that
command does, except "make volume groups unknown to the system" (manpage)

I have a brand spanking new 146gb disk which could hold everything
exported from the volumegroup on /dev/sda2 + /dev/sdb1, but I'm not
capable of extracting the outcome of using this command from the man-page.

--
martin s.

Jon Martin Solaas wrote:
>
> I was also considering using vgexport, but I'm not really sure what that
> command does, except "make volume groups unknown to the system" (manpage)
>
> I have a brand spanking new 146gb disk which could hold everything
> exported from the volumegroup on /dev/sda2 + /dev/sdb1, but I'm not
> capable of extracting the outcome of using this command from the man-page.
>

vgreduce --removemissing VolGroup00

did the trick.

Just before I had partitioned and formatted the new disk, and changed
partition type from Linux LVM to linux, just in case ...

Now everything is up and running again.

--
martin s.