Post

Restoring Corrupted LVM Physical Volumes: A Complete Guide

Step-by-step guide to recover corrupted LVM Physical Volumes using backup metadata. Learn how to restore the LABELONE header, recover volume groups, and prevent PV corruption in Linux systems.

Posted Updated
By Nishant Thakur
12 min read
Restoring Corrupted LVM Physical Volumes: A Complete Guide

Introduction

Logical Volume Manager (LVM) is a powerful storage management system in Linux that provides flexibility in managing disk space. However, like any storage system, LVM configurations can become corrupted due to accidental disk operations, failed hardware, or administrative errors. When a Physical Volume (PV) loses its LVM metadata particularly the critical “LABELONE” header the entire Volume Group (VG) and its Logical Volumes (LVs) can become inaccessible.

This guide demonstrates how to recover from PV corruption using LVM’s built-in backup and restore mechanisms. We’ll walk through a real-world scenario where a PV’s metadata was accidentally overwritten and show the complete restoration process.

Understanding the Environment

First, let’s examine our storage configuration. We have three 2GB NVMe drives that we’ll configure as LVM physical volumes:

1
2
3
4
5
6
7

root@idefend~# lsblk | grep nvme
nvme0n1     259:0    0    50G  0 disk
├─nvme0n1p1 259:1    0     1G  0 part /boot/efi
└─nvme0n1p2 259:2    0  48.9G  0 part /
nvme0n2     259:3    0     2G  0 disk
nvme0n3     259:4    0     2G  0 disk
nvme0n4     259:5    0     2G  0 disk

Setting Up LVM

Installing LVM2

Before working with LVM, ensure the LVM2 package is installed:

1

root@idefend~# apt install lvm2

Verify the installation by checking the version:

1
2
3
4

root@idefend~# lvm version
  LVM version:     2.03.27(2) (2024-10-02)
  Library version: 1.02.201 (2024-10-02)
  Driver version:  4.49.0

Creating Physical Volumes

Important Note: The example below creates PVs directly on raw disks. In production environments, always create partitions first, apply a GPT label, and then create PVs on the partitions. This provides better disk organization and prevents accidental data loss.

1
2
3
4

root@idefend~# pvcreate /dev/nvme0n2 /dev/nvme0n3 /dev/nvme0n4
  Physical volume "/dev/nvme0n2" successfully created.
  Physical volume "/dev/nvme0n3" successfully created.
  Physical volume "/dev/nvme0n4" successfully created.

Command Explanation: pvcreate initializes block devices for use with LVM by writing metadata headers that identify them as LVM physical volumes. This metadata includes a unique UUID and the “LABELONE” signature at offset 0x200.

Creating a Volume Group

Combine all three PVs into a single Volume Group:

1
2

root@idefend~# vgcreate logs-vg /dev/nvme0n2 /dev/nvme0n3 /dev/nvme0n4
  Volume group "logs-vg" successfully created

Command Explanation: vgcreate aggregates multiple physical volumes into a single storage pool (Volume Group). The VG acts as a container from which logical volumes can be carved out.

Verify the Volume Group:

1
2
3

root@idefend~# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  logs-vg   3   0   0 wz--n- <5.99g <5.99g

Creating a Logical Volume

Create a logical volume using all available space:

1
2

root@idefend~# lvcreate --name backup -l+100%FREE logs-vg
  Logical volume "backup" created.

Command Explanation: lvcreate carves out a logical volume from the volume group. The -l+100%FREE parameter allocates all remaining free extents to this LV.

Verify the Logical Volume:

1
2
3

root@idefend~# lvs
  LV     VG      Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  backup logs-vg -wi-a----- <5.99g

Creating and Mounting the Filesystem

Create a mount point and format the logical volume:

1
2
3
4
5
6
7
8
9
10
11
12

root@idefend~# mkdir /backups

root@idefend~# mkfs.ext4 /dev/mapper/logs--vg-backup
mke2fs 1.47.2 (1-Jan-2025)
Creating filesystem with 1569792 4k blocks and 392448 inodes
Filesystem UUID: 505333ed-03de-41c0-9967-0c1f1cf89568
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

Command Explanation: mkfs.ext4 creates an ext4 filesystem on the logical volume. The device mapper path /dev/mapper/logs--vg-backup is automatically created by LVM, with hyphens in the VG name doubled.

Mount the filesystem:

1
2
3
4
5

root@idefend~# mount /dev/mapper/logs--vg-backup /backups/

root@idefend~# df -hT /backups/
Filesystem                  Type  Size  Used Avail Use% Mounted on
/dev/mapper/logs--vg-backup ext4  5.9G  1.6M  5.5G   1% /backups

Understanding LVM Metadata Structure

Let’s examine the LVM metadata headers on our physical volumes. Each PV contains a “LABELONE” signature at byte offset 512 (0x200):

1
2
3
4
5
6
7
8
9
10

root@idefend~# for i in {2..4}; do dd if=/dev/nvme0n${i} count=2 of=/dev/stdout | hexdump -C;done
2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 5.2959e-05 s, 19.3 MB/s
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00000210  5f 6a ef 83 20 00 00 00  4c 56 4d 32 20 30 30 31  |_j.. ...LVM2 001|
00000220  6a 61 34 44 48 4c 55 67  4e 32 4f 58 6d 33 52 55  |ja4DHLUgN2OXm3RU|
00000230  75 64 33 5a 4f 69 66 70  63 52 64 48 47 77 4d 6e  |ud3ZOifpcRdHGwMn|

Key Metadata Elements:

  • Offset 0x200: “LABELONE” signature identifies this as an LVM physical volume
  • Offset 0x218: “LVM2 001” indicates the metadata format version
  • Offset 0x220-0x23F: The 32-character PV UUID (ja4DHLUgN2OXm3RUud3ZOifpcRdHGwMn)

Backing Up LVM Configuration

LVM automatically maintains configuration backups in /etc/lvm/backup/. These text-based files are crucial for recovery:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

root@idefend~# cat /etc/lvm/backup/logs-vg
# Generated by LVM2 version 2.03.27(2) (2024-10-02): Tue Jan  6 22:56:52 2026

contents = "Text Format Volume Group"
version = 1

description = "Created *after* executing 'lvcreate --name backup -l+100%FREE logs-vg'"

creation_host = "idefend"
creation_time = 1767719512	# Tue Jan  6 22:56:52 2026

logs-vg {
	id = "6YuWd4-UIJV-IWdd-6kOZ-mitp-NTpe-17jV47"
	seqno = 2
	format = "lvm2"
	status = ["RESIZEABLE", "READ", "WRITE"]
	flags = []
	extent_size = 8192		# 4 Megabytes
	max_lv = 0
	max_pv = 0
	metadata_copies = 0

	physical_volumes {

		pv0 {
			id = "ja4DHL-UgN2-OXm3-RUud-3ZOi-fpcR-dHGwMn"
			device = "/dev/nvme0n2"	# Hint only

			status = ["ALLOCATABLE"]
			flags = []
			dev_size = 4194304	# 2 Gigabytes
			pe_start = 2048
			pe_count = 511	# 1.99609 Gigabytes
		}

You can manually trigger a backup:

1
2
3
4
5
6

root@idefend~# vgcfgbackup logs-vg
  Volume group "logs-vg" successfully backed up.

root@idefend~# ll /etc/lvm/backup
total 4.0K
-rw------- 1 root root 2.1K Jan  7 02:18 logs-vg

Command Explanation: vgcfgbackup forces an immediate backup of the VG configuration. These backups are automatically created after each LVM metadata change but can be manually triggered for extra safety.

Simulating PV Corruption

Let’s simulate a common corruption scenario where someone accidentally creates a partition table on an LVM PV:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

root@idefend~# parted /dev/nvme0n2
GNU Parted 3.6
Using /dev/nvme0n2
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p free
Error: /dev/nvme0n2: unrecognised disk label
Model: VMware Virtual NVMe Disk (nvme)
Disk /dev/nvme0n2: 2147MB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
(parted) mklabel gpt
(parted) q
Information: You may need to update /etc/fstab.

What Happened: The mklabel gpt command writes a GPT partition table header at the beginning of the disk, overwriting the LVM metadata including the critical “LABELONE” signature.

Detecting the Corruption

Now when we check our physical volumes, we see the corruption:

1
2
3
4
5
6
7
8

root@idefend~# pvs
  WARNING: Couldn't find device with uuid ja4DHL-UgN2-OXm3-RUud-3ZOi-fpcR-dHGwMn.
  WARNING: VG logs-vg is missing PV ja4DHL-UgN2-OXm3-RUud-3ZOi-fpcR-dHGwMn (last written to /dev/nvme0n2).
  WARNING: Couldn't find all devices for LV logs-vg/backup while checking used and assumed devices.
  PV           VG      Fmt  Attr PSize  PFree
  /dev/nvme0n3 logs-vg lvm2 a--  <2.00g    0
  /dev/nvme0n4 logs-vg lvm2 a--  <2.00g    0
  [unknown]    logs-vg lvm2 a-m  <2.00g    0

Command Explanation: pvs displays all physical volumes. The warnings indicate that LVM can no longer find the PV with the specific UUID, and it now shows as “[unknown]” with the “m” (missing) attribute.

Examining the Corrupted Disk

Let’s look at what replaced the LVM header:

1
2
3
4
5
6
7
8
9
10
11
12

root@idefend~# dd if=/dev/nvme0n2 count=2 of=/dev/stdout | hexdump -C
2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 6.2962e-05 s, 16.3 MB/s
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  02 00 ee ff ff ff 01 00  00 00 ff ff 3f 00 00 00  |............?...|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  2d 1b 3c b4 00 00 00 00  01 00 00 00 00 00 00 00  |-.<.............|

Notice that at offset 0x200, instead of “LABELONE”, we now see “EFI PART” (the GPT header signature). The LVM metadata has been completely overwritten.

Restoring the Physical Volume

Step 1: Clear the Corrupted Header

First, zero out the corrupted disk header:

1
2
3
4

root@idefend~# dd if=/dev/zero bs=1k count=2 of=/dev/nvme0n2
2+0 records in
2+0 records out
2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000116285 s, 17.6 MB/s

Command Explanation: dd if=/dev/zero bs=1k count=2 writes 2KB of zeros to the beginning of the disk, clearing both the MBR/GPT area and the LVM metadata region. This provides a clean slate for restoration.

Verify the disk is clean:

1
2
3
4
5
6
7

root@idefend~# for i in {2..4}; do dd if=/dev/nvme0n${i} count=2 of=/dev/stdout | hexdump -C;done
2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 7.3211e-05 s, 14.0 MB/s
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400

Step 2: Extract the Original UUID

Retrieve the original PV UUID from the backup file:

1
2
3
4

root@idefend~# grep -A2 pv0 /etc/lvm/backup/logs-vg
		pv0 {
			id = "ja4DHL-UgN2-OXm3-RUud-3ZOi-fpcR-dHGwMn"
			device = "/dev/nvme0n2"	# Hint only

Command Explanation: grep -A2 searches for “pv0” in the backup file and shows the next 2 lines, revealing the original UUID that must be restored to maintain VG consistency.

Step 3: Restore the Physical Volume

It is recommended to deactivate the corresponding volume group before recovering the corrupted PV(s).

1
2
3
4

root@idefend~# vgchange -an logs-vg
  WARNING: Couldn't find device with uuid ja4DHL-UgN2-OXm3-RUud-3ZOi-fpcR-dHGwMn.
  WARNING: VG logs-vg is missing PV ja4DHL-UgN2-OXm3-RUud-3ZOi-fpcR-dHGwMn (last written to /dev/nvme0n2).
  0 logical volume(s) in volume group "logs-vg" now active

Now perform the restoration using the original UUID:

1
2
3
4
5

root@idefend~# pvcreate -ff --uuid ja4DHL-UgN2-OXm3-RUud-3ZOi-fpcR-dHGwMn --restorefile /etc/lvm/backup/logs-vg /dev/nvme0n2
  WARNING: Couldn't find device with uuid ja4DHL-UgN2-OXm3-RUud-3ZOi-fpcR-dHGwMn.
  WARNING: Couldn't find device with uuid FsaY8x-o0g1-mBhr-u0CS-zNVA-nS7D-HNMNjh.
  WARNING: Couldn't find device with uuid Yu7nGD-6v1Y-HIXf-bFqI-D5KU-x26F-klBJno.
  Physical volume "/dev/nvme0n2" successfully created.

Command Explanation:

  • -ff: Force flag that bypasses safety checks (use with caution)
  • --uuid: Specifies the exact UUID to write, matching the original PV
  • --restorefile: Uses the backup configuration to restore metadata with correct parameters
  • The warnings during restoration are expected as LVM is still detecting the missing PVs

Step 4: Verify Restoration

Check that the LABELONE header has been restored:

1
2
3
4
5
6
7
8
9

root@idefend~# dd if=/dev/nvme0n2 of=/dev/stdout count=2 | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
2+0 records in
2+0 records out
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000378129 s, 2.7 MB/s00000210  b1 9f a6 82 20 00 00 00  4c 56 4d 32 20 30 30 31  |.... ...LVM2 001|
00000220  6a 61 34 44 48 4c 55 67  4e 32 4f 58 6d 33 52 55  |ja4DHLUgN2OXm3RU|
00000230  75 64 33 5a 4f 69 66 70  63 52 64 48 47 77 4d 6e  |ud3ZOifpcRdHGwMn|

The “LABELONE” signature at 0x200 and the original UUID are both restored.

Step 5: Restore LVM configuration from backup file

Once the PV is recreated, we can now restore configuration from previous volume group configuration backup using vgcfgstore tool.

1

root@idefend~# vgcfgrestore --force logs-vg

Then, the deactivated volume group should be executed by running vgchange -ay <volume group name> command and mount the corresponding logical volumes to respective mountpoints by executing mount -a command.

1
2
3

root@idefend~# vgchange -ay logs-vg
  WARNING: PV /dev/nvme0n2 in VG logs-vg is missing the used flag in PV header.
  1 logical volume(s) in volume group "logs-vg" now active

1

root@idefend~# mount -a

1
2
3

root@idefend~# df -hT /backups
Filesystem                  Type  Size  Used Avail Use% Mounted on
/dev/mapper/logs--vg-backup ext4  5.9G  1.6M  5.5G   1% /backups

Common Causes of PV Corruption

Understanding how PV corruption occurs can help prevent future incidents:

  1. Accidental Partition Table Creation: Using tools like fdisk, parted, or gdisk on an LVM PV overwrites the metadata
  2. Disk Cloning Issues: Improperly cloning disks can duplicate UUIDs or corrupt metadata
  3. Direct Disk Writes: Using dd or similar tools without proper offsets can overwrite LVM headers
  4. Hardware Failures: Disk failures or controller issues can corrupt the metadata area
  5. Failed Operations: Interrupted LVM operations or system crashes during metadata updates
  6. Initialization Mistakes: Running pvcreate on an existing PV without proper flags

Best Practices for LVM Management

To minimize the risk of PV corruption and ensure recoverability:

  1. Always Use Partitions: Create partitions with GPT labels before initializing PVs, rather than using raw disks
  2. Regular Backups: Use vgcfgbackup regularly and store backups in multiple locations
  3. Document Your Configuration: Keep records of VG/LV layouts and UUIDs
  4. Test Restores: Periodically verify that your LVM backups are usable
  5. Use Descriptive Names: Name VGs and LVs clearly to avoid confusion
  6. Monitor LVM Health: Regularly run pvs, vgs, and lvs to detect issues early
  7. Automate Backups: Include /etc/lvm/backup/ and /etc/lvm/archive/ in system backups

Conclusion

LVM’s built-in backup mechanism provides a robust solution for recovering from physical volume corruption. By maintaining current backups in /etc/lvm/backup/ and knowing the original PV UUIDs, administrators can quickly restore corrupted PVs without data loss. Remember that this process only restores LVM metadata if actual data is corrupted, additional recovery steps may be necessary. Always maintain regular filesystem-level backups alongside LVM configuration backups.

Linux Administration, Disk Management, LVM
lvm storage-management disk-recovery physical-volumes volume-groups system-administration data-recovery sysadmin linux-server
This post is licensed under CC BY 4.0 by the author.