One situation that I run into all too frequently in Linux investigations is disk images the come from multiple systems but which were originally created from a common “master” image (or which use a generic LVM2 volume group name like “linux_lvm“). This creates a problem if you are trying to mount these images simultaneously on the same analysis workstation. Linux simply refuses to mount your image if the volume group name, volume group UUID, or the individual file system UUIDs are the same as an already mounted file system.
One workaround is to create a working copy of your image file and then use tools like vgrename, vgchange, pvchange, etc to destructively modify the necessary names and UUIDs. Or you could use overlayfs, which does an implicit copy of your image file into the merged directory (and adds overhead). But what I’ve been really looking for is a way to create an on-the-fly write cache that would absorb the necessary changes without impacting the original underlying image.
When I ran into this issue again on a recent case and my research ran down the same useless rabbit holes, I threw the question out to my social media contacts. James Dunn gave me the nudge I needed to finally solve the problem to my satisfaction– using xmount with the “--cache” option.
Before we begin, let me document the starting checksums for our sample disk images, so that we can verify at the end of this process that they have not changed:
# md5sum host*/*
e3b85cad126731e76955b2240b69f39d host1/disk1.raw
e3b85cad126731e76955b2240b69f39d host2/disk1.raw
For this test scenario, I am actually using two copies of the exact same image. This is your worst nightmare in real casework. Two different hosts that were spawned from an identical master image. Every single UUID down to the individual file system level will be duplicated.
Getting Set Up With xmount
Your Linux distro will likely have a pre-compiled version of the xmount package, but you may have to install it (“apt install xmount” or similar). We are also going to need to create directories where xmount is going to create its virtual image files:
# mkdir -p /mnt/host1/xmount/disk1
# mkdir -p /mnt/host2/xmount/disk1
If you have multiple disks, you can simply repeat the above commands for the other disk numbers.
Now for the xmount magic:
# xmount --cache /mnt/host1/xmount/cache1 --in raw host1/disk1.raw /mnt/host1/xmount/disk1
# xmount --cache /mnt/host2/xmount/cache1 --in raw host2/disk1.raw /mnt/host2/xmount/disk1
# ls -lh /mnt/host*/xmount/*
-rw-r--r-- 1 root root 720K Aug 2 16:38 /mnt/host1/xmount/cache1
-rw-r--r-- 1 root root 720K Aug 2 16:38 /mnt/host2/xmount/cache1
/mnt/host1/xmount/disk1:
total 0
-rw-rw-rw- 1 root root 60G Jan 1 1970 disk1.dd
-r--r--r-- 1 root root 266 Jan 1 1970 disk1.info
/mnt/host2/xmount/disk1:
total 0
-rw-rw-rw- 1 root root 60G Jan 1 1970 disk1.dd
-r--r--r-- 1 root root 266 Jan 1 1970 disk1.info
The disk1.dd objects are virtual files created by xmount. At the moment they have not changed compared to the original image files on disk. But as we change LVM and UUID data, those changes will be reflected in the .../cache* files without impacting the underlying disk images. Note that xmount can also handle initial disk images in E01 and AFF format, but we will want the resulting virtual file to be in the default raw format (*.dd).
So it’s xmount‘s cache-backed disk1.dd files that we want to work with going forward. The next step is to set up loopback devices pointing at these virtual files:
# losetup -fP --show /mnt/host1/xmount/disk1/disk1.dd
/dev/loop5
# losetup -fP --show /mnt/host2/xmount/disk1/disk1.dd
/dev/loop6
Typically we would use the “-r” (read-only) option when setting up a loopback device for forensic purposes. But in this case we are anticipating having to change the volume metadata in our virtual disk images, so read-only is not appropriate.
Shout out to the “-P” option which causes losetup to automatically create sub-devices for the partitions in each disk image:
# file -Ls /dev/loop[56]p*
/dev/loop5p1: Linux rev 1.0 ext4 filesystem data, UUID=13fe4d4f-9291-4c1b-b0df-14b58d2a3e87 (extents) (64bit) (large files) (huge files)
/dev/loop5p2: LVM2 PV (Linux Logical Volume Manager), UUID: v4T9wI-LDPP-1fJG-7Siu-LrYh-dFr3-pcAEVP, size: 62423826432
/dev/loop6p1: Linux rev 1.0 ext4 filesystem data, UUID=13fe4d4f-9291-4c1b-b0df-14b58d2a3e87 (extents) (64bit) (large files) (huge files)
/dev/loop6p2: LVM2 PV (Linux Logical Volume Manager), UUID: v4T9wI-LDPP-1fJG-7Siu-LrYh-dFr3-pcAEVP, size: 62423826432
In typical Linux fashion, each disk has a Linux file system partition up front, which will be the /boot file system, and an LVM2 volume for the other file systems and swap. But because we are using two copies of the same image you will note that the LVM2 volume UUIDs and even the Linux filesystem UUIDs are the same. And of course this pattern continues throughout the rest of the volume metadata.
Physical Volume Conflicts
The first indication that a problem exists is when we try a typical command to access the LVM2 volume:
# vgscan
WARNING: Not using device /dev/loop6p2 for PV v4T9wI-LDPP-1fJG-7Siu-LrYh-dFr3-pcAEVP.
WARNING: PV v4T9wI-LDPP-1fJG-7Siu-LrYh-dFr3-pcAEVP prefers device /dev/loop5p2 because device was seen first.
Found volume group "LabVM" using metadata type lvm2
vgscan is sensing the two LVM physical volumes with the same UUID. It sees the lower numbered loop device first, /dev/loop5p2, so /dev/loop6p2 throws an error. Until we resolve the conflict here, we will not be able to proceed.
pvchange allows us to change the LVM2 physical volume UUID, but the command fails with an error:
# pvchange -u /dev/loop6p2
WARNING: Not using device /dev/loop6p2 for PV v4T9wI-LDPP-1fJG-7Siu-LrYh-dFr3-pcAEVP.
WARNING: PV v4T9wI-LDPP-1fJG-7Siu-LrYh-dFr3-pcAEVP prefers device /dev/loop5p2 because device was seen first.
0 physical volumes changed / 0 physical volumes not changed
While both loopback devices exist, the duplicate UUIDs are going to cause our commands to fail. So we actually need to tear down the /dev/loop6 device in order to make changes to host1 disk image:
# losetup -d /dev/loop6
# pvdisplay
--- Physical volume ---
PV Name /dev/loop5p2
VG Name LabVM
[...]
PV UUID v4T9wI-LDPP-1fJG-7Siu-LrYh-dFr3-pcAEVP
# pvchange -u /dev/loop5p2
Physical volume "/dev/loop5p2" changed
1 physical volume changed / 0 physical volumes not changed
# pvdisplay
--- Physical volume ---
PV Name /dev/loop5p2
VG Name LabVM
[...]
PV UUID 0FNYGG-WUYv-wTDe-OBVt-G1Xq-KygF-4bSxDy
Logical Volume Conflicts
The physical volume UUID is changed, so we should be good to go with the second disk, right? Yeah, about that:
# losetup -fP --show /mnt/host2/xmount/disk1/disk1.dd
/dev/loop6
# vgscan
WARNING: ignoring metadata seqno 4 on /dev/loop6p2 for seqno 5 on /dev/loop5p2 for VG LabVM.
WARNING: Inconsistent metadata found for VG LabVM.
See vgck --updatemetadata to correct inconsistency.
WARNING: outdated PV /dev/loop6p2 seqno 4 has been removed in current VG LabVM seqno 5.
See vgck --updatemetadata to clear outdated metadata.
Found volume group "LabVM" using metadata type lvm2
The good news is that we’ve resolved the conflict with the physical volume UUID, but now the volume group name is a conflict. Again we have to tear down /dev/loop6 so we can make additional changes:
# losetup -d /dev/loop6
# vgrename LabVM vg1
Volume group "LabVM" successfully renamed to "vg1"
# vgchange -u vg1
Volume group "vg1" successfully changed.
# vgdisplay vg1
--- Volume group ---
VG Name vg1
[...]
VG UUID 3lprSv-oRnK-FxdG-t2ns-Bcog-j1H3-HlJWuV
vgrename changes the volume group name (pick any name that is meaningful to you), but in this case that is insufficient. We also need to use “vgchange -u” to change the volume group UUID of the volume.
File System UUIDs
Unfortunately, this isn’t the end of our troubles. File systems have UUIDs too, and duplication here will prevent Linux from mounting the file systems from the LVM group.
First we need to activate the file systems from our LVM volume:
# vgchange -a y vg1
3 logical volume(s) in volume group "vg1" now active
# file -Ls /dev/vg1/*
/dev/vg1/home: Linux rev 1.0 ext4 filesystem data, UUID=d19118ff-f5d5-40a0-970a-d4b310934f46 (extents) (64bit) (large files) (huge files)
/dev/vg1/root: Linux rev 1.0 ext4 filesystem data, UUID=ee7fb811-d8f1-4584-8657-69e1298fe122 (extents) (64bit) (large files) (huge files)
/dev/vg1/var: Linux rev 1.0 ext4 filesystem data, UUID=df99460c-b71c-435f-8410-ef0e1ecaac91 (extents) (64bit) (large files) (huge files)
The file command shows us the three file systems and their respective UUIDs. To change the UUID for an EXT4 file system, use “tune2fs -U random” (for XFS, “xfs_admin -U generate“). But we have another issue:
# tune2fs -U random /dev/vg1/root
tune2fs 1.47.2 (1-Jan-2025)
This operation requires a freshly checked filesystem.
Please run e2fsck -f on the filesystem.
The file system is hasn’t been consistency checked recently. So we continue to lean on our xmount cache and run e2fsck on the file system:
# e2fsck -f /dev/vg1/root
e2fsck 1.47.2 (1-Jan-2025)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vg1/root: 389899/1831424 files (0.1% non-contiguous), 4405452/7323648 blocks
# tune2fs -U random /dev/vg1/root
tune2fs 1.47.2 (1-Jan-2025)
Setting the UUID on this filesystem could take some time.
Proceed anyway (or wait 5 seconds to proceed) ? (y,N) y
# file -Ls /dev/vg1/root
/dev/vg1/root: Linux rev 1.0 ext4 filesystem data, UUID=ae23562b-2e78-42fc-a212-c34bfc6fdd0b (extents) (64bit) (large files) (huge files)
That’s one file system UUID changed, but we need to repeat this process for the other two file systems:
# e2fsck -f /dev/vg1/var
[...]
# tune2fs -U random /dev/vg1/var
[...]
# e2fsck -f /dev/vg1/home
[...]
# tune2fs -U random /dev/vg1/home
[...]
Mount First Modified Image
With all of the renaming out of the way, we can now go through the usual process for mounting this modified disk image:
# mkdir -p /mnt/host1/files
# mount -o ro,noexec /dev/vg1/root /mnt/host1/files
# mount -o ro,noexec /dev/vg1/var /mnt/host1/files/var
# mount -o ro,noexec /dev/vg1/home /mnt/host1/files/home
# ls /mnt/host1/files
bin etc initrd.img lib32 lost+found opt run sys var
boot home initrd.img.old lib64 media proc sbin tmp vmlinuz
dev images lib libx32 mnt root srv usr vmlinuz.old
For completeness, we really should mount the /boot file system from the first partition on the disk:
# mount -o ro,noexec /dev/loop5p1 /mnt/host1/files/boot
# ls /mnt/host1/files/boot
System.map-5.10.0-20-amd64 grub vmlinuz-5.10.0-20-amd64
System.map-5.10.0-21-amd64 initrd.img-5.10.0-20-amd64 vmlinuz-5.10.0-21-amd64
config-5.10.0-20-amd64 initrd.img-5.10.0-21-amd64
config-5.10.0-21-amd64 lost+found
We can see the file systems are mounted. Now let’s verify that mounting the file system and all of the various changes we’ve made have not changed the original image:
# md5sum host1/disk1.raw
e3b85cad126731e76955b2240b69f39d host1/disk1.raw
# ls -lh /mnt/host1/xmount/*
-rw-r--r-- 1 root root 328M Aug 2 17:17 /mnt/host1/xmount/cache1
/mnt/host1/xmount/disk1:
total 0
-rw-rw-rw- 1 root root 60G Jan 1 1970 disk1.dd
-r--r--r-- 1 root root 266 Jan 1 1970 disk1.info
The original image checksum is unchanged, but the cache1 file has grown from its initial 720K size to 328MB, reflecting the necessary changes that were required to get to this point.
The Second Image
With all of the changes we’ve made to our first image, we could simply bring the second image online with no further modifications. In fact we wouldn’t need to use xmount here at all. But what if we had a third system image that was also derived from the same base image with the same volume group name and various UUIDs? We’d have to fix our second image before we could mount the third image.
I am going to proactively repeat the above process for the second image. Not only to save myself future pain, but also to review all of the steps necessary.
First we have the xmount and loopback device setup:
# mkdir -p /mnt/host2/xmount/disk1
# xmount --cache /mnt/host2/xmount/cache1 --in raw host2/disk1.raw /mnt/host2/xmount/disk1
# losetup -fP --show /mnt/host2/xmount/disk1/disk1.dd
/dev/loop6
# file -Ls /dev/loop6p*
/dev/loop6p1: Linux rev 1.0 ext4 filesystem data, UUID=13fe4d4f-9291-4c1b-b0df-14b58d2a3e87 (extents) (64bit) (large files) (huge files)
/dev/loop6p2: LVM2 PV (Linux Logical Volume Manager), UUID: v4T9wI-LDPP-1fJG-7Siu-LrYh-dFr3-pcAEVP, size: 62423826432
Then we fix the physical volume UUID and the volume group name and UUID:
# pvchange -u /dev/loop6p2
Physical volume "/dev/loop6p2" changed
1 physical volume changed / 0 physical volumes not changed
# vgscan
Found volume group "LabVM" using metadata type lvm2
Found volume group "vg1" using metadata type lvm2
# vgrename LabVM vg2
Volume group "LabVM" successfully renamed to "vg2"
# vgchange -u vg2
Volume group "vg2" successfully changed.
Then we can bring the volumes online and change the file system UUIDs:
# vgchange -a y vg2
3 logical volume(s) in volume group "vg2" now active
# file -Ls /dev/vg2/*
/dev/vg2/home: Linux rev 1.0 ext4 filesystem data, UUID=d19118ff-f5d5-40a0-970a-d4b310934f46 (extents) (64bit) (large files) (huge files)
/dev/vg2/root: Linux rev 1.0 ext4 filesystem data, UUID=ee7fb811-d8f1-4584-8657-69e1298fe122 (extents) (64bit) (large files) (huge files)
/dev/vg2/var: Linux rev 1.0 ext4 filesystem data, UUID=df99460c-b71c-435f-8410-ef0e1ecaac91 (extents) (64bit) (large files) (huge files)
# for dev in /dev/vg2/*; do
e2fsck -f $dev
tune2fs -U random $dev
done
[...]
Finally we mount all of the file systems:
# mkdir -p /mnt/host2/files
# mount -o ro,noexec /dev/vg2/root /mnt/host2/files
# mount -o ro,noexec /dev/vg2/var /mnt/host2/files/var
# mount -o ro,noexec /dev/vg2/home /mnt/host2/files/home
# mount -o ro,noexec /dev/loop6p1 /mnt/host2/files/boot
The “Easy” Button
Does this need to be automated? Yes, of course it does!
I’ve added “-V” and “-A” options to my mtt.sh script. “-A newvg” will set up the xmount cache, change the LVM volume group name to “newvg” (choose any name you want), and change all UUIDs:
# mtt.sh -A vg1 -d /mnt/host1 host1/disk1.raw
# mtt.sh -A vg2 -d /mnt/host2 host2/disk1.raw
# ls /mnt/host*/files
/mnt/host1/files:
bin etc initrd.img lib32 lost+found opt run sys var
boot home initrd.img.old lib64 media proc sbin tmp vmlinuz
dev images lib libx32 mnt root srv usr vmlinuz.old
/mnt/host2/files:
bin etc initrd.img lib32 lost+found opt run sys var
boot home initrd.img.old lib64 media proc sbin tmp vmlinuz
dev images lib libx32 mnt root srv usr vmlinuz.old
As always, you can unmount everything with the “-U” option:
# mtt.sh -U /mnt/host1
# mtt.sh -U /mnt/host2
# ls /mnt/host*/files
/mnt/host1/files:
/mnt/host2/files:
If you only need to change the LVM volume group name without changing any UUIDs, use “-V newvg” instead of “-A“. If you want to set up a writable xmount cache for your mounted image but don’t need to make any LVM changes, just use “-W“.
Righteous IT 