My recent foray into the RAID superblock while working on mtt.sh required a great deal of manipulation of binary data. I’ll be the first to admit that the Linux shell is not a great environment for working with this kind of data. But I got through it and I developed a number of tricks that other folks might find useful.
“dd” is a great tool for extracting binary data, but when the data loads into a shell variable bash has trouble digesting the null bytes. Here is an example of trying to extract the raw RAID superblock into a shell variable. The superblock is a 256 byte structure that sits 4096 bytes from the front of a RAID disk.
# superblock=$(dd if=/dev/loop0p1 bs=256 skip=16 count=1 2>/dev/null)
bash: warning: command substitution: ignored null byte in input
To deal with this issue, encode the data into some “binary safe” encoding. Base64 encoding is one option, but I needed to manipulate this data numerically. So I chose to hex encode the data.
# superblock=$(dd if=/dev/loop0p1 bs=256 skip=16 count=1 2>/dev/null | xxd -p -c 256)
# echo $superblock
fc4e2ba9010000000100000000000000cccdcc9bf86a0b9898df79498f2f7bb86c6f63616c686f73743a626f6f74000000000000000000000000000000000000ffea435e00000000010000000000000000287700000000000000000002000000080000000000000000000000000000000000000000000000000000000000000000180000000000000028770000000000080000000000000000000000000000000000000000000000d8ce6d122f2b584f2ee20f3902e05ee4000008001000000015cad26a000000003000000000000000ffffffffffffffffc8f7706c800000000000000000000000000000000000000000000000000000000000000000000000
“xxd -p” (“plain” mode) dumps the data as hex without the offset markers and ASCII decoding that “xxd” would normally produce. The “-c 256” says to output 256 “columns” of output with no line breaks. But “xxd” interprets this as 256 bytes of output, which is actually 512 total characters of output since each byte outputs as two characters.
Chopping and Transforming
The next task was extracting various fields from the superblock. Happily, bash has a straightforward syntax for selecting substrings from a larger string: “${stringvar:start:length}“. For example, the “name” field in the RAID superblock is 32 bytes long at offset 32 from the start of the superblock–but don’t forget that each byte is two characters in the string!
# raid_name=${superblock:64:64}
# echo $raid_name
6c6f63616c686f73743a626f6f74000000000000000000000000000000000000
Now the problem is that the field is still hex encoded. But the “name” field is actually a string and those are the hex encoded ASCII character values rather than the desired string value. Use “xxd -r -p” to reverse (“-r“) a plain (“-p“) hex encoded string back into its original binary form. For string data this gets back to the original string.
# string_version=$(echo $raid_name | xxd -r -p)
bash: warning: command substitution: ignored null byte in input
# echo $string_version
localhost:boot
Bash ignores the null bytes at the end of the string, albeit with a warning. The best way that I’ve found to get rid of the warning is to use a little bit of “sed” magic.
# string_version=$(echo $raid_name | sed -E 's/(00)*$//' | xxd -r -p)
# echo $string_version
localhost:boot
Note that I am being careful to remove the null bytes as pairs of characters. After all, the last byte in the string might be “P” (0x50) or “p” (0x70). If I just stripped all zeros from the end of the string, it would corrupt the output.
But what about numeric fields? For example, the offset to the RAID bitmap superblock is a four byte integer at byte offset 96.
# bitmap_offset=${superblock:192:8}
# echo $bitmap_offset
08000000
That gets us the encoded version, but what about turning that back into a number? Use the bash “$(( ... ))” math operator to turn hex encoded values into numbers.
# bitmap_offset=$(( 0x${superblock:192:8} ))
# echo $bitmap_offset
134217728
Note the added “0x” in front of the substring value so it gets interpreted as a hex string.
A further complication, however, is that the RAID superblock data is “little-endian” encoded. So the actual number isn’t “0x08000000”. It’s really “0x00000008”. Time for some more “sed“.
# bitmap_offset=$(( 0x$(echo ${superblock:192:8} | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/') ))
# echo $bitmap_offset
8
The trick here is using the “sed -E 's/(..)(..)(..)(..)/\4\3\2\1/'” idiom to convert from little-endian to big-endian (network) byte order. The expression matches each byte (group of two hex encoded characters) and outputs them in reverse order.
Bulk Conversions
While the “sed” trick above works great for byte-swapping individual fields, what about bulk-converting an entire data set from little-endian to big-endian? It turns out that simply adding “g” (“global”) to the end of the “sed” expression will convert the entire superblock, four bytes at a time.
# echo $superblock | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/g'
a92b4efc0000000100000001000000009bcccdcc980b6af84979df98b87b2f8f61636f6c736f686c6f623a740000746f000000000000000000000000000000005e43eaff00000000000000010000000000772800000000000000000000000002000000080000000000000000000000000000000000000000000000000000000000001800000000000077280000000000000000080000000000000000000000000000000000000000126dced84f582b2f390fe22ee45ee00200080000000000106ad2ca15000000000000003000000000ffffffffffffffff6c70f7c8000000800000000000000000000000000000000000000000000000000000000000000000
But I had a slightly different use case. I needed to read a chunk of data off the disk after the superblock and treat it as a series of four byte unsigned integers to compute a checksum. I could have slurped the data into shell variable and used the “sed” trick above to make it big endian.
An alternative is to use “xxd -e” which will treat the input as little-endian but show the hex encoding in big-endian form. By default it does the byte swapping in four byte chunks, but the “-g” (“grouping size”) option can change the default.
# dd if=/dev/loop0p1 bs=256 skip=16 count=1 2>/dev/null | xxd -e
00000000: a92b4efc 00000001 00000001 00000000 .N+.............
00000010: 9bcccdcc 980b6af8 4979df98 b87b2f8f .....j....yI./{.
00000020: 61636f6c 736f686c 6f623a74 0000746f localhost:boot..
[...]
What’s trippy about this output is that the bytes in the hex dump are swapped, but the ASCII translation column is showing the ASCII from the original little-endian encoding. Look at the last four bytes on the third line above. Translated as shown the string would be “<NUL><NUL>to“, but the ASCII translation column is giving “ot<NUL><NUL>” and the string makes sense.
Unfortunately, the “-p” option doesn’t work with “-e“. I needed the bytes in the superblock as a series of four byte values (in big-endian byte order) that I could add together. Time for some “awk“.
# dd if=/dev/loop0p1 bs=256 skip=16 count=1 2>/dev/null |
xxd -e -c 4 | awk '{print "0x"$2}'
0xa92b4efc
0x00000001
0x00000001
[...]
First there’s “xxd -e -c 4” to output the big-endian conversion but in lines of only four bytes. Normally this would be annoying to look at, but it makes it easy for “awk” to pull out the hex encoded integers from the second (middle) column. For a final flourish, “awk” is even outputting the data with the leading “0x” already tacked onto the front.
Output as Hex
Having computed a new checksum as a decimal value, I needed to transform it into hex encoded data so I could incorporate it into my modified superblock. Rather than calling “xxd” to convert a single value, just use the bash builtin “printf” function.
# printf "%08x\n" 31337
00007a69
# printf -v hex_string "%08x" 31337
# echo $hex_string
00007a69
The format string “%08x” tells “printf” to output the number as an 8 digit hex string, with leading zeroes as necessary. In the first example, I add a “\n” to get a newline when outputting to the terminal. But “printf -v” will assign the output directly into a shell variable, which is generally more useful in a shell programming context. In these cases the newline is typically not wanted.
However, in my case I needed the checksum as a little-endian hex string. So I used the “sed” trick to go from big-endian back to little-endian.
new_csum_le=$(printf "%08x" $new_csum | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/')
Back to Binary and Back to Disk
Ultimately I had a modified RAID superblock that I needed to write back to disk. “xxd -r -p” will convert hex encoded data back to binary form. But I needed to write the new superblock 4096 bytes from the front of the device. “dd” can “skip” blocks when reading, but not when writing.
So I reconstructed the entire “4096 null bytes plus superblock” into a temp file and then just wrote the whole thing to the front of the RAID device:
# dd if=/dev/zero of=tempfile bs=4096 count=1 2>/dev/null
# echo $new_superblock | xxd -r -p >>tempfile
# dd if=tempfile of=/dev/loop0p1 2>/dev/null
Scary, right? But it works surprisingly well.
Actually, the real situation was worse. I had to write 4096 bytes of nulls, the new RAID superblock, and then the remaining 3840 bytes of the block the superblock sits in (this 3840 bytes of data contains an important metadata array that follows the superblock), and then a new 256 byte “bitmap superblock” that I had also modified. For these details and other examples of dealing with (little-endian) binary data, refer to the code for mtt.sh.
