This show has been flagged as Clean by the host.
This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.
In UNIX Curio #8 (
1
credit or debit card, the first six digits identify the card issuer (such as a bank), the next nine digits are assigned to you by the issuer, and the last digit is a check digit. The check digit is calculated using the values of the previous 15 digits, and it is a simple way to avoid typos in entering a card number.
In another example, every
3
. It generates a 32-bit CRC based on the Ethernet algorithm. It operates on either a named file or standard input and outputs the CRC value, the length of the input, and the pathname if a file was given as an argument. Unlike most modern hashing programs, the checksum is printed as a decimal integer and is not padded, so it can be anywhere from one to ten digits long. The length value is the number of bytes in the input (actually specified as the number of
octets
, as systems could potentially use a byte that isn't eight bits long), also expressed as a decimal integer.
There are two major ways that one could use
cksum
to check the validity of a file. First, if you are transferring a file from one UNIX-like system to another, you could run
cksum
against it on both systems and check that the CRC and length are the same. The utility can also be given multiple filenames as arguments, which would generate a list that can then be compared. The second way would be for someone publishing a file or set of files to also publish the CRC values, lengths, and names so that people downloading them could verify that they match. However, I don't think the practice of publishing lists like this really started until more recent hash functions like MD5 and SHA-1 came about so it is unlikely that anyone would publish CRC values instead.
The advantage of these tools should be pretty obvious in comparison to
cmp
, one of the utilities discussed in UNIX Curio #8. To verify a file using
cmp
, you need two files to compare—if you're trying to check a large file you downloaded, you would need to spend the time and bandwidth to download a second copy. And if they didn't match, you would have no idea which of the two, if either, was correct. By contrast,
cksum
is quicker to run, doesn't require downloading a massive amount of excess data, and if run against the original file, makes clear what the correct value is.
This utility is a follow-on from a program called
sum
, which operated very much the same. I had a bit of trouble tracking down the exact development history, but what seems clear is that
5
, which could possibly be true internal to the algorithm, but I have tested several independent implementations of the utility and all of them output a 16-bit value for both the System V and BSD algorithms.)
From what I can tell,
8,9
, which was AT&T's 1979 port of Seventh Edition UNIX to the VAX and became one of the ancestors of System III. The divergence seems to have started with System III, released in 1980;
12
was used for passwords while
sum
and later
cksum
were used to confirm a file's integrity. So even rather early on, these two use cases employed algorithms with different properties, but I haven't dived into the history deeply enough to know how intentional this was. My discussion in this episode focuses on the file use case, so understand that I'm largely avoiding the topic of password hashing. Digital signatures are yet another use case, one that I'm ignoring entirely.
Every few years, some security researcher declares a particular hash algorithm to be "broken" and that everyone should move over to a new one, which generally has a longer hash. While the larger hash space certainly reduces the opportunity for collisions, this disrupts workflows, such as publishing information about software releases by e-mail, which still tends to observe a
https://en.wikipedia.org/wiki/Payment_card_number
https://pubs.opengroup.org/onlinepubs/009695399/utilities/cksum.html
https://man.freebsd.org/cgi/man.cgi?query=sum&sektion=1&manpath=FreeBSD+15.0-RELEASE+and+Ports
https://www.tuhs.org/cgi-bin/utree.pl?file=3BSD/usr/src/cmd/sum.c
https://www.tuhs.org/cgi-bin/utree.pl?file=32V/usr/src/cmd/sum.c
https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/cmd/sum.c
https://datatracker.ietf.org/doc/html/rfc2822#section-2.1.1
14
. At the bottom, it includes an SHA-1 hash and an SHA-2 256-bit hash for the available gzipped
tarfiles. That longer hash is encoded with Base64 because if it were given as a hexadecimal number, it would make the line longer than 78 bytes. The MD5.L hash of the file would be one character shorter than the SHA-1 hash, as shown below. (The extra length of the
name
makes them both consume the same number of characters. The hashes shown are for the "portable" version of OpenSSH.)
Some people claim SHA-1 is also broken, seeking to have people use newer and longer hash functions. For an attacker to compromise MD5.L in this example, they would not only have to create a valid
tar
file compressed with
gzip
containing a malicious payload having the right MD5 hash, that file would have to be exactly 1,972,831 bytes long (the decimal equivalent of 1e1a5f). While there are still many possible inputs that could be tried (256
1972831
, to be exact*), this is far fewer than the infinite possibilities for plain MD5, SHA-1, or SHA-2.
If for some reason it is super important to have a fixed hash length, let's imagine another variation called MD5+L. In this one, instead of L being the input length, it is the input length .
SOCIAL SHARE CARD GENERATOR