Fixes #128 by trying different decompression methods if the checksum match fails.

Hi @aptsurfer

Why would you want to try to decompress something when the checksum fails ?

I am surprised that apt-offline doesn't decompress them automatically on my system.

apt-offline tries to draw a proper line in the tasks that apt is supposed to do and the gap (that of the network) that apt-offline is supposed to bridge. On newer systems, even apt doesn't decompress the package indices on disk. This mostly would have been in the direction given Debian systems are common playground on embedded systems.

I guess it goes back to the problem I was having in #128: I was downloading bundles whose integrity was correct, but apt-offline reported all the checksums as not having been matched. It turns out that the checksums refer to the decompressed version of the file.

This is why I am decompressing the files if the checksum fails--upon doing so, the checksum matches and all is good. Testing shows that the filename passed into the function call does not have an extension, so I figured I had to resort to brute force.

I am using the latest version of Debian (10.7.0) as the offline system with only the default (DVD 1) packages installed, and Ubuntu 18.04 as the system that downloads packages. I guess I could look further into the code, but if you need another test case, just reference the files back in #128 (comment)

Let me know what a better way of doing this would be.

I guess it goes back to the problem I was having in #128: I was downloading bundles whose integrity was correct, but apt-offline reported all the checksums as not having been matched. It turns out that the checksums refer to the decompressed version of the file.

So lately, the checksum validation in apt is very realtime. And apt-offline needs to do the same.

For some context, have a read of this bug report.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871656

So how apt-offline has an implementation is described in this blog post.
https://www.researchut.com/blog/apt-offline-1_8_2/

I haven't been able to reproduce your problem on my setup but I will see if I can reproduce it some time. But the issue you are describing, if common, should have shown up in my usage, which hasn't been the case.

So the integrity checksum for package indices is maintained on the remote apt repository server. For example, in here: http://ftp.debian.org/debian/dists/buster/InRelease

Here's a snippet from the above:

MD5Sum:
 811507576f30ac20cbbf60727a806ec6  1356901 contrib/Contents-amd64
 71c6b30a40bc923c803520b2c1b1c22f   102635 contrib/Contents-amd64.gz
 f15ab65dc63759004ec988abecd65e9e  1075476 contrib/Contents-arm64
 ecc9200020eb4b13aa4813171eb9118d    84016 contrib/Contents-arm64.gz
 676e2e1aba5bac2737acba11457c1b80  1072443 contrib/Contents-armel
 a797ac094917d1cf47a9225a7a9c3216    83766 contrib/Contents-armel.gz
 47bc4a4cd68ebc9481fab48e1f0b9003  1080904 contrib/Contents-armhf
 82aa19149bdf972afba35c8e9e77bb17    84798 contrib/Contents-armhf.gz

So it has the checksum listed for both, uncompressed data as well as the compressed data (depending on what compression type the remote server provides support for).

So apt-offline downloads the payload and then validates the checksum. If the payload is tampered the checksum would be wrong, which is what I think is your case.

Thank you kindly for still following this issue. I read both of those pages, and I think I've narrowed the problem down (as in the issue).

And ultimately, here's the problem: apt-offline is trying to use the checksums of the uncompressed data while the files that are downloaded contain the compressed data. I have verified this to be true. The data is not being tampered with, it's just that the checksum to compare the data with should be different.

(Looks like you just mentioned that in the issue. Hopefully we can narrow down what is wrong and find a better fix!)