Skip to content
This repository has been archived by the owner on May 15, 2024. It is now read-only.

Zip64 writes incorrect central directory file header/extended information extra field #260

Open
BhaaLseN opened this issue Jul 27, 2022 · 0 comments · Fixed by mihula/ProDotNetZip#6 · May be fixed by #262
Open

Zip64 writes incorrect central directory file header/extended information extra field #260

BhaaLseN opened this issue Jul 27, 2022 · 0 comments · Fixed by mihula/ProDotNetZip#6 · May be fixed by #262

Comments

@BhaaLseN
Copy link

I noticed that some of my files couldn't be extracted properly using System.IO.Compression, and after investigating it seems that it discards the extra field for being of unexpected size. This affects anything zip64; which is practically either zip files above 4gb size, files inside the zip file above 4gb size and/or files above the 4gb boundary (where the relative offsets cannot be represented anymore in just 4 bytes).

Other tools (such as recent 7zip) also note inconsistencies, but they often just show warnings or ignore the issue altogether; and read the file/extract it correctly anyways.

To reproduce, simply create a zip file and force zip64 to be used (the actual size doesn't matter for this, since it has the same issue as large files):

// <PackageReference Include="DotNetZip" Version="1.16.0" />
using DNZip = Ionic.Zip.ZipFile;
using SICZip = System.IO.Compression.ZipFile;

const string targetZip = "test.zip";
if (!File.Exists(targetZip))
{
    using var czip = new DNZip { UseZip64WhenSaving = Ionic.Zip.Zip64Option.Always };
    czip.AddEntry("SingleFile", (fn, s) => s.WriteByte(0));
    czip.Save(targetZip);
}

using var zip = SICZip.OpenRead(targetZip);
var singleFile = zip.Entries.First();
_ = singleFile.CompressedLength; // this is returned as 0x0000_0000_ffff_ffff because the extra field is ignored
//using var _ = singleFile.Open(); // throws System.IO.InvalidDataException (A local file header is corrupt)
//SICZip.ExtractToDirectory(targetZip, "test"); // throws System.IO.InvalidDataException (A local file header is corrupt)

When staring at the file (for almost too long), it turns out that the compressed, uncompressed file size and relative header offset are set to 0xffff_ffff in the central directory file header; but the extra field has size 0x1c (28, which means it includes the disk start number). This extra field repeats the disk start number as zero (which is not set to 0xffff in the central directory file header).

All but the most recent version of System.IO.Compression have this piece of validation code that requires the fields to conform to the PKWare specification (which must have the disk start number as 0xffff in the regular record, or omit it from the extra field):
https://github.com/dotnet/runtime/blob/83adfae6a6273d8fb4c69554aa3b1cc7cbf01c71/src/libraries/System.IO.Compression/src/System/IO/Compression/ZipBlocks.cs#L205-L206

They expect that extra field to be 0x18 (24), and ignore it altogether.

This was since addressed by dotnet/runtime#68106 to fix dotnet/runtime#49580; but it doesn't seem to have landed in any public release yet (at least none that aren't previews).

DotNetZip on the other hand should either:

  1. set the central directory file header field for the disk start number to 0xffff and keep the extra field as-is, or
  2. don't write the disk start number to the extra field.
BhaaLseN added a commit to BhaaLseN/DotNetZip.Semverd that referenced this issue Aug 5, 2022
@BhaaLseN
Zip: force Disk Start Number to -1 in a Zip64 Central Directory Entry
3a392a6 
When using Zip64 in the Central Directory Entry, the Extra block always
includes the Disk Start Number (regardless of whether is it needed or not.)

Clause 4.5.3 of the PKWare spec clearly states that it MUST only appear in
the Extra block when the corresponding field in the CDE (or LDE) was set to
-1 (0xFFFF in the case of the Disk Start Number.)

Some tools (such as 7-Zip) show warnings in this case, but often work as
expected. Others (such as older versions of System.IO.Compression) follow
the spec more stringently and will refuse to use the values from the Zip64
Extra block when the related values do not match their expectation.

Fixes haf#260
@BhaaLseN BhaaLseN linked a pull request Aug 5, 2022 that will close this issue
Draft
BhaaLseN added a commit to BhaaLseN/DotNetZip.Semverd that referenced this issue Mar 10, 2023
@BhaaLseN
Zip: omit Disk Start Number from the Zip64 Central Directory Entry
e34edb6 
Clause 4.5.3 of the PKWare spec clearly states that it MUST only appear in
the Extra block when the corresponding field in the CDE (or LDE) was set to
-1 (0xFFFF in the case of the Disk Start Number.)

Previously, this would only be the case when Zip64 was presumed or the disk
number was actually outside the range for the CDE. In every other case, the
Disk Start Number would go into both locations, violating the spec.

Some tools (such as 7-Zip) show warnings in this case, but often work as
expected. Others (such as older versions of System.IO.Compression) follow
the spec more stringently and will refuse to use the values from the Zip64
Extra block when the related values do not match their expectation.

However, fixing this to always write a conformant CDE that has its Disk
Start Number as 0xFFFF with the actual value in the Extra block breaks other
tools (such as marmelroy/Zip on iOS) that do not fully support Zip64.

The spec does allow the Extra block to omit the Disk Start Number (and the
Relativ Header Offset) when they fit in the CDE, so we do just that for
general compatibility.

Fixes haf#260
@Venkatprasanth96 Venkatprasanth96 mentioned this issue Jul 10, 2023
Open
mihula pushed a commit to mihula/ProDotNetZip that referenced this issue Mar 13, 2024
@BhaaLseN
Zip: omit Disk Start Number from the Zip64 Central Directory Entry
5ce3732 
Clause 4.5.3 of the PKWare spec clearly states that it MUST only appear in
the Extra block when the corresponding field in the CDE (or LDE) was set to
-1 (0xFFFF in the case of the Disk Start Number.)

Previously, this would only be the case when Zip64 was presumed or the disk
number was actually outside the range for the CDE. In every other case, the
Disk Start Number would go into both locations, violating the spec.

Some tools (such as 7-Zip) show warnings in this case, but often work as
expected. Others (such as older versions of System.IO.Compression) follow
the spec more stringently and will refuse to use the values from the Zip64
Extra block when the related values do not match their expectation.

However, fixing this to always write a conformant CDE that has its Disk
Start Number as 0xFFFF with the actual value in the Extra block breaks other
tools (such as marmelroy/Zip on iOS) that do not fully support Zip64.

The spec does allow the Extra block to omit the Disk Start Number (and the
Relativ Header Offset) when they fit in the CDE, so we do just that for
general compatibility.

Fixes haf#260
This was referenced Mar 13, 2024
Closed
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@BhaaLseN