Can we define a memory structure ( or File structure ) in C#

[John0King]

I'm not a C/C++ developer,
but from some blog post , I notice that it seems very simple to define a binary structure use typedef, but it seems very hard for C# because follow reason

• the Big-Endian/ Little-Endian problem for int, uint , [u]short, [u]long etc.
• control exact the type use by the field
• reference type array
• windows COM size vs CLR Size vs use wanted size eg. bool is 4 byte for COM , 1 byte for CLR , and probably 1 bit is users want (separate 1 byte to 8 bit filed )

[Clockwork-Muse]

None of these are unique to C#, and all behave identically or similarly in C/C++ (some of them, particularly "type used" may depend on compiler flags, which is its own pain).

Why are you asking, and what are you wanting to achieve? If you're trying to create a struct that exactly maps to some binary format, you'd normally look at the StructLayout Attribute, but there may be other or better ways to do whatever it is you're trying to achieve.

[John0King]

why are you asking

because I suffering from the Big-Endian/ Little-Endian problem 🤣, and I saw C/C++ use typedef for file structure (PE or March-O) without a struct reader/writer.
and for C# side, it seem to me is to create a struct reader/writer for produce the binary (I'm working with something similar to network stream )

[Clockwork-Muse]

You're conflating multiple things here.

because I suffering from the Big-Endian/ Little-Endian problem

Most consumer desktops and servers are currently Little-Endian (although most phones and many tablet devices are ARM, and which endianness it has is possibly per-device). That aside, C/C++ doesn't do extra magic here, and you'd still be manually flipping byte order.

I saw C/C++ use typedef for file structure (PE or March-O)

Presumably, this was behind an #ifdef or something for conditional compilation depending on whether you were building a Windows or Mac binary, so that the rest of the reading code just references one type. You can do the exact same thing in C# (and in fact there are a number of places where that's used in this repository for Windows vs Linux interop).

without a struct reader/writer.
and for C# side, it seem to me is to create a struct reader/writer for produce the binary (I'm working with something similar to network stream )

You can create a struct in C# that can be mapped directly to native or byte memory, or to files - that's part of the point of the StructLayout attribute, if you need to tweak things. You don't have to create a reader/writer for that.

What you will have to create a reader/writer for, even in C/C++, is flipping byte order for those fields that require it, to interface with those parts of the protocol that requires a specific endianness (ie, host to network endianness). If you're only doing on-machine stuff, though (like interprocess communication, or memory-mapped files), that's not a concern.

---

Unless you can provide a more concrete example here, I'm not immediately seeing anything that's actionable.

[John0King]

@Clockwork-Muse
1st, thank you for answer me those question. but I'm still confusing how those thing be done in C++ ，
https://github.com/zhlynn/zsign/blob/master/archo.cpp
and https://github.com/zhlynn/zsign/blob/27016df1150eeaf27e8d8c148ccb3cad10d31e73/archo.h#L24-L39 there are uint8_t ,uint_32_t type (I'm don't know much C++, and I'm sorry for that) this is about reading file, so I think it must be the same for big-endian/littel-endian.

[Clockwork-Muse]

What do you mean?

uint8_t is equivalent to byte
uint_32_t is equivalent to uint32

Those are type definitions being used to be clear about byte sizes of types (Because C/C++ doesn't actually explicitly define the size of integral types) - and nothing else. They're used in struct definitions in C/C++, just as they would be in C#.

In ZArchO::Init, the passed byte pointer is split and mapped to types. m_pHeader = (mach_header *)m_pBase; is taking the raw bytes and mapping it to a struct, equivalent to MemoryMarshal.AsRef. At this point, it's possible to access portions of the mapped memory via struct properties. However, for any multi-byte fields, there's a requirement to convert from the file byte order (which could be either endianness) to the endianness of the system (which appears to be assumed to be little-endian). That's done by the BO function - for instance, the line:

for (uint32_t i = 0; i < BO(m_pHeader->ncmds); i++)

Is iterating through all the commands in the file, so it's required to correctly swap the bytes of the variable if required, or the element count would be wrong.

[John0King]

Thanks a lot !

[John0King]

@Clockwork-Muse May I ask about the Marshal and the SizeOf?

> [System.Runtime.InteropServices.StructLayout(LayoutKind.Sequential,Pack = 1, Size = 7)]public struct Good { public int A; public bool B; public short C; }
> System.Runtime.CompilerServices.Unsafe.SizeOf<Good>()
7
> System.Runtime.InteropServices.Marshal.SizeOf<Good>()
10
> [System.Runtime.InteropServices.StructLayout(LayoutKind.Explicit, Size = 7)]public struct Good { [FieldOffset(0)]public int A;[FieldOffset(0)] public bool B;[FieldOffset(0)] public short C; }
> System.Runtime.InteropServices.Marshal.SizeOf<Good>()
7
> System.Runtime.CompilerServices.Unsafe.SizeOf<Good>()
7
> [System.Runtime.InteropServices.StructLayout(LayoutKind.Sequential,Pack = 1, Size = 7)]public struct Good { public int A; public bool B; public short C; }
> System.Runtime.CompilerServices.Unsafe.SizeOf<Good>()
7
> System.Runtime.InteropServices.Marshal.SizeOf<Good>()
10

> public enum G : byte { Y,G,X  }
> [System.Runtime.InteropServices.StructLayout(LayoutKind.Sequential)]public struct Good { public int A; public bool B; public short C; public G D; }
> System.Runtime.CompilerServices.Unsafe.SizeOf<Good>()
12
> System.Runtime.InteropServices.Marshal.SizeOf<Good>()
12
> [System.Runtime.InteropServices.StructLayout(LayoutKind.Auto)]public struct Good { public int A; public bool B; public short C; public G D; }
> System.Runtime.InteropServices.Marshal.SizeOf<Good>()
System.ArgumentException: Type 'Submission#59+Good' cannot be marshaled as an unmanaged structure; no meaningful size or offset can be computed.
  + System.Runtime.InteropServices.Marshal.SizeOfHelper(System.Type, bool)
  + System.Runtime.InteropServices.Marshal.SizeOf<T>()
> System.Runtime.CompilerServices.Unsafe.SizeOf<Good>()
8
> 

[Clockwork-Muse]

I didn't look too hard at this before:

• In at least some instances, the size is being driven by StructLayout.Size (ie, the size might normally be smaller, see next)
• The use of multiple [FieldOffset(0)]s means that the various fields overlay each other. This is rarely what you want.

[Clockwork-Muse]

System.Runtime.CompilerServices.Unsafe.SizeOf<Good>()
System.Runtime.InteropServices.Marshal.SizeOf<Good>()

From the look of this and this

Unsafe.SizeOf returns the managed size, and Marshal.SizeOf returns the size required for native interop scenarios.

cannot be marshaled as an unmanaged structure; no meaningful size or offset can be computed.

See this

[John0King]

it's painful that Marshal and Unsafe tread it differently .
base on my test

1. Unsafe.SizeOf<T>() is compatible with MemoryMarshel.Write<T>(Span<byte> buffer)
2. Marshal.SizeOf<T>() is compatible with Marshal.StructureToPtr<T>()

and there is a Unsafe.Wite Unafe.WriteUnaligned I wasn't test it yet.

[John0King]

Write and WriteUnaligned seems the same as MemoryMarshel.Write

byte[] UnsafeWrite<T>(T x) {
.     var s = Unsafe.SizeOf<T>();
.     Span<byte> buffer = new byte[s];
.     ref var buf_r = ref MemoryMarshal.GetReference(buffer);
.     Unsafe.WriteUnaligned(ref buf_r, x);
.     return buffer.ToArray();
.  }
> UnsafeWrite(g)
byte[8] { 33, 0, 0, 0, 1, 0, 55, 0 }
> structToByte(g)
byte[12] { 33, 0, 0, 0, 1, 0, 0, 0, 55, 0, 0, 0 }
> unsafe byte[] UnsafeWrite2<T>(T x)
. {
.     var s = Unsafe.SizeOf<T>();
.     Span<byte> buffer = new byte[s];
.     ref var buf_r = ref MemoryMarshal.GetReference(buffer);
.     var pp = Unsafe.AsPointer(ref buf_r);
.     Unsafe.Write(pp, x);
.     return buffer.ToArray();
. }
> UnsafeWrite2(g)
byte[8] { 33, 0, 0, 0, 1, 0, 55, 0 }
> UnsafeWrite(g)
byte[8] { 33, 0, 0, 0, 1, 0, 55, 0 }
> structToByte(g)
byte[12] { 33, 0, 0, 0, 1, 0, 0, 0, 55, 0, 0, 0 }
> 

[ChristophTF]

Marshal.StructureToPtr<T>() always copies the struct into a newly allocated buffer with artificially created layout (which I don't even know the rules for). This is seldomly what you would want, since it kills the benefit of interacting directly with memory.

[Clockwork-Muse]

Unsafe.SizeOf() is compatible with MemoryMarshel.Write(Span buffer)

Sort of. Unsafe.SizeOf is only good for "same compiler" comparisons. If you're paranoid, this means that you can only use it for same-deployment serialization (in-process or maybe same-machine cross-process serialization). You cannot use it for any real persistence mechanisms, because if you upgrade your app the layout may have changed, unless you control the managed representation.

[John0King]

Unsafe.SizeOf is only good for "same compiler" comparisons.

but from follow result , does that means MemoryMarshal is also for the same-machine serialization ?

> [System.Runtime.InteropServices.StructLayout(LayoutKind.Sequential)]public struct Good { public int A; public bool B; public short C;  }
> var g = new Good { A = 33, B = true, C = 55 };
> UnsafeWrite2(g)
byte[8] { 33, 0, 0, 0, 1, 0, 55, 0 }
> UnsafeWrite(g)
byte[8] { 33, 0, 0, 0, 1, 0, 55, 0 }
> structToByte(g)
byte[12] { 33, 0, 0, 0, 1, 0, 0, 0, 55, 0, 0, 0 }
> 

in my understand is that Marshal is about interop with C/C++ dll （specifically win32）, so a bool will be 4 byte (align with win32 BOOL), but I wonder will it be different for linux/mac/android/ios. and the MemoryMarshal will represent CLR native size (it should be the same for different CLR implantation) , but I'm note sure about that ,

Maybe someone who in the runtime/CLR team should post an answer or blog about the new Unsafe MemroryMarshal stuff.

[Clockwork-Muse]

Make your type blittable, so it has the same layout on both the managed and unmanaged sides. bool isn't blittable - and apparently there are three potential versions/sizes on Windows alone - so you'd normally use whatever the underlying sized type is, potentially wrapping it in a struct to make it act more like a boolean.

That aside, there are additional ways to control how booleans and other non-blittable types are marshaled/serialized, but you haven't provided any info about what you're actually trying to do, so it's not possible to suggest further solutions.

[ChristophTF]

If you want to deal with binary file formats that specify a certain endianess for their fields, and are concerned with not knowing the endianess of the executing system (which, as @Clockwork-Muse pointed out, is Little-Endian for the most popular platforms), one option is to leverage the type system:

For representing a 64-bit unsigned little-endian integer, you could define your own type

[StructLayout(LayoutKind.Sequential)]
readonly struct UInt64LE
{
    private readonly ulong val;
    
    private UInt64LE(ulong val) => this.val = val;
    
    public static implicit operator ulong(UInt64LE num)
        => BitConverter.IsLittleEndian
        ? num.val
        : BinaryPrimitives.ReverseEndianness(num.val);
    
    public static implicit operator UInt64LE(ulong num)
        => BitConverter.IsLittleEndian
        ? new(num)
        : new(BinaryPrimitives.ReverseEndianness(num));
}

which handles the endianess-conversion transparently and can be employed in a file-structure mapping struct:

[StructLayout(LayoutKind.Sequential)]
struct FileBinaryLayoutStruct
{
    // ...
    public UInt64LE field1;
    // ...
}

Notice how in the SharpLab sample the condition gets eliminated by the JIT-compiler, making the access as efficient as it could be in C/C++. Also, by utilizing the conversion operator, the accessing code looks the same as if it was a regular integer.

I would therefore argue that C#'s typesystem suits very well for this kind of stuff, not even needing to specify the endianess at compile time via preprocessor macros.

[Clockwork-Muse]

Note that this example assumes that the value is read from a little-endian source stream, which will not be the case for most network-related scenarios, and may not in various file formats. This may not be sufficient for all file formats - sample code posted earlier allows the file format to be either little or big endian. Check documentation as appropriate.

[alexrp]

[StructLayout(LayoutKind.Sequential)]

By the way, you probably don't want platform-specific packing rules for something like this, so set Pack = 1.