7.8.0

FlatSharp 7.8.0 contains a small number of new features and bugfixes thanks to @sssooonnnggg and @trumully! Thank you for the contributions!

Features

• Unions can how have manually set discriminator values, ie union MyUnion { str : string = 3, MyTable = 6 }
• Using the <FlatSharpGenerateMethods>true</FlatSharpGenerateMethods> in your .csproj file now allows generating .ToString() methods on certain FlatSharp objects. More in the future!
• Builds targeting .NET 9 (no .NET 9 features used yet)

Fixes

• Fix an issue where \ and " characters were not escaped when present in a FlatBuffer attribute value.

Other Changes

• Removal of the FlatSharp object pooling experimental feature.

What's Changed

• Fix pipelines by @jamescourtney in #446
• chore: create output directory if not exists by @sssooonnnggg in #444
• fix: emit attribute with escaped quote and backslash by @sssooonnnggg in #445
• fix: fixed union generator when processing union with custom enum value by @sssooonnnggg in #447
• Add net 9 by @jamescourtney in #449
• Remove Object Pooling by @jamescourtney in #450
• Add ToString methods by @trumully in #437

New Contributors

• @sssooonnnggg made their first contribution in #444

Full Changelog: 7.7.0...7.8.0

7.7.0

FlatSharp 7.7.0 is a small feature release. Thanks to @trumully and @parched, unions now have implicit operators for assignment:

SomeUnion union = "hello";

Thanks for the contribution!

Additionally, a bug has been fixed relating to use of the required modifier with various setter options. Previous versions of FlatSharp would generate invalid C# if using required with a non-public setter. The new behavior is to always enforce the concept of required when serializing/parsing, but to only add the C# required keyword to a property when the setter's visibility is public.

What's Changed

• Fix required properties by @jamescourtney in #439
• Add implicit operator to Union types by @trumully in #438

New Contributors

• @trumully made their first contribution in #438

Full Changelog: 7.6.0...7.7.0

7.6.0

FlatSharp 7.6.0 is a small feature release that adds a couple of quality of life features:

1. New Match methods on Union types that accept delegates. These aren't the most performant but work well for one-offs or cases where you don't want to implement a Visitor.

2. Optional support for file visibility on types in FlatSharp-generated code. This reduces the clutter you'll see in your code from FlatSharp. Since file visibility is only supported on C# 11 and above, you will need to opt into this behavior:

        <PropertyGroup>
            <FlatSharpFileVisibility>true</FlatSharpFileVisibility>
        </PropertyGroup>

   From the command line: dotnet FlatSharp.Compiler.dll --file-visibility

In addition to these features, FlatSharp's primary unit tests now all run in NativeAOT mode as well as JIT mode thanks to the new NativeAOT-compatible MSTest runner.

What's Changed

• Update Samples for 7.5 by @jamescourtney in #429
• Add union match method by @jamescourtney in #430
• Update build system by @jamescourtney in #431
• Convert E2E Tests to use MSTest to enable NativeAOT unit testing by @jamescourtney in #432
• Refactor Benchmarks to work with AOT by @jamescourtney in #433
• Build refactor by @jamescourtney in #434
• Add file visibility option by @jamescourtney in #435

Full Changelog: 7.5.1...7.6.0

7.5.1

FlatSharp 7.5.1 is a minor release that contains important fixes for NativeAOT support.

• Fix .NET 8 Native AOT, Mono AOT, and add CI pipelines to ensure these don't regress again in the future.
• Remove some methods from IInputBuffer and ISpanWriter that led to the issues with AOT.

What's Changed

• Native aot stuff by @jamescourtney in #425
• Add some nativeaot test cases by @jamescourtney in #426
• Mono AOT Validation by @jamescourtney in #427
• Update packages and SpanWriter by @jamescourtney in #428

Full Changelog: 7.5.0...7.5.1

7.5.0

FlatSharp 7.5.0 is a medium-sized release with a few changes that may be significant for you. It's now published on Nuget.org. There are no breaking changes.

The primary focus of this release is to significantly reduce the size of the x64 assembly produced by the JITer. There are 3 ways that this is accomplished:

• String serialization is no longer inlined. This reduces code size substantially but does have a modest impact on serialization speed. There are other changes that offset most of the performance loss. However, FlatSharp should play much more nicely with your instruction cache now.

• Using ThrowHelper-style methods for throwing exceptions from hot paths.

• Remove most checked arithmetic. FlatSharp already uses safe methods for interacting with memory. The only checked operations that remain are multiplications and left shifts. This removes many branch instructions and further compacts the generated assembly. If you have a security need to retain checked arithmetic everywhere, please consider compiling with <CheckForOverflowUnderflow>true</CheckForOverflowUnderflow> in your csproj.

How much of a difference does this make? Let's look at a contrived example:

table MailingAddress (fs_serializer)
{
    to : string;
    street : string;
    city : string;
    zip_code : int;
}

Examining the bytes of code generated to serialize this results in:

Version	Performance	Table Serialize Bytes	String Serialize Bytes	Total
7.4.0	60ns	3888 (inlined)	0	3888
7.5.0	58ns	988 (method calls)	550	1538

This effect will scale for each string property in your schema, so while this example is contrived, the benefit should be large for applications where strings are a common data type.

Note: The focus is on shrinking the size of the code generated by the JIT. The generated C# is largely unchanged.

There are some other changes as well:

• Add README.md files to the FlatSharp NuGet packages.
• Add the [DebuggerTypeProxy] attribute to all generated classes. This ensures that much of the internal FlatSharp state is excluded from debugging views and makes the debugging experience more seamless.

What's Changed

• Update samples to latest by @jamescourtney in #413
• Refactor exceptions and string concat by @jamescourtney in #420
• Reduce JIT'ed Code Size and Optimize by @jamescourtney in #421
• Add NuGet READMEs by @jamescourtney in #422
• Add [DebuggerTypeProxy] to generated classes. by @jamescourtney in #423
• Give reference unions the exception treatment by @jamescourtney in #424

Full Changelog: 7.4.0...7.5.0

7.4.0

FlatSharp 7.4.0 is available on Nuget with a couple of nonbreaking changes:

• FlatSharp.Runtime supports .NET 8
• FlatSharp.Compiler now requires any of .NET 8, .NET 7, or .NET 6 to be installed (previously it supported only .NET 6).
• Fixes a bug where FlatSharp generated invalid code if FlatSharp was declared as a non-top-level namespace, such as Foo.Bar.FlatSharp.

What's Changed

• Update Google flatbuffers, add verify tests by @jamescourtney in #394
• Add Stryker Tests to Code Coverage by @jamescourtney in #397
• Delete Legacy Tests by @jamescourtney in #398
• Metadata attributes by @jamescourtney in #403
• Add global namespace test by @jamescourtney in #409
• Net8 by @jamescourtney in #412

Full Changelog: 7.2.3...7.4.0

7.2.3

FlatSharp 7.2.3 is a small release that fixes a few issues:

• FBS validation issues
• Issue where FBS files couldn't be loaded when on different drives on Windows systems
• Cleanup of FlatSharp Unit Tests

7.2.0

7.2.0 is a minor feature release that resolves a few bugs and adds a small number of enhancements to the FlatSharp compiler package.

New Compiler Switches

Select Deserializers

You can now select deserializers FlatSharp should generate. This allows some degree of control over code size from FlatSharp:

<PropertyGroup>
  <FlatSharpDeserializers>Lazy;Greedy</FlatSharpDeserializers>
</PropertyGroup>

Using the command line, you can pass --deserializers Progressive;GreedyMutable

Class Definitions Only

In the same vein, FlatSharp can be configured to only emit class definitions. This allows sharing a set of common schemas between projects.

<PropertyGroup>
  <FlatSharpClassDefinitionsOnly>true</FlatSharpClassDefinitionsOnly>
</PropertyGroup>

Using the command line, you can pass --class-definitions-only.

Input Files Only

Normally, FlatSharp will process all FBS files in the recursive graph, including include references that were not explicitly passed to FlatSharp. With this switch, FlatSharp can be configured to only emit output for explicitly-passed input files.

<PropertyGroup>
  <FlatSharpInputFilesOnly>true</FlatSharpInputFilesOnly>
</PropertyGroup>

Using the command line, you can pass --input-files-only.

Other Enhancements

7.2.0 includes a small set of quality-of-life enhancements:

• The set of command line options is now part of the file hash in the generated file header. This will make FlatSharp regenerate .cs files when the command line options have changed.
• All generated classes and vectors now have [DebuggerDisplay] attributes to make using the debugger a little less cumbersome.
• Bug #384 should now be fixed.

7.1.1

7.1.1 is a bugfix release to partially address #372. Negative default values should now work correctly.

7.1.0

FlatSharp 7.1 is an important update to 7.0 that brings several notable improvements! Thanks to @joncham and @bangfalse for their contributions to this release.

Deeper Unity Integration

Thanks to @joncham of Unity3D, FlatSharp now supports Unity's NativeArray as a built-in vector type. To enable this functionality, pass the --unity-assembly-path argument with the value of the path to UnityEngine.dll to the FlatSharp Compiler. This enables the new vector type UnityNativeArray:

table MyTable
{
    Values : [ Vec3 ] (fs_vector:"UnityNativeArray");
}

Performance Improvements

One of the big improvements in FlatSharp 7.0 was vector performance. I even spent a lot of time bragging about it in the release notes! And I wasn't wrong.... when using default settings. FlatSharp 7.0 improved vector performance by almost 25% relative to 6.3.5, which is indeed a nice improvement. However, what I didn't observe was that FlatSharp 7.0 actually regressed by 32% relative to 6.3.5 when PGO was turned on!

	Type	<---	Default	--->	<---	PGO	--->
		6.3.5	7.0.2	7.1.0	6.3.5	7.0.2	7.1.0
Lazy	Ref	281ns	184ns	172ns	111ns	118ns	115ns
Lazy	Value	172	81	81	51	56	51
Progressive	Ref	472	355	232	283	298	195
Progressive	Value	156	136	124	83	127	83
Greedy	Ref	416	301	214	253	373	174
Greedy	Value	175	139	99	91	112	74
GreedyMutable	Ref	404	297	268	232	367	211
GreedyMutable	Value	151	141	110	78	113	75
Relative Perf		100%	73 %	58%	53%	70%	44%

The table above shows all the permutations of FlatSharp version, Reference/Value type, Deserialization mode, and PGO On/Off. The tl;dr of the data is that FlatSharp 7.1 is the fastest FlatSharp ever, with PGO on or off.

• When PGO is off, It's typically about 25% faster than version 7.0, and almost twice (!) as fast as 6.3.5.
• With PGO on, it's about 20% faster than 6.3.5 and about 40% faster than 7.0.
• Greedy, GreedyMutable, and Progressive vectors see the biggest gains.

So, what changed?

Up until version 7.1, FlatSharp vectors have all included a generic class hosted in FlatSharp.Runtime that looked something like this:

public sealed class FlatBufferVector<T, TInputBuffer> : IList<T> where TInputBuffer : IInputBuffer { ... }

While C# generics accomplish many of the same roles as C++ templates, they are fundamentally different beasts. FlatSharp 7.1 emits unique class definitions for each vector much like how the C++ compiler emits unique instances for each template combination:

public sealed class VectorString<TInputBuffer> : IList<string> where TInputBuffer : IInputBuffer { ... }
public sealed class VectorInt<TInputBuffer> : IList<int> where TInputBuffer : IInputBuffer { ... }

An astute reader will notice that the largest improvements in the table above were for reference types instead of value types. This is not a coincidence. When using generic methods and classes, the JIT is very lazy and will often share implementations of methods with reference type generic arguments. This necessarily adds overhead for virtual indirection, which slows things down. By using entirely separate classes, we make the JIT to generate vector code that is tailored to each derived type, instead of the base type. PGO covers some of this up with devirtualization, which is why the gains are a bit less impressive there.

Finally, please note that these benchmarks are designed to stress the vector handling aspects of FlatSharp, so this does not mean that your overall performance will improve by these numbers when using version 7.1 unless your schema is trivially simple.

Testing Investments

FlatSharp is used in production in real-life applications, which is great. Please tell your friends! However, as a mostly-solo project, this means that releasing a new version is sometimes a little stressful, especially in a version like 7.1 where the entire Vector stack is being rewritten.

The combination of "used in production" and "solo developer" is a tricky line to walk, which is why I've held project is held to such a high bar (95%) on code coverage. There's always room to do better, and FlatSharp 7.1 address two major test gaps that should further increase the quality of the project.

The first is a dedicated test to ensure that generated code will build with C# language version 8, which prevents accidental uses of new(), switch expressions, or other fancy new language features from creeping into the generated code. This has caused a few bugs in the past because FlatSharp itself uses C# 11 features, so catching these accidents with automation is a win.

The second new automated test suite uses Stryker Mutator, which is a tool to measure the effectiveness of tests by injecting bugs into code and seeing which are not caught by the tests. Essentially, Stryker adds a bug into the source code, and if no tests fail, then that code clearly isn't tested. After all, tests that don't capture any bugs aren't very useful. While this sounds great (and it is!), the implementation is a little tricky with FlatSharp, since it is code to generate code. For example, FlatSharp might contain something like

public string GetAddMethod() => return "public void Add(int a, int b) { return a + b; }";

Stryker might perform this mutation

public string GetAddMethod() => return string.Empty;

While this makes a lot of sense for normal code, it's not very useful for FlatSharp. What we really want Stryker to do is to mutate the code inside the string so a + b changes to a - b. The solution, of course, is to run Stryker on the output of FlatSharp, not FlatSharp itself.

The FlatSharp Stryker tests exhaustively test the code FlatSharp emits for a single, relatively broad FlatBuffer schema. The unit tests do the opposite and cover every scenario across a ton of tiny schemas. It isn't practical to cover everything using Stryker, but the new tests exercise a wide range of FlatSharp's functionality. While not a substitute for unit tests, Stryker testing it is a great complement since it ensures FlatSharp's test suite is resilient to a wide variety of incidental changes.

And finally, despite spending the last few paragraphs talking about FlatSharp's previous test gaps, the new tests have identified no breaking bugs. They've exposed plenty of small things about code that could be better factored to be more testable and some small problems like inconsistent Exception types depending on deserialization mode. But overall, FlatSharp is in an excellent place from a quality standpoint.

What's Changed

• Move more tests! by @jamescourtney in #351
• Bump Newtonsoft.Json from 13.0.1 to 13.0.2 in /src/Benchmarks/Benchmark by @dependabot in #356
• Add support for Unity NativeArray as vector type by @joncham in #319
• Improve native array coverage by @jamescourtney in #360
• Continue Test Migration by @jamescourtney in #355
• RPC: allow null serviceImpl in BindService by @bangfalse in #361
• Add C# 8 Build Step by @jamescourtney in #362
• Create stryker.yml by @jamescourtney in #311
• Exclude some progressive stuff by @jamescourtney in #364
• Faster Vectors by @jamescourtney in #363
• Improve Stryker Coverage by @jamescourtney in #365

New Contributors

• @dependabot made their first contribution in #356
• @bangfalse made their first contribution in #361

Full Changelog: 7.0.2...7.1.0