.Net 7.0 Regex SourceGen

[TonyValenti]

@stephentoub is doing amazing work on the regex source gen.

1. Are sourcegen-compiled regexes expected to execute faster than today's runtime-compiled post-warmup regexes? If so, are there any metrics out yet about that? (Would love a blog post once the work is finished!)

2. Will 7.0's runtime compiled regexes be updated to use the sourcegen code? Why or why not?

[stephentoub]

@stephentoub is doing amazing work on the regex source gen.

Thank you.

Are sourcegen-compiled regexes expected to execute faster than today's runtime-compiled post-warmup regexes?

There are a few answers here:

• All of the parsing, analysis, optimization, and code generation (C#/IL) is done at build time with the source generator vs run-time for RegexOptions.Compiled, so first use / startup is faster with source generation.
• Source-generated regexes still require System.Text.RegularExpressions.dll, but they don't require much of the code in it, so a trimmed application can omit much of the size from System.Text.RegularExpressions.dll, and if nothing else is using reflection emit, much of the code associated with reflection emit.
• Compiled regexes are faster than interpreted ones, and some environments don't support JIT'ing, which RegexOptions.Compiled requires. So with the source generator, you get the throughput at least that of RegexOptions.Compiled regexes but it can be ahead-of-time compiled and thus support environments that don't support JIT'ing, in which case the source generator will be much faster than Compiled (which would fall back to the interpreter).
• RegexOptions.Compiled uses dynamic methods, which today don't support tiering. There are optimizations the JIT applies based on information it gleans from code having executed in tier 0 (e.g. dynamic PGO), and so whereas Compiled doesn't participate in tiering, source generated regexes do.
• For the most part, the structure of the regex compiler and the regex source generator are the same; in fact, the compiler was entirely rewritten for .NET 7 at the same time the source generator was being written so that they mirror each other, and as a result, the vast majority of what they do is the same, albeit with one spitting out IL directly and the other spitting out C#, which is then subject to whatever strategies the C# compiler employs for turning that C# into IL. There are, however, a handful of optimizations today that only apply to the source generator and not to RegexOptions.Compiled, typically in places where we can benefit from optimizations the C# compiler employs where we'd otherwise need to duplicate those in our direct writing to IL with RegexOptions.Compiled. For example, the C# compiler has multiple optimizations around switch statements and how to best emit them, and we take advantage of that in the source generator; so, for example, if you have a character class like [\p{L}], we will emit that with the source generator in a way that benefits from a switch statement and how the C# compiler lowers it, whereas we take a fallback path for Compiled that's good but not quite as good. Or with alternations, if the source generator can prove that every branch of the alternation begins with a different literal, it can emit it as a switch over the next input character, whereas the compiler will fall back to the normal implementation of trying each branch in turn. Whether these optimizations make an impactful difference depends on the pattern and the input.

Would love a blog post once the work is finished!

That's the plan.

Will 7.0's runtime compiled regexes be updated to use the sourcegen code?

There aren't that many regexes used in the core libraries, but the ones that are there and employ a constant pattern have mostly all been moved over already, e.g.

runtime/src/libraries/System.ComponentModel.TypeConverter/src/System/ComponentModel/Design/DesignerVerb.cs

Lines 28 to 29 in a06befc

	[RegexGenerator(@"\(\&.\)")]
	private static partial Regex GetParameterReplacementRegex();

runtime/src/libraries/System.Private.Xml/src/System/Xml/Serialization/SourceInfo.cs

Lines 19 to 24 in 6a889d2

	[RegexGenerator("([(][(](?<t>[^)]+)[)])?(?<a>[^[]+)[[](?<ia>.+)[]][)]?")]
	private static partial Regex Regex1();
	//((global::Microsoft.CFx.Test.Common.TypeLibrary.IXSType_9)o), @"IXSType_9", @"", true, true);
	[RegexGenerator("[(][(](?<cast>[^)]+)[)](?<arg>[^)]+)[)]")]
	private static partial Regex Regex2();

runtime/src/libraries/System.Private.DataContractSerialization/src/System/Runtime/Serialization/Globals.cs

Lines 834 to 836 in 8d12bc1

	public const string FullSRSInternalsVisiblePattern = @"^[\s]*System\.Runtime\.Serialization[\s]*,[\s]*PublicKey[\s]*=[\s]*(?i:00240000048000009400000006020000002400005253413100040000010001008d56c76f9e8649383049f383c44be0ec204181822a6c31cf5eb7ef486944d032188ea1d3920763712ccb12d75fb77e9811149e6148e5d32fbaab37611c1878ddc19e20ef135d0cb2cff2bfec3d115810c3d9069638fe4be215dbf795861920e5ab6f7db2e2ceef136ac23d5dd2bf031700aec232f6c6b1c785b4305c123b37ab)[\s]*$";
	[RegexGenerator(FullSRSInternalsVisiblePattern)]
	public static partial Regex FullSRSInternalsVisibleRegex();

https://github.com/dotnet/winforms/blob/1616cf133035beadf3c97573c8abecbdb6750967/src/System.Windows.Forms/src/System/Windows/Forms/ToolStripSettingsManager.cs#L212-L213
and so on.

Why or why not?

As high-level guidance, if a regex pattern is known at compile-time (rather than being dynamically-generated at run-time) and performance of the regex matters (e.g. it's something you would have previously used RegexOptions.Compiled for), it should use the source generator instead.