feat: TestAdapter BenchmarkCase constraints

[workgroupengineering]

Add TestAdapter BenchmarkCase constraints

Use Case

Ensure that the performance of a method remains within specification throughout versions.

How works

Adding attributes that specify the constraints to apply to BenchmarkCase. You can add a custom constraint by adding a class that inherits from BenchmarkCaseConstraintAttribute.

Example of use

using BenchmarkDotNet.Attributes;
using System.Threading;

namespace BenchmarkDotNet.Samples
{
    // It is very easy to use BenchmarkDotNet. You should just create a class
    public class IntroBasic
    {
        // And define a method with the Benchmark attribute
        [Benchmark]
        [TestAdapter.Annotations.MeanConstraint(TestAdapter.Annotations.ComparisonOperator.Less, 14)]
        public void Sleep() => Thread.Sleep(10);

        // You can write a description for your method.
        [Benchmark(Description = "Thread.Sleep(10)")]
        public void SleepWithDescription() => Thread.Sleep(10);
    }
}

Example of output

BenchmarkDotNet.Samples (net8.0)
  Tests in group: 1
   Total Duration: 23,2 sec

Outcomes
   1 Failed

 BenchmarkDotNet.Samples.IntroBasic.Sleep
   Source: IntroBasic.cs line 10
   Duration: 23,2 sec

  Message: 
// Constraint Errors: Mean is greater or equal that expected value 14


  Standard Output: 
IntroBasic.Sleep: DefaultJob
Runtime = .NET 8.0.6 (8.0.624.26715), X64 RyuJIT AVX2; GC = Concurrent Workstation
-------------------- Histogram --------------------
[15.459 ms ; 15.651 ms) | @@@@@@@@@@@@@@@
---------------------------------------------------
Mean = 15.531 ms, StdErr = 0.010 ms (0.07%), N = 15, StdDev = 0.040 ms
Min = 15.480 ms, Q1 = 15.493 ms, Median = 15.542 ms, Q3 = 15.550 ms, Max = 15.630 ms
IQR = 0.057 ms, LowerFence = 15.407 ms, UpperFence = 15.636 ms
ConfidenceInterval = [15.488 ms; 15.573 ms] (CI 99.9%), Margin = 0.043 ms (0.27% of Mean)
Skewness = 0.68, Kurtosis = 3.05, MValue = 2

[AndreyAkinshin]

@workgroupengineering thanks for the PR! I like the idea of analytical constraints for test cases, but the current approach is too fragile (Mean and StdDev are not robust metrics) and non-portable (absolute thresholds may be valid on a fixed hardware, but can't be reused as-is on an arbitrary machine). Providing such features out of the box is dangerous since it provokes library misuse and misleading test failures.

[workgroupengineering]

@workgroupengineering thanks for the PR!

Thanks for the reply.

I like the idea of analytical constraints for test cases, but the current approach is too fragile (Mean and StdDev are not robust metrics) and non-portable (absolute thresholds may be valid on a fixed hardware, but can't be reused as-is on an arbitrary machine).

It is not intended to be portable at all, but rather intended for specific scenarios with a specific hardware platform(s), and allow the developer to create custom constraints based on the needs.

Providing such features out of the box is dangerous since it provokes library misuse and misleading test failures.

I think by documenting the functionality, this problem might not occur.

[AndreyAkinshin]

I think by documenting the functionality, this problem might not occur.

Not so many people read the documentation. And when they misuse the API, they blame the library and increase the number of support requests. Therefore, I don't like adding non-essential APIs that are easy to misuse.

If you feel like there are use cases for these APIs/features, you are welcome to publish your own NuGet package with extensions for BenchmarkDotNet.