Skip to content

Commit

Permalink
rename SampleConcurrent to SampleParallel
Browse files Browse the repository at this point in the history
  • Loading branch information
AnthonyLloyd committed Aug 21, 2024
1 parent c5d1894 commit 4ec257e
Show file tree
Hide file tree
Showing 10 changed files with 88 additions and 88 deletions.
2 changes: 1 addition & 1 deletion Comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Size is also a better representation of comparison especially for collections or
There are examples where increasing on one axis while decreasing on others can lead to smaller cases e.g. if Version fails for `2 * ma + mi + bu ≥ 255 * 2`
CsCheck will be able to shrink to `255.0.0` but [Hedgehog](https://github.com/hedgehogqa) won't.
For concurrency testing random shrinkers also has an advantage. Concurrency tests may not fail deterministically.
For parallel testing random shrinkers also has an advantage. Parallel tests may not fail deterministically.
This is a real problem for path explorer shrinkers. The only solution is to repeat each test multiple times (10 for QuickCheck) since they need to follow defined paths.
For a random shrinker you can just continue testing different random cases until one fails and limit the size to that each time.

Expand Down
108 changes: 54 additions & 54 deletions CsCheck/Check.cs

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions CsCheck/CsCheck.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,17 @@ This gives the following advantages:
- Random testing and shrinking are parallelized. This and PCG make it very fast.
- Shrunk cases have a seed value. Simpler examples can easily be reproduced.
- Shrinking can be continued later to give simpler cases for high dimensional problems.
- Concurrency testing and random shrinking work well together.
- Parallel testing and random shrinking work well together.

CsCheck also makes concurrency, performance and regression testing simple and fast.
CsCheck also makes parallel, performance and regression testing simple and fast.
</Description>
<Authors>Anthony Lloyd</Authors>
<Owners>Anthony Lloyd</Owners>
<Copyright>Copyright 2024</Copyright>
<PackageLicenseExpression>Apache-2.0</PackageLicenseExpression>
<PackageProjectUrl>http://github.com/AnthonyLloyd/CsCheck</PackageProjectUrl>
<PackageIcon>CsCheck.png</PackageIcon>
<PackageTags>quickcheck;random;model-based;metamorphic;concurrency;performance;causal-profiling;regression;testing</PackageTags>
<PackageTags>quickcheck;random;model-based;metamorphic;parallel;performance;causal-profiling;regression;testing</PackageTags>
<Version>4.0.0</Version>
<PackageReleaseNotes>
Update to .NET 8
Expand Down
8 changes: 4 additions & 4 deletions CsCheck/Utils.cs
Original file line number Diff line number Diff line change
Expand Up @@ -475,7 +475,7 @@ public static bool ModelEqual<T, M>(T actual, M model)
return actual.Equals(model);
}

internal static void Run<T>(T concurrentState, (string, Action<T>)[] operations, int threads, int[]? threadIds = null)
internal static void Run<T>(T parallelState, (string, Action<T>)[] operations, int threads, int[]? threadIds = null)
{
Exception? exception = null;
var opId = -1;
Expand All @@ -488,7 +488,7 @@ internal static void Run<T>(T concurrentState, (string, Action<T>)[] operations,
while ((i = Interlocked.Increment(ref opId)) < operations.Length)
{
if (threadIds is not null) threadIds[i] = tid;
try { operations[i].Item2(concurrentState); }
try { operations[i].Item2(parallelState); }
catch (Exception e)
{
if (exception is null)
Expand All @@ -505,7 +505,7 @@ internal static void Run<T>(T concurrentState, (string, Action<T>)[] operations,
if (exception is not null) throw exception;
}

internal static void RunReplay<T>(T concurrentState, (string, Action<T>)[] operations, int threads, int[] threadIds)
internal static void RunReplay<T>(T parallelState, (string, Action<T>)[] operations, int threads, int[] threadIds)
{
Exception? exception = null;
var runners = new Thread[threads];
Expand All @@ -518,7 +518,7 @@ internal static void RunReplay<T>(T concurrentState, (string, Action<T>)[] opera
{
if (threadIds[i] == tid)
{
try { operations[i].Item2(concurrentState); }
try { operations[i].Item2(parallelState); }
catch (Exception e)
{
if (exception is null)
Expand Down
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ This gives the following advantages over tree based shrinking libraries:
- Random testing and shrinking are parallelized. This and PCG make it very fast.
- Shrunk cases have a seed value. Simpler examples can easily be reproduced.
- Shrinking can be continued later to give simpler cases for high dimensional problems.
- Concurrency testing and random shrinking work well together. Repeat is not needed.
- Parallel testing and random shrinking work well together. Repeat is not needed.

See [why](https://github.com/AnthonyLloyd/CsCheck/blob/master/Why.md) you should use it, the [comparison](https://github.com/AnthonyLloyd/CsCheck/blob/master/Comparison.md) with other random testing libraries, or how CsCheck does in the [shrinking challenge](https://github.com/jlink/shrinking-challenge).
In one [shrinking challenge test](https://github.com/jlink/shrinking-challenge/blob/main/challenges/binheap.md) CsCheck managed to shrink to a new smaller example than was thought possible and is not reached by any other testing library.
Expand All @@ -21,7 +21,7 @@ CsCheck also has functionality to make multiple types of testing simple and fast
- [Random testing](#Random-testing)
- [Model-based testing](#Model-based-testing)
- [Metamorphic testing](#Metamorphic-testing)
- [Concurrency testing](#Concurrency-testing)
- [Parallel testing](#Parallel-testing)
- [Causal profiling](#Causal-profiling)
- [Regression testing](#Regression-testing)
- [Performance testing](#Performance-testing)
Expand Down Expand Up @@ -267,21 +267,21 @@ public void MapSlim_Metamorphic()
}
```

## Concurrency testing
## Parallel testing

CsCheck has support for concurrency testing with full shrinking capability.
A concurrent sequence of operations are run on an initial state and the result is compared to all the possible linearized versions.
At least one of these must be equal to the concurrent version.
CsCheck has support for parallels testing with full shrinking capability.
A number of operations are run on an initial state in parallel and the result is compared to all the possible linearized versions.
At least one of these must be equal to the parallel result.

Idea from John Hughes [talk](https://youtu.be/1LNEWF8s1hI?t=1603) and [paper](https://github.com/AnthonyLloyd/AnthonyLloyd.github.io/raw/master/public/cscheck/finding-race-conditions.pdf). This is actually easier to implement with CsCheck than QuickCheck because the random shrinking does not need to repeat each step as QuickCheck does (10 times by default) to make shrinking deterministic.

### SetSlim
```csharp
[Fact]
public void SetSlim_Concurrency()
public void SetSlim_Parallel()
{
Gen.Byte.Array.Select(a => new SetSlim<byte>(a))
.SampleConcurrent(
.SampleParallel(
Gen.Byte.Operation<SetSlim<byte>>((l, i) => { lock (l) l.Add(i); }),
Gen.Int.NonNegative.Operation<SetSlim<byte>>((l, i) => { if (i < l.Count) { var _ = l[i]; } }),
Gen.Byte.Operation<SetSlim<byte>>((l, i) => { var _ = l.IndexOf(i); }),
Expand Down Expand Up @@ -610,7 +610,7 @@ timeout - The timeout in seconds to use for Faster (default 60 seconds).
print - A function to convert the state to a string for error reporting (default Check.Print).
equal - A function to check if the two states are the same (default Check.Equal).
sigma - For Faster sigma is the number of standard deviations from the null hypothesis (default 6).
replay - The number of times to retry the seed to reproduce a SampleConcurrent fail (default 100).
replay - The number of times to retry the seed to reproduce a SampleParallel fail (default 100).

Global defaults can also be set via environment variables:

Expand Down
4 changes: 2 additions & 2 deletions Tests/CacheTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ class ConcurrentDictionaryCache<K, V> : ConcurrentDictionary<K, V>, ICache<K, V>
}

[Fact]
public void GetOrAddAtomicAsync_SampleConcurrent()
public void GetOrAddAtomicAsync_SampleParallel()
{
Check.SampleConcurrent(
Check.SampleParallel(
Gen.Const(() => new ConcurrentDictionaryCache<int, int>()),
Gen.Int[1, 5].Operation<ConcurrentDictionaryCache<int, int>>((d, i) => d.GetOrAddAtomicAsync(i, i => Task.FromResult(i)).AsTask()),
equal: (a, b) => Check.Equal(a.Keys, b.Keys),
Expand Down
12 changes: 6 additions & 6 deletions Tests/CheckTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -228,20 +228,20 @@ public void SampleModelBased_ConcurrentBag()
}

[Fact]
public void SampleConcurrent_ConcurrentBag()
public void SampleParallel_ConcurrentBag()
{
Gen.Int.List[0, 5].Select(l => new ConcurrentBag<int>(l))
.SampleConcurrent(
.SampleParallel(
Gen.Int.Operation<ConcurrentBag<int>>(i => $"Add({i})", (bag, i) => bag.Add(i)),
Gen.Operation<ConcurrentBag<int>>("TryTake()", bag => bag.TryTake(out _))
);
}

[Fact]
public void SampleConcurrent_ConcurrentDictionary()
public void SampleParallel_ConcurrentDictionary()
{
Gen.Dictionary(Gen.Int[0, 100], Gen.Byte)[0, 10].Select(l => new ConcurrentDictionary<int, byte>(l))
.SampleConcurrent(
.SampleParallel(
Gen.Int[0, 100].Select(Gen.Byte)
.Operation<ConcurrentDictionary<int, byte>>(t =>$"d[{t.Item1}] = {t.Item2}", (d, t) => d[t.Item1] = t.Item2),

Expand All @@ -251,10 +251,10 @@ public void SampleConcurrent_ConcurrentDictionary()
}

[Fact]
public void SampleConcurrent_ConcurrentQueue()
public void SampleParallel_ConcurrentQueue()
{
Gen.Int.List[0, 5].Select(l => new ConcurrentQueue<int>(l))
.SampleConcurrent(
.SampleParallel(
Gen.Int.Operation<ConcurrentQueue<int>>(i => $"Enqueue({i})", (q, i) => q.Enqueue(i)),
Gen.Operation<ConcurrentQueue<int>>("TryDequeue()", q => q.TryDequeue(out _))
);
Expand Down
4 changes: 2 additions & 2 deletions Tests/SieveLruCacheTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,9 @@ public void SampleModelBased()
}

[Fact]
public void SampleConcurrent()
public void SampleParallel()
{
Check.SampleConcurrent(
Check.SampleParallel(
Gen.Const(() => new SieveLruCache<int, int>(4)),
Gen.Int[1, 5].Operation<SieveLruCache<int, int>>((d, i) => d.GetOrAdd(i, i => i)),
equal: (a, b) => Check.Equal(a.Keys, b.Keys),
Expand Down
12 changes: 6 additions & 6 deletions Tests/SlimCollectionsTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ public void ListSlim_ModelBased()
}

[Fact]
public void ListSlim_Concurrency()
public void ListSlim_Parallel()
{
Gen.Byte.Array.Select(a => new ListSlim<byte>(a))
.SampleConcurrent(
.SampleParallel(
Gen.Byte.Operation<ListSlim<byte>>((l, i) => { lock (l) l.Add(i); }),
Gen.Int.NonNegative.Operation<ListSlim<byte>>((l, i) => { if (i < l.Count) { var _ = l[i]; } }),
Gen.Int.NonNegative.Select(Gen.Byte).Operation<ListSlim<byte>>((l, t) => { if (t.Item1 < l.Count) l[t.Item1] = t.Item2; }),
Expand All @@ -46,10 +46,10 @@ public void SetSlim_ModelBased()
}

[Fact]
public void SetSlim_Concurrency()
public void SetSlim_Parallel()
{
Gen.Byte.Array.Select(a => new SetSlim<byte>(a))
.SampleConcurrent(
.SampleParallel(
Gen.Byte.Operation<SetSlim<byte>>((l, i) => { lock (l) l.Add(i); }),
Gen.Int.NonNegative.Operation<SetSlim<byte>>((l, i) => { if (i < l.Count) { var _ = l[i]; } }),
Gen.Byte.Operation<SetSlim<byte>>((l, i) => { var _ = l.IndexOf(i); }),
Expand Down Expand Up @@ -116,10 +116,10 @@ public void MapSlim_Metamorphic()
}

[Fact]
public void MapSlim_Concurrency()
public void MapSlim_Parallel()
{
Gen.Dictionary(Gen.Int, Gen.Byte).Select(d => new MapSlim<int, byte>(d))
.SampleConcurrent(
.SampleParallel(
Gen.Int.Select(Gen.Byte).Operation<MapSlim<int, byte>>((m, t) => { lock (m) m[t.Item1] = t.Item2; }),
Gen.Int.NonNegative.Operation<MapSlim<int, byte>>((m, i) => { if (i < m.Count) { var _ = m.Key(i); } }),
Gen.Int.Operation<MapSlim<int, byte>>((m, i) => { var _ = m.IndexOf(i); }),
Expand Down
2 changes: 1 addition & 1 deletion Why.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Fluent style composition similar to LINQ is a much more robust and extensible op
- Caches and collections - often a key part of server and client side code these can be tested against a suitable simplified test model with `Model Based` testing.
- Calculations and algorithms - often possible to generalize examples for calculations and algorithms and check the result given the input. Algorithm often have properties they must guarantee. Rounding error issues automatically tested.
- Code refactoring - keep a copy of the original code with the test, refactor for simplicity and performance, safe in the knowledge it still produces the same results. Pair with a `Faster` test to monitor the relative performance over a range of inputs. Or if a copy is not feasible create a `Regression` test to comprehensively make sure there is no change.
- Multithreading and concurrency - test on the same object instance across multiple threads and examples. Shrink even works for `Concurrency` testing.
- Multithreading - test on the same object instance across multiple threads and examples. Shrink even works for `Parallel` testing.

## Raise Your Game

Expand Down

0 comments on commit 4ec257e

Please sign in to comment.