internal/testingiface: New testing.T equivalent interface and unit testing helpers #308
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sting helpers Reference: #254 Reference: #303 _Note: This change is easiest to view with whitespace changes ignored._ As part of upcoming efforts to consider prerelease versioning more pragmatically in the `tfversion` package checks, the need to properly unit test `(*testing.T).Skip()` and friends behaviors is more apparent. Similar to issues with testing `(*testing.T).Fatal()` behaviors, the Go standard library in general is not the most friendly to capture test helper behaviors since the `testing.T` implementation is not intended to be extensible. This change was initially intended to extend our current usage of `github.com/mitchellh/go-testing-interface` as a stopgap, however while attempting to implement skip assertion testing, the `RuntimeT` implementation did not call `runtime.Goexit()` and would cause tests to continue to execute test logic after skipping even though in reality they would not. While the tests could be fixed to ensure they still passed due to that behavior change, it felt like a good time as any to replace the now-unsuitable and archived `github.com/mitchellh/go-testing-interface` with our own internal implementation. The new `internal/testingiface.T` interface is a wholesale duplicate interface to the prior `github.com/mitchellh/go-testing-interface.T` interface. Swapping this interface in exported functions, such as `helper/resource.Test()` is only a compatibility issue for consumers if by chance they were implementing a wrapping function that also depended on `github.com/mitchellh/go-testing-interface.T` and passed that implementation detail directly to function calls of this Go module. This is a gray compatibility area as the intention of using `github.com/mitchellh/go-testing-interface.T` was for consumers to use real `*testing.T` implementations, which will work the same after this change, but it was not appropriately documented that extensions of exported functionality using `github.com/mitchellh/go-testing-interface.T` should not rely on that explicitly. Changing this detail is therefore a pragmatic compromise that it is more important to prioritize and document the intended usage while removing the archived dependency, which is almost exclusively how the ecosystem interacts with the functionality, over the potential of compatibility issues in a very small subset of the ecosystem. Since `(*testing.T).Skip()` is considered a passing test, there are now assertions whether skipping was called or not, as expected, in the existing `tfversion` package tests. The new `internal/testingiface` package also includes similar assertion helpers for `(*testing.T).Fatal()` and `(*testing.T).Parallel()` for consistency and cleans up some existing logic.
|
Draft Note: This change warrants additional consideration and changelog entries, so leaving in draft for now. |
bflad
added a commit
that referenced
this pull request
Mar 27, 2024
…versions in SkipBelow Reference: #303 Reference: #308 This change is mainly due to the internals of how github.com/hashicorp/go-version treats version comparisons when there is prerelease data. When the running Terraform CLI is a prerelease version and the given version is a patch version, `SkipBelow` will currently skip the test. However, Terraform CLI prerelease versions are semantically treated as candidates for the associated patch version and therefore should be tested. This adjusts `SkipBelow` for that intended behavior. In the unexpected use case that prerelease version checking is still needed, callers can (still) opt into giving a prerelease version, which will perform the check including prerelease data. The unit testing for the `tfversion` package skip functionality is still manual because `github.com/mitchellh/go-testing-interface` will not immediately stop the test logic Goroutine when the equivalent of `(*testing.T).Skip()` is called nor does it provide helpful troubleshooting information should a test fail as it raises a panic with a generic error message. Future changes could switch to using a different testing interface, however they would require a breaking change to the exported API of this Go module, so that design and effort is being done separately. With the addition of the new tests, but no logic changes: ``` === RUN Test_SkipBelow_SkipTest skip_below_test.go:23: Terraform CLI version 1.0.7 is below minimum version 1.1.0: skipping test --- SKIP: Test_SkipBelow_SkipTest (3.91s) === RUN Test_SkipBelow_RunTest --- PASS: Test_SkipBelow_RunTest (3.32s) === RUN Test_SkipBelow_Prerelease_EqualCoreVersion skip_below_test.go:77: Terraform CLI version 1.8.0-rc1 is below minimum version 1.8.0: skipping test --- SKIP: Test_SkipBelow_Prerelease_EqualCoreVersion (4.23s) === RUN Test_SkipBelow_Prerelease_HigherCoreVersion skip_below_test.go:101: Terraform CLI version 1.7.0-rc1 is below minimum version 1.8.0: skipping test --- SKIP: Test_SkipBelow_Prerelease_HigherCoreVersion (4.40s) === RUN Test_SkipBelow_Prerelease_HigherPrerelease skip_below_test.go:122: Terraform CLI version 1.7.0-rc1 is below minimum version 1.7.0-rc2: skipping test --- SKIP: Test_SkipBelow_Prerelease_HigherPrerelease (3.32s) === RUN Test_SkipBelow_Prerelease_LowerCoreVersion --- PASS: Test_SkipBelow_Prerelease_LowerCoreVersion (3.93s) === RUN Test_SkipBelow_Prerelease_LowerPrerelease --- PASS: Test_SkipBelow_Prerelease_LowerPrerelease (3.38s) ``` After logic changes (note only difference is `Test_SkipBelow_Prerelease_EqualCoreVersion`): ``` === RUN Test_SkipBelow_SkipTest skip_below_test.go:22: Terraform CLI version 1.0.7 is below minimum version 1.1.0: skipping test --- SKIP: Test_SkipBelow_SkipTest (4.17s) === RUN Test_SkipBelow_RunTest --- PASS: Test_SkipBelow_RunTest (3.21s) === RUN Test_SkipBelow_Prerelease_EqualCoreVersion --- PASS: Test_SkipBelow_Prerelease_EqualCoreVersion (3.21s) === RUN Test_SkipBelow_Prerelease_HigherCoreVersion skip_below_test.go:99: Terraform CLI version 1.7.0-rc1 is below minimum version 1.8.0: skipping test --- SKIP: Test_SkipBelow_Prerelease_HigherCoreVersion (3.18s) === RUN Test_SkipBelow_Prerelease_HigherPrerelease skip_below_test.go:120: Terraform CLI version 1.7.0-rc1 is below minimum version 1.7.0-rc2: skipping test --- SKIP: Test_SkipBelow_Prerelease_HigherPrerelease (3.21s) === RUN Test_SkipBelow_Prerelease_LowerCoreVersion --- PASS: Test_SkipBelow_Prerelease_LowerCoreVersion (3.04s) === RUN Test_SkipBelow_Prerelease_LowerPrerelease --- PASS: Test_SkipBelow_Prerelease_LowerPrerelease (3.41s) ```
bflad
added a commit
that referenced
this pull request
Mar 28, 2024
…versions in SkipBelow (#316) Reference: #303 Reference: #308 This change is mainly due to the internals of how github.com/hashicorp/go-version treats version comparisons when there is prerelease data. When the running Terraform CLI is a prerelease version and the given version is a patch version, `SkipBelow` will currently skip the test. However, Terraform CLI prerelease versions are semantically treated as candidates for the associated patch version and therefore should be tested. This adjusts `SkipBelow` for that intended behavior. In the unexpected use case that prerelease version checking is still needed, callers can (still) opt into giving a prerelease version, which will perform the check including prerelease data. The unit testing for the `tfversion` package skip functionality is still manual because `github.com/mitchellh/go-testing-interface` will not immediately stop the test logic Goroutine when the equivalent of `(*testing.T).Skip()` is called nor does it provide helpful troubleshooting information should a test fail as it raises a panic with a generic error message. Future changes could switch to using a different testing interface, however they would require a breaking change to the exported API of this Go module, so that design and effort is being done separately. With the addition of the new tests, but no logic changes: ``` === RUN Test_SkipBelow_SkipTest skip_below_test.go:23: Terraform CLI version 1.0.7 is below minimum version 1.1.0: skipping test --- SKIP: Test_SkipBelow_SkipTest (3.91s) === RUN Test_SkipBelow_RunTest --- PASS: Test_SkipBelow_RunTest (3.32s) === RUN Test_SkipBelow_Prerelease_EqualCoreVersion skip_below_test.go:77: Terraform CLI version 1.8.0-rc1 is below minimum version 1.8.0: skipping test --- SKIP: Test_SkipBelow_Prerelease_EqualCoreVersion (4.23s) === RUN Test_SkipBelow_Prerelease_HigherCoreVersion skip_below_test.go:101: Terraform CLI version 1.7.0-rc1 is below minimum version 1.8.0: skipping test --- SKIP: Test_SkipBelow_Prerelease_HigherCoreVersion (4.40s) === RUN Test_SkipBelow_Prerelease_HigherPrerelease skip_below_test.go:122: Terraform CLI version 1.7.0-rc1 is below minimum version 1.7.0-rc2: skipping test --- SKIP: Test_SkipBelow_Prerelease_HigherPrerelease (3.32s) === RUN Test_SkipBelow_Prerelease_LowerCoreVersion --- PASS: Test_SkipBelow_Prerelease_LowerCoreVersion (3.93s) === RUN Test_SkipBelow_Prerelease_LowerPrerelease --- PASS: Test_SkipBelow_Prerelease_LowerPrerelease (3.38s) ``` After logic changes (note only difference is `Test_SkipBelow_Prerelease_EqualCoreVersion`): ``` === RUN Test_SkipBelow_SkipTest skip_below_test.go:22: Terraform CLI version 1.0.7 is below minimum version 1.1.0: skipping test --- SKIP: Test_SkipBelow_SkipTest (4.17s) === RUN Test_SkipBelow_RunTest --- PASS: Test_SkipBelow_RunTest (3.21s) === RUN Test_SkipBelow_Prerelease_EqualCoreVersion --- PASS: Test_SkipBelow_Prerelease_EqualCoreVersion (3.21s) === RUN Test_SkipBelow_Prerelease_HigherCoreVersion skip_below_test.go:99: Terraform CLI version 1.7.0-rc1 is below minimum version 1.8.0: skipping test --- SKIP: Test_SkipBelow_Prerelease_HigherCoreVersion (3.18s) === RUN Test_SkipBelow_Prerelease_HigherPrerelease skip_below_test.go:120: Terraform CLI version 1.7.0-rc1 is below minimum version 1.7.0-rc2: skipping test --- SKIP: Test_SkipBelow_Prerelease_HigherPrerelease (3.21s) === RUN Test_SkipBelow_Prerelease_LowerCoreVersion --- PASS: Test_SkipBelow_Prerelease_LowerCoreVersion (3.04s) === RUN Test_SkipBelow_Prerelease_LowerPrerelease --- PASS: Test_SkipBelow_Prerelease_LowerPrerelease (3.41s) ```
bflad
added a commit
that referenced
this pull request
Mar 28, 2024
Reference: #303 Reference: #308 Reference: #316 These changes fully update the rest of the `tfversion` built-in checks for treating prereleases as semantically equal to the patch version. It also includes a fix for `RequireBelow` to handle the exactly equal condition. Previously, `RequireBelow` when given 1.8.0 would still run on 1.8.0 (and prereleases variants). Prior to logic changes: ``` === RUN Test_RequireAbove --- PASS: Test_RequireAbove (3.01s) === RUN Test_RequireAbove_Error --- PASS: Test_RequireAbove_Error (3.24s) === RUN Test_RequireAbove_Prerelease_EqualCoreVersion require_above_test.go:77: expected Terraform CLI version above 1.8.0 but detected version is 1.8.0-rc1 --- FAIL: Test_RequireAbove_Prerelease_EqualCoreVersion (3.97s) === RUN Test_RequireAbove_Prerelease_HigherCoreVersion --- PASS: Test_RequireAbove_Prerelease_HigherCoreVersion (3.17s) === RUN Test_RequireAbove_Prerelease_HigherPrerelease --- PASS: Test_RequireAbove_Prerelease_HigherPrerelease (3.03s) === RUN Test_RequireAbove_Prerelease_LowerCoreVersion --- PASS: Test_RequireAbove_Prerelease_LowerCoreVersion (3.13s) === RUN Test_RequireAbove_Prerelease_LowerPrerelease --- PASS: Test_RequireAbove_Prerelease_LowerPrerelease (3.14s) === RUN Test_RequireBelow --- PASS: Test_RequireBelow (2.94s) === RUN Test_RequireBelow_Error --- PASS: Test_RequireBelow_Error (2.59s) === RUN Test_RequireBelow_Prerelease_EqualCoreVersion require_below_test.go:80: expected t.Fatal(), got none --- FAIL: Test_RequireBelow_Prerelease_EqualCoreVersion (3.29s) === RUN Test_RequireBelow_Prerelease_HigherCoreVersion --- PASS: Test_RequireBelow_Prerelease_HigherCoreVersion (3.04s) === RUN Test_RequireBelow_Prerelease_HigherPrerelease --- PASS: Test_RequireBelow_Prerelease_HigherPrerelease (3.22s) === RUN Test_RequireBelow_Prerelease_LowerCoreVersion --- PASS: Test_RequireBelow_Prerelease_LowerCoreVersion (2.74s) === RUN Test_RequireBelow_Prerelease_LowerPrerelease --- PASS: Test_RequireBelow_Prerelease_LowerPrerelease (2.80s) === RUN Test_RequireBetween --- PASS: Test_RequireBetween (2.69s) === RUN Test_RequireBetween_Error_BelowMin --- PASS: Test_RequireBetween_Error_BelowMin (2.57s) === RUN Test_RequireBetween_Error_EqToMax --- PASS: Test_RequireBetween_Error_EqToMax (2.29s) === RUN Test_RequireBetween_Prerelease_MaxEqualCoreVersion require_between_test.go:114: expected t.Fatal(), got none --- FAIL: Test_RequireBetween_Prerelease_MaxEqualCoreVersion (3.24s) === RUN Test_RequireBetween_Prerelease_MinEqualCoreVersion require_between_test.go:141: expected Terraform CLI version between 1.8.0 and 1.9.0 but detected version is 1.8.0-rc1 --- FAIL: Test_RequireBetween_Prerelease_MinEqualCoreVersion (2.83s) === RUN Test_RequireBetween_Prerelease_MaxHigherCoreVersion --- PASS: Test_RequireBetween_Prerelease_MaxHigherCoreVersion (3.17s) === RUN Test_RequireBetween_Prerelease_MinHigherCoreVersion --- PASS: Test_RequireBetween_Prerelease_MinHigherCoreVersion (3.09s) === RUN Test_RequireBetween_Prerelease_MaxHigherPrerelease --- PASS: Test_RequireBetween_Prerelease_MaxHigherPrerelease (3.84s) === RUN Test_RequireBetween_Prerelease_MinHigherPrerelease --- PASS: Test_RequireBetween_Prerelease_MinHigherPrerelease (3.02s) === RUN Test_RequireBetween_Prerelease_MaxLowerCoreVersion --- PASS: Test_RequireBetween_Prerelease_MaxLowerCoreVersion (2.84s) === RUN Test_RequireBetween_Prerelease_MinLowerCoreVersion --- PASS: Test_RequireBetween_Prerelease_MinLowerCoreVersion (4.07s) === RUN Test_RequireBetween_Prerelease_MaxLowerPrerelease --- PASS: Test_RequireBetween_Prerelease_MaxLowerPrerelease (2.79s) === RUN Test_RequireBetween_Prerelease_MinLowerPrerelease --- PASS: Test_RequireBetween_Prerelease_MinLowerPrerelease (3.25s) === RUN Test_RequireNot --- PASS: Test_RequireNot (3.70s) === RUN Test_RequireNot_Error --- PASS: Test_RequireNot_Error (2.76s) === RUN Test_RequireNot_Prerelease_EqualCoreVersion require_not_test.go:73: expected t.Fatal(), got none --- FAIL: Test_RequireNot_Prerelease_EqualCoreVersion (3.06s) === RUN Test_RequireNot_Prerelease_HigherCoreVersion --- PASS: Test_RequireNot_Prerelease_HigherCoreVersion (3.19s) === RUN Test_RequireNot_Prerelease_HigherPrerelease --- PASS: Test_RequireNot_Prerelease_HigherPrerelease (3.25s) === RUN Test_RequireNot_Prerelease_LowerCoreVersion --- PASS: Test_RequireNot_Prerelease_LowerCoreVersion (5.37s) === RUN Test_RequireNot_Prerelease_LowerPrerelease --- PASS: Test_RequireNot_Prerelease_LowerPrerelease (3.13s) === RUN Test_SkipAbove_SkipTest skip_above_test.go:22: Terraform CLI version 1.3.0 is above maximum version 1.2.9: skipping test --- SKIP: Test_SkipAbove_SkipTest (2.34s) === RUN Test_SkipAbove_RunTest --- PASS: Test_SkipAbove_RunTest (2.51s) === RUN Test_SkipAbove_Prerelease_EqualCoreVersion --- PASS: Test_SkipAbove_Prerelease_EqualCoreVersion (2.96s) === RUN Test_SkipAbove_Prerelease_HigherCoreVersion --- PASS: Test_SkipAbove_Prerelease_HigherCoreVersion (3.22s) === RUN Test_SkipAbove_Prerelease_HigherPrerelease --- PASS: Test_SkipAbove_Prerelease_HigherPrerelease (3.29s) === RUN Test_SkipAbove_Prerelease_LowerCoreVersion skip_above_test.go:145: Terraform CLI version 1.8.0-rc1 is above maximum version 1.7.0: skipping test --- SKIP: Test_SkipAbove_Prerelease_LowerCoreVersion (2.68s) === RUN Test_SkipAbove_Prerelease_LowerPrerelease skip_above_test.go:166: Terraform CLI version 1.8.0-rc1 is above maximum version 1.8.0-beta1: skipping test --- SKIP: Test_SkipAbove_Prerelease_LowerPrerelease (2.62s) === RUN Test_SkipBetween_SkipTest skip_between_test.go:24: Terraform CLI version 1.2.0 is between 1.2.0 and 1.3.0: skipping test. --- SKIP: Test_SkipBetween_SkipTest (2.70s) === RUN Test_SkipBetween_RunTest_AboveMax --- PASS: Test_SkipBetween_RunTest_AboveMax (2.71s) === RUN Test_SkipBetween_RunTest_EqToMin --- PASS: Test_SkipBetween_RunTest_EqToMin (2.76s) === RUN Test_SkipBetween_Prerelease_MaxEqualCoreVersion skip_between_test.go:106: Terraform CLI version 1.8.0-rc1 is between 1.7.0 and 1.8.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxEqualCoreVersion (3.04s) === RUN Test_SkipBetween_Prerelease_MinEqualCoreVersion --- PASS: Test_SkipBetween_Prerelease_MinEqualCoreVersion (3.36s) === RUN Test_SkipBetween_Prerelease_MaxHigherCoreVersion skip_between_test.go:154: Terraform CLI version 1.7.0-rc1 is between 1.6.0 and 1.8.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxHigherCoreVersion (2.82s) === RUN Test_SkipBetween_Prerelease_MinHigherCoreVersion --- PASS: Test_SkipBetween_Prerelease_MinHigherCoreVersion (3.81s) === RUN Test_SkipBetween_Prerelease_MaxHigherPrerelease skip_between_test.go:198: Terraform CLI version 1.7.0-rc1 is between 1.6.0 and 1.7.0-rc2: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxHigherPrerelease (3.37s) === RUN Test_SkipBetween_Prerelease_MinHigherPrerelease --- PASS: Test_SkipBetween_Prerelease_MinHigherPrerelease (4.33s) === RUN Test_SkipBetween_Prerelease_MaxLowerCoreVersion --- PASS: Test_SkipBetween_Prerelease_MaxLowerCoreVersion (3.33s) === RUN Test_SkipBetween_Prerelease_MinLowerCoreVersion skip_between_test.go:261: Terraform CLI version 1.8.0-rc1 is between 1.7.0 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinLowerCoreVersion (3.12s) === RUN Test_SkipBetween_Prerelease_MaxLowerPrerelease --- PASS: Test_SkipBetween_Prerelease_MaxLowerPrerelease (3.08s) === RUN Test_SkipBetween_Prerelease_MinLowerPrerelease skip_between_test.go:303: Terraform CLI version 1.8.0-rc1 is between 1.8.0-beta1 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinLowerPrerelease (2.86s) === RUN Test_SkipIf_SkipTest skip_if_test.go:24: Terraform CLI version is 1.4.3: skipping test. --- SKIP: Test_SkipIf_SkipTest (2.52s) === RUN Test_SkipIf_RunTest --- PASS: Test_SkipIf_RunTest (3.15s) === RUN Test_SkipIf_Prerelease_EqualCoreVersion --- PASS: Test_SkipIf_Prerelease_EqualCoreVersion (4.58s) === RUN Test_SkipIf_Prerelease_HigherCoreVersion --- PASS: Test_SkipIf_Prerelease_HigherCoreVersion (3.46s) === RUN Test_SkipIf_Prerelease_HigherPrerelease --- PASS: Test_SkipIf_Prerelease_HigherPrerelease (3.28s) === RUN Test_SkipIf_Prerelease_LowerCoreVersion --- PASS: Test_SkipIf_Prerelease_LowerCoreVersion (3.10s) === RUN Test_SkipIf_Prerelease_LowerPrerelease --- PASS: Test_SkipIf_Prerelease_LowerPrerelease (3.06s) ``` Now (only showing skip testing as require testing will fail if incorrect): ``` === RUN Test_SkipAbove_SkipTest skip_above_test.go:22: Terraform CLI version 1.3.0 is above maximum version 1.2.9: skipping test --- SKIP: Test_SkipAbove_SkipTest (2.87s) === RUN Test_SkipAbove_RunTest --- PASS: Test_SkipAbove_RunTest (2.66s) === RUN Test_SkipAbove_Prerelease_EqualCoreVersion --- PASS: Test_SkipAbove_Prerelease_EqualCoreVersion (2.98s) === RUN Test_SkipAbove_Prerelease_HigherCoreVersion --- PASS: Test_SkipAbove_Prerelease_HigherCoreVersion (3.19s) === RUN Test_SkipAbove_Prerelease_HigherPrerelease --- PASS: Test_SkipAbove_Prerelease_HigherPrerelease (3.29s) === RUN Test_SkipAbove_Prerelease_LowerCoreVersion skip_above_test.go:145: Terraform CLI version 1.8.0-rc1 is above maximum version 1.7.0: skipping test --- SKIP: Test_SkipAbove_Prerelease_LowerCoreVersion (3.32s) === RUN Test_SkipAbove_Prerelease_LowerPrerelease skip_above_test.go:166: Terraform CLI version 1.8.0-rc1 is above maximum version 1.8.0-beta1: skipping test --- SKIP: Test_SkipAbove_Prerelease_LowerPrerelease (2.72s) === RUN Test_SkipBetween_SkipTest skip_between_test.go:24: Terraform CLI version 1.2.0 is between 1.2.0 and 1.3.0: skipping test. --- SKIP: Test_SkipBetween_SkipTest (2.97s) === RUN Test_SkipBetween_RunTest_AboveMax --- PASS: Test_SkipBetween_RunTest_AboveMax (2.69s) === RUN Test_SkipBetween_RunTest_EqToMin --- PASS: Test_SkipBetween_RunTest_EqToMin (2.47s) === RUN Test_SkipBetween_Prerelease_MaxEqualCoreVersion --- PASS: Test_SkipBetween_Prerelease_MaxEqualCoreVersion (3.65s) === RUN Test_SkipBetween_Prerelease_MinEqualCoreVersion skip_between_test.go:131: Terraform CLI version 1.8.0-rc1 is between 1.8.0 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinEqualCoreVersion (3.10s) === RUN Test_SkipBetween_Prerelease_MaxHigherCoreVersion skip_between_test.go:154: Terraform CLI version 1.7.0-rc1 is between 1.6.0 and 1.8.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxHigherCoreVersion (2.86s) === RUN Test_SkipBetween_Prerelease_MinHigherCoreVersion --- PASS: Test_SkipBetween_Prerelease_MinHigherCoreVersion (3.46s) === RUN Test_SkipBetween_Prerelease_MaxHigherPrerelease skip_between_test.go:198: Terraform CLI version 1.7.0-rc1 is between 1.6.0 and 1.7.0-rc2: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxHigherPrerelease (3.04s) === RUN Test_SkipBetween_Prerelease_MinHigherPrerelease --- PASS: Test_SkipBetween_Prerelease_MinHigherPrerelease (3.39s) === RUN Test_SkipBetween_Prerelease_MaxLowerCoreVersion --- PASS: Test_SkipBetween_Prerelease_MaxLowerCoreVersion (3.01s) === RUN Test_SkipBetween_Prerelease_MinLowerCoreVersion skip_between_test.go:261: Terraform CLI version 1.8.0-rc1 is between 1.7.0 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinLowerCoreVersion (2.66s) === RUN Test_SkipBetween_Prerelease_MaxLowerPrerelease --- PASS: Test_SkipBetween_Prerelease_MaxLowerPrerelease (3.24s) === RUN Test_SkipBetween_Prerelease_MinLowerPrerelease skip_between_test.go:303: Terraform CLI version 1.8.0-rc1 is between 1.8.0-beta1 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinLowerPrerelease (2.91s) === RUN Test_SkipIf_SkipTest skip_if_test.go:24: Terraform CLI version is 1.4.3: skipping test. --- SKIP: Test_SkipIf_SkipTest (2.51s) === RUN Test_SkipIf_RunTest --- PASS: Test_SkipIf_RunTest (3.12s) === RUN Test_SkipIf_Prerelease_EqualCoreVersion skip_if_test.go:70: Terraform CLI version is 1.8.0: skipping test. --- SKIP: Test_SkipIf_Prerelease_EqualCoreVersion (2.51s) === RUN Test_SkipIf_Prerelease_HigherCoreVersion --- PASS: Test_SkipIf_Prerelease_HigherCoreVersion (3.22s) === RUN Test_SkipIf_Prerelease_HigherPrerelease --- PASS: Test_SkipIf_Prerelease_HigherPrerelease (3.27s) === RUN Test_SkipIf_Prerelease_LowerCoreVersion --- PASS: Test_SkipIf_Prerelease_LowerCoreVersion (3.03s) === RUN Test_SkipIf_Prerelease_LowerPrerelease --- PASS: Test_SkipIf_Prerelease_LowerPrerelease (3.15s) ```
bflad
added a commit
that referenced
this pull request
Mar 29, 2024
Reference: #303 Reference: #308 Reference: #316 These changes fully update the rest of the `tfversion` built-in checks for treating prereleases as semantically equal to the patch version. It also includes a fix for `RequireBelow` to handle the exactly equal condition. Previously, `RequireBelow` when given 1.8.0 would still run on 1.8.0 (and prereleases variants). Prior to logic changes: ``` === RUN Test_RequireAbove --- PASS: Test_RequireAbove (3.01s) === RUN Test_RequireAbove_Error --- PASS: Test_RequireAbove_Error (3.24s) === RUN Test_RequireAbove_Prerelease_EqualCoreVersion require_above_test.go:77: expected Terraform CLI version above 1.8.0 but detected version is 1.8.0-rc1 --- FAIL: Test_RequireAbove_Prerelease_EqualCoreVersion (3.97s) === RUN Test_RequireAbove_Prerelease_HigherCoreVersion --- PASS: Test_RequireAbove_Prerelease_HigherCoreVersion (3.17s) === RUN Test_RequireAbove_Prerelease_HigherPrerelease --- PASS: Test_RequireAbove_Prerelease_HigherPrerelease (3.03s) === RUN Test_RequireAbove_Prerelease_LowerCoreVersion --- PASS: Test_RequireAbove_Prerelease_LowerCoreVersion (3.13s) === RUN Test_RequireAbove_Prerelease_LowerPrerelease --- PASS: Test_RequireAbove_Prerelease_LowerPrerelease (3.14s) === RUN Test_RequireBelow --- PASS: Test_RequireBelow (2.94s) === RUN Test_RequireBelow_Error --- PASS: Test_RequireBelow_Error (2.59s) === RUN Test_RequireBelow_Prerelease_EqualCoreVersion require_below_test.go:80: expected t.Fatal(), got none --- FAIL: Test_RequireBelow_Prerelease_EqualCoreVersion (3.29s) === RUN Test_RequireBelow_Prerelease_HigherCoreVersion --- PASS: Test_RequireBelow_Prerelease_HigherCoreVersion (3.04s) === RUN Test_RequireBelow_Prerelease_HigherPrerelease --- PASS: Test_RequireBelow_Prerelease_HigherPrerelease (3.22s) === RUN Test_RequireBelow_Prerelease_LowerCoreVersion --- PASS: Test_RequireBelow_Prerelease_LowerCoreVersion (2.74s) === RUN Test_RequireBelow_Prerelease_LowerPrerelease --- PASS: Test_RequireBelow_Prerelease_LowerPrerelease (2.80s) === RUN Test_RequireBetween --- PASS: Test_RequireBetween (2.69s) === RUN Test_RequireBetween_Error_BelowMin --- PASS: Test_RequireBetween_Error_BelowMin (2.57s) === RUN Test_RequireBetween_Error_EqToMax --- PASS: Test_RequireBetween_Error_EqToMax (2.29s) === RUN Test_RequireBetween_Prerelease_MaxEqualCoreVersion require_between_test.go:114: expected t.Fatal(), got none --- FAIL: Test_RequireBetween_Prerelease_MaxEqualCoreVersion (3.24s) === RUN Test_RequireBetween_Prerelease_MinEqualCoreVersion require_between_test.go:141: expected Terraform CLI version between 1.8.0 and 1.9.0 but detected version is 1.8.0-rc1 --- FAIL: Test_RequireBetween_Prerelease_MinEqualCoreVersion (2.83s) === RUN Test_RequireBetween_Prerelease_MaxHigherCoreVersion --- PASS: Test_RequireBetween_Prerelease_MaxHigherCoreVersion (3.17s) === RUN Test_RequireBetween_Prerelease_MinHigherCoreVersion --- PASS: Test_RequireBetween_Prerelease_MinHigherCoreVersion (3.09s) === RUN Test_RequireBetween_Prerelease_MaxHigherPrerelease --- PASS: Test_RequireBetween_Prerelease_MaxHigherPrerelease (3.84s) === RUN Test_RequireBetween_Prerelease_MinHigherPrerelease --- PASS: Test_RequireBetween_Prerelease_MinHigherPrerelease (3.02s) === RUN Test_RequireBetween_Prerelease_MaxLowerCoreVersion --- PASS: Test_RequireBetween_Prerelease_MaxLowerCoreVersion (2.84s) === RUN Test_RequireBetween_Prerelease_MinLowerCoreVersion --- PASS: Test_RequireBetween_Prerelease_MinLowerCoreVersion (4.07s) === RUN Test_RequireBetween_Prerelease_MaxLowerPrerelease --- PASS: Test_RequireBetween_Prerelease_MaxLowerPrerelease (2.79s) === RUN Test_RequireBetween_Prerelease_MinLowerPrerelease --- PASS: Test_RequireBetween_Prerelease_MinLowerPrerelease (3.25s) === RUN Test_RequireNot --- PASS: Test_RequireNot (3.70s) === RUN Test_RequireNot_Error --- PASS: Test_RequireNot_Error (2.76s) === RUN Test_RequireNot_Prerelease_EqualCoreVersion require_not_test.go:73: expected t.Fatal(), got none --- FAIL: Test_RequireNot_Prerelease_EqualCoreVersion (3.06s) === RUN Test_RequireNot_Prerelease_HigherCoreVersion --- PASS: Test_RequireNot_Prerelease_HigherCoreVersion (3.19s) === RUN Test_RequireNot_Prerelease_HigherPrerelease --- PASS: Test_RequireNot_Prerelease_HigherPrerelease (3.25s) === RUN Test_RequireNot_Prerelease_LowerCoreVersion --- PASS: Test_RequireNot_Prerelease_LowerCoreVersion (5.37s) === RUN Test_RequireNot_Prerelease_LowerPrerelease --- PASS: Test_RequireNot_Prerelease_LowerPrerelease (3.13s) === RUN Test_SkipAbove_SkipTest skip_above_test.go:22: Terraform CLI version 1.3.0 is above maximum version 1.2.9: skipping test --- SKIP: Test_SkipAbove_SkipTest (2.34s) === RUN Test_SkipAbove_RunTest --- PASS: Test_SkipAbove_RunTest (2.51s) === RUN Test_SkipAbove_Prerelease_EqualCoreVersion --- PASS: Test_SkipAbove_Prerelease_EqualCoreVersion (2.96s) === RUN Test_SkipAbove_Prerelease_HigherCoreVersion --- PASS: Test_SkipAbove_Prerelease_HigherCoreVersion (3.22s) === RUN Test_SkipAbove_Prerelease_HigherPrerelease --- PASS: Test_SkipAbove_Prerelease_HigherPrerelease (3.29s) === RUN Test_SkipAbove_Prerelease_LowerCoreVersion skip_above_test.go:145: Terraform CLI version 1.8.0-rc1 is above maximum version 1.7.0: skipping test --- SKIP: Test_SkipAbove_Prerelease_LowerCoreVersion (2.68s) === RUN Test_SkipAbove_Prerelease_LowerPrerelease skip_above_test.go:166: Terraform CLI version 1.8.0-rc1 is above maximum version 1.8.0-beta1: skipping test --- SKIP: Test_SkipAbove_Prerelease_LowerPrerelease (2.62s) === RUN Test_SkipBetween_SkipTest skip_between_test.go:24: Terraform CLI version 1.2.0 is between 1.2.0 and 1.3.0: skipping test. --- SKIP: Test_SkipBetween_SkipTest (2.70s) === RUN Test_SkipBetween_RunTest_AboveMax --- PASS: Test_SkipBetween_RunTest_AboveMax (2.71s) === RUN Test_SkipBetween_RunTest_EqToMin --- PASS: Test_SkipBetween_RunTest_EqToMin (2.76s) === RUN Test_SkipBetween_Prerelease_MaxEqualCoreVersion skip_between_test.go:106: Terraform CLI version 1.8.0-rc1 is between 1.7.0 and 1.8.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxEqualCoreVersion (3.04s) === RUN Test_SkipBetween_Prerelease_MinEqualCoreVersion --- PASS: Test_SkipBetween_Prerelease_MinEqualCoreVersion (3.36s) === RUN Test_SkipBetween_Prerelease_MaxHigherCoreVersion skip_between_test.go:154: Terraform CLI version 1.7.0-rc1 is between 1.6.0 and 1.8.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxHigherCoreVersion (2.82s) === RUN Test_SkipBetween_Prerelease_MinHigherCoreVersion --- PASS: Test_SkipBetween_Prerelease_MinHigherCoreVersion (3.81s) === RUN Test_SkipBetween_Prerelease_MaxHigherPrerelease skip_between_test.go:198: Terraform CLI version 1.7.0-rc1 is between 1.6.0 and 1.7.0-rc2: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxHigherPrerelease (3.37s) === RUN Test_SkipBetween_Prerelease_MinHigherPrerelease --- PASS: Test_SkipBetween_Prerelease_MinHigherPrerelease (4.33s) === RUN Test_SkipBetween_Prerelease_MaxLowerCoreVersion --- PASS: Test_SkipBetween_Prerelease_MaxLowerCoreVersion (3.33s) === RUN Test_SkipBetween_Prerelease_MinLowerCoreVersion skip_between_test.go:261: Terraform CLI version 1.8.0-rc1 is between 1.7.0 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinLowerCoreVersion (3.12s) === RUN Test_SkipBetween_Prerelease_MaxLowerPrerelease --- PASS: Test_SkipBetween_Prerelease_MaxLowerPrerelease (3.08s) === RUN Test_SkipBetween_Prerelease_MinLowerPrerelease skip_between_test.go:303: Terraform CLI version 1.8.0-rc1 is between 1.8.0-beta1 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinLowerPrerelease (2.86s) === RUN Test_SkipIf_SkipTest skip_if_test.go:24: Terraform CLI version is 1.4.3: skipping test. --- SKIP: Test_SkipIf_SkipTest (2.52s) === RUN Test_SkipIf_RunTest --- PASS: Test_SkipIf_RunTest (3.15s) === RUN Test_SkipIf_Prerelease_EqualCoreVersion --- PASS: Test_SkipIf_Prerelease_EqualCoreVersion (4.58s) === RUN Test_SkipIf_Prerelease_HigherCoreVersion --- PASS: Test_SkipIf_Prerelease_HigherCoreVersion (3.46s) === RUN Test_SkipIf_Prerelease_HigherPrerelease --- PASS: Test_SkipIf_Prerelease_HigherPrerelease (3.28s) === RUN Test_SkipIf_Prerelease_LowerCoreVersion --- PASS: Test_SkipIf_Prerelease_LowerCoreVersion (3.10s) === RUN Test_SkipIf_Prerelease_LowerPrerelease --- PASS: Test_SkipIf_Prerelease_LowerPrerelease (3.06s) ``` Now (only showing skip testing as require testing will fail if incorrect): ``` === RUN Test_SkipAbove_SkipTest skip_above_test.go:22: Terraform CLI version 1.3.0 is above maximum version 1.2.9: skipping test --- SKIP: Test_SkipAbove_SkipTest (2.87s) === RUN Test_SkipAbove_RunTest --- PASS: Test_SkipAbove_RunTest (2.66s) === RUN Test_SkipAbove_Prerelease_EqualCoreVersion --- PASS: Test_SkipAbove_Prerelease_EqualCoreVersion (2.98s) === RUN Test_SkipAbove_Prerelease_HigherCoreVersion --- PASS: Test_SkipAbove_Prerelease_HigherCoreVersion (3.19s) === RUN Test_SkipAbove_Prerelease_HigherPrerelease --- PASS: Test_SkipAbove_Prerelease_HigherPrerelease (3.29s) === RUN Test_SkipAbove_Prerelease_LowerCoreVersion skip_above_test.go:145: Terraform CLI version 1.8.0-rc1 is above maximum version 1.7.0: skipping test --- SKIP: Test_SkipAbove_Prerelease_LowerCoreVersion (3.32s) === RUN Test_SkipAbove_Prerelease_LowerPrerelease skip_above_test.go:166: Terraform CLI version 1.8.0-rc1 is above maximum version 1.8.0-beta1: skipping test --- SKIP: Test_SkipAbove_Prerelease_LowerPrerelease (2.72s) === RUN Test_SkipBetween_SkipTest skip_between_test.go:24: Terraform CLI version 1.2.0 is between 1.2.0 and 1.3.0: skipping test. --- SKIP: Test_SkipBetween_SkipTest (2.97s) === RUN Test_SkipBetween_RunTest_AboveMax --- PASS: Test_SkipBetween_RunTest_AboveMax (2.69s) === RUN Test_SkipBetween_RunTest_EqToMin --- PASS: Test_SkipBetween_RunTest_EqToMin (2.47s) === RUN Test_SkipBetween_Prerelease_MaxEqualCoreVersion --- PASS: Test_SkipBetween_Prerelease_MaxEqualCoreVersion (3.65s) === RUN Test_SkipBetween_Prerelease_MinEqualCoreVersion skip_between_test.go:131: Terraform CLI version 1.8.0-rc1 is between 1.8.0 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinEqualCoreVersion (3.10s) === RUN Test_SkipBetween_Prerelease_MaxHigherCoreVersion skip_between_test.go:154: Terraform CLI version 1.7.0-rc1 is between 1.6.0 and 1.8.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxHigherCoreVersion (2.86s) === RUN Test_SkipBetween_Prerelease_MinHigherCoreVersion --- PASS: Test_SkipBetween_Prerelease_MinHigherCoreVersion (3.46s) === RUN Test_SkipBetween_Prerelease_MaxHigherPrerelease skip_between_test.go:198: Terraform CLI version 1.7.0-rc1 is between 1.6.0 and 1.7.0-rc2: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MaxHigherPrerelease (3.04s) === RUN Test_SkipBetween_Prerelease_MinHigherPrerelease --- PASS: Test_SkipBetween_Prerelease_MinHigherPrerelease (3.39s) === RUN Test_SkipBetween_Prerelease_MaxLowerCoreVersion --- PASS: Test_SkipBetween_Prerelease_MaxLowerCoreVersion (3.01s) === RUN Test_SkipBetween_Prerelease_MinLowerCoreVersion skip_between_test.go:261: Terraform CLI version 1.8.0-rc1 is between 1.7.0 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinLowerCoreVersion (2.66s) === RUN Test_SkipBetween_Prerelease_MaxLowerPrerelease --- PASS: Test_SkipBetween_Prerelease_MaxLowerPrerelease (3.24s) === RUN Test_SkipBetween_Prerelease_MinLowerPrerelease skip_between_test.go:303: Terraform CLI version 1.8.0-rc1 is between 1.8.0-beta1 and 1.9.0: skipping test. --- SKIP: Test_SkipBetween_Prerelease_MinLowerPrerelease (2.91s) === RUN Test_SkipIf_SkipTest skip_if_test.go:24: Terraform CLI version is 1.4.3: skipping test. --- SKIP: Test_SkipIf_SkipTest (2.51s) === RUN Test_SkipIf_RunTest --- PASS: Test_SkipIf_RunTest (3.12s) === RUN Test_SkipIf_Prerelease_EqualCoreVersion skip_if_test.go:70: Terraform CLI version is 1.8.0: skipping test. --- SKIP: Test_SkipIf_Prerelease_EqualCoreVersion (2.51s) === RUN Test_SkipIf_Prerelease_HigherCoreVersion --- PASS: Test_SkipIf_Prerelease_HigherCoreVersion (3.22s) === RUN Test_SkipIf_Prerelease_HigherPrerelease --- PASS: Test_SkipIf_Prerelease_HigherPrerelease (3.27s) === RUN Test_SkipIf_Prerelease_LowerCoreVersion --- PASS: Test_SkipIf_Prerelease_LowerCoreVersion (3.03s) === RUN Test_SkipIf_Prerelease_LowerPrerelease --- PASS: Test_SkipIf_Prerelease_LowerPrerelease (3.15s) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #254
Reference: #303
Note: This change is easiest to view with whitespace changes ignored.
As part of upcoming efforts to consider prerelease versioning more pragmatically in the
tfversionpackage checks, the need to properly unit test(*testing.T).Skip()and friends behaviors is more apparent. Similar to issues with testing(*testing.T).Fatal()behaviors, the Go standard library in general is not the most friendly to capture test helper behaviors since thetesting.Timplementation is not intended to be extensible. This change was initially intended to extend our current usage ofgithub.com/mitchellh/go-testing-interfaceas a stopgap, however while attempting to implement skip assertion testing, theRuntimeTimplementation did not callruntime.Goexit()and would cause tests to continue to execute test logic after skipping even though in reality they would not. While the tests could be fixed to ensure they still passed due to that behavior change, it felt like a good time as any to replace the now-unsuitable and archivedgithub.com/mitchellh/go-testing-interfacewith our own internal implementation.The new
internal/testingiface.Tinterface is a wholesale duplicate interface to the priorgithub.com/mitchellh/go-testing-interface.Tinterface. Swapping this interface in exported functions, such ashelper/resource.Test()is only a compatibility issue for consumers if by chance they were implementing a wrapping function that also depended ongithub.com/mitchellh/go-testing-interface.Tand passed that implementation detail directly to function calls of this Go module. This is a gray compatibility area as the intention of usinggithub.com/mitchellh/go-testing-interface.Twas for consumers to use real*testing.Timplementations, which will work the same after this change, but it was not appropriately documented that extensions of exported functionality usinggithub.com/mitchellh/go-testing-interface.Tshould not rely on that explicitly. Changing this detail is therefore a pragmatic compromise that it is more important to prioritize and document the intended usage while removing the archived dependency, which is almost exclusively how the ecosystem interacts with the functionality, over the potential of compatibility issues in a very small subset of the ecosystem.Since
(*testing.T).Skip()is considered a passing test, there are now assertions whether skipping was called or not, as expected, in the existingtfversionpackage tests. The newinternal/testingifacepackage also includes similar assertion helpers for(*testing.T).Fatal()and(*testing.T).Parallel()for consistency and cleans up some existing logic.