Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky integration test #17

Closed
calebschoepp opened this issue Feb 2, 2024 · 7 comments
Closed

Flaky integration test #17

calebschoepp opened this issue Feb 2, 2024 · 7 comments
Labels
bug Something isn't working

Comments

@calebschoepp
Copy link
Contributor

calebschoepp commented Feb 2, 2024 •
edited
Loading

I'm experiencing the integration tests flaking occasionally because of a race condition. I've been unable to diagnose the issue so far but I think it has something to do with this code.

test -s /Users/caleb/fermyon/spin-operator/bin/controller-gen && /Users/caleb/fermyon/spin-operator/bin/controller-gen --version | grep -q v0.13.0 || \
	GOBIN=/Users/caleb/fermyon/spin-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.13.0
/Users/caleb/fermyon/spin-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./api/..." paths="./cmd/..." paths="./internal/..." paths="./pkg/..." output:crd:artifacts:config=config/crd/bases
/Users/caleb/fermyon/spin-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./api/..." paths="./cmd/..." paths="./internal/..." paths="./pkg/..."
go fmt ./...
go vet ./...
test -s /Users/caleb/fermyon/spin-operator/bin/setup-envtest || GOBIN=/Users/caleb/fermyon/spin-operator/bin go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
/Users/caleb/fermyon/spin-operator/bin/setup-envtest use 1.28.3 --bin-dir /Users/caleb/fermyon/spin-operator/bin
Version: 1.28.3
OS/Arch: darwin/arm64
Path: /Users/caleb/fermyon/spin-operator/bin/k8s/1.28.3-darwin-arm64
KUBEBUILDER_ASSETS="/Users/caleb/fermyon/spin-operator/bin/k8s/1.28.3-darwin-arm64" go test ./... -coverprofile cover.out
?   	github.com/fermyon/spin-operator/api/v1	[no test files]
?   	github.com/fermyon/spin-operator/cmd	[no test files]
?   	github.com/fermyon/spin-operator/internal/constants	[no test files]
?   	github.com/fermyon/spin-operator/internal/logging	[no test files]
?   	github.com/fermyon/spin-operator/pkg/spinapp	[no test files]
ok  	github.com/fermyon/spin-operator/internal/controller	7.827s	coverage: 34.3% of statements
fatal error: concurrent map iteration and map write

goroutine 43 [running]:
sigs.k8s.io/controller-runtime/pkg/webhook/conversion.objectGVKs(0x14000331570, {0x1039c4e10?, 0x14000b82310})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/conversion/conversion.go:291 +0x130
sigs.k8s.io/controller-runtime/pkg/webhook/conversion.IsConvertible(0x14000331570?, {0x1039c4e10?, 0x14000b82310})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/conversion/conversion.go:227 +0x34
sigs.k8s.io/controller-runtime/pkg/envtest.modifyConversionWebhooks({0x14000818960, 0x3, 0x10?}, 0x14000331570, {{0x1400028aa80, 0x1, 0x1}, {0x1400032a6b8, 0x1, 0x1}, ...})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/crd.go:357 +0x110
sigs.k8s.io/controller-runtime/pkg/envtest.InstallCRDs(_, {0x14000331570, {0x14000111360, 0x1, 0x1}, {0x14000818960, 0x3, 0x3}, 0x1, 0x2540be400, ...})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/crd.go:101 +0xf4
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start(0x14000180500)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/server.go:282 +0x980
github.com/fermyon/spin-operator/internal/webhook.setupEnvTest(0x1400031a9c0)
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:55 +0x308
github.com/fermyon/spin-operator/internal/webhook.TestCreateSpinAppWithNoExecutor(0x14000320401?)
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:131 +0x28
testing.tRunner(0x1400031a9c0, 0x1039b5d60)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x33c

goroutine 1 [chan receive]:
testing.tRunner.func1()
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1561 +0x434
testing.tRunner(0x1400031a820, 0x14000193c28)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1601 +0x124
testing.runTests(0x1400026c280?, {0x1045dbf20, 0x8, 0x8}, {0x40?, 0x10387f5c0?, 0x1045f5b20?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:2052 +0x3b4
testing.(*M).Run(0x1400026c280)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1925 +0x538
main.main()
	_testmain.go:95 +0x1c8

goroutine 44 [select]:
github.com/stretchr/testify/assert.Eventually({0x10bb09318, 0x1400031ad00}, 0x14000202be0, 0x14000026000?, 0x14000202be0?, {0x0, 0x0, 0x0})
	/Users/caleb/go/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1847 +0x14c
github.com/stretchr/testify/require.Eventually({0x1039c1bb8, 0x1400031ad00}, 0x1400057bb98?, 0x2?, 0x2?, {0x0, 0x0, 0x0})
	/Users/caleb/go/pkg/mod/github.com/stretchr/testify@v1.8.4/require/require.go:401 +0x94
github.com/fermyon/spin-operator/internal/webhook.startWebhookServer(0x1400031ad00, 0x140000afcc0)
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:117 +0x428
github.com/fermyon/spin-operator/internal/webhook.TestCreateSpinAppWithSingleExecutor(0x14000320401?)
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:151 +0x3c
testing.tRunner(0x1400031ad00, 0x1039b5d68)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x33c

goroutine 45 [select]:
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start(0x140001785a0, {0x0?, 0x0}, {0x0?, 0x0})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:183 +0x33c
sigs.k8s.io/controller-runtime/pkg/internal/testing/controlplane.(*APIServer).Start(0x1400038a000)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/controlplane/apiserver.go:165 +0x4c
sigs.k8s.io/controller-runtime/pkg/internal/testing/controlplane.(*ControlPlane).Start(0x1400022c000)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/controlplane/plane.go:67 +0x134
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane(0x1400022c000)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/server.go:313 +0x94
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start(0x1400022c000)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/server.go:244 +0x380
github.com/fermyon/spin-operator/internal/webhook.setupEnvTest(0x1400031b040)
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:55 +0x308
github.com/fermyon/spin-operator/internal/webhook.TestCreateSpinAppWithMultipleExecutors(0x14000320401?)
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:186 +0x2c
testing.tRunner(0x1400031b040, 0x1039b5d58)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x33c

goroutine 46 [runnable]:
k8s.io/apimachinery/pkg/runtime/serializer.CodecFactory.UniversalDecoder({0x14000331570, {0x1039bde60, 0x14000303068}, {0x140003428c0, 0x3, 0x4}, {0x1039cc170, 0x14000323f40}}, {0x0?, 0x102f3a578?, ...})
	/Users/caleb/go/pkg/mod/k8s.io/apimachinery@v0.29.1/pkg/runtime/serializer/codec_factory.go:289 +0xf0
k8s.io/client-go/discovery.setDiscoveryDefaults(0x1400094f860)
	/Users/caleb/go/pkg/mod/k8s.io/client-go@v0.29.1/discovery/discovery_client.go:710 +0xc4
k8s.io/client-go/discovery.NewDiscoveryClientForConfigAndClient(0x14000c1a001?, 0x1038e3420?)
	/Users/caleb/go/pkg/mod/k8s.io/client-go@v0.29.1/discovery/discovery_client.go:739 +0x54
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0x140003ae900?, 0x14000c12180?)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/client/apiutil/restmapper.go:40 +0x20
sigs.k8s.io/controller-runtime/pkg/client.newClient(0x1400031b300?, {0x14000c04900, 0x14000331570, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/client/client.go:159 +0x1c0
sigs.k8s.io/controller-runtime/pkg/client.New(0x1039c1bb8?, {0x0, 0x14000331570, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/client/client.go:110 +0x54
github.com/fermyon/spin-operator/internal/webhook.setupEnvTest(0x1400031b380)
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:69 +0x460
github.com/fermyon/spin-operator/internal/webhook.TestCreateInvalidSpinApp(0x14000320401?)
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:231 +0x28
testing.tRunner(0x1400031b380, 0x1039b5d50)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x33c

goroutine 72 [syscall]:
syscall.syscall6(0x104bb0a68?, 0x14000003617?, 0x1400008ee28?, 0x10230c5c8?, 0x1400008ee28?, 0x9000010101c654?, 0x10b9ead50?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x1400008ee88?, 0x1023e9768?, 0x90?, 0x10394f260?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x18?, 0x1400008eec4, 0x3?, 0x8?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x140005ac1b0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x1400017c9a0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 45
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 164 [sleep]:
time.Sleep(0x5f5e100)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/time.go:195 +0x10c
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.pollURLUntilOK({{0x1033cb2ab, 0x5}, {0x0, 0x0}, 0x0, {0x14000336fe0, 0xf}, {0x1033cf6d5, 0x8}, {0x0, ...}, ...}, ...)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:239 +0xd8
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 45
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:163 +0x270

goroutine 15 [syscall]:
syscall.syscall6(0x0?, 0x0?, 0x0?, 0x0?, 0x0?, 0x100010000?, 0x104f551b0?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x140002fa688?, 0x1023e9768?, 0x90?, 0x10394f260?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x0?, 0x140002fa6c4, 0x0?, 0x0?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x140004e0000)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x14000344420)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 46
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 16 [syscall]:
syscall.syscall6(0x104bb0108?, 0x17?, 0x0?, 0x0?, 0x0?, 0x90000101010000?, 0x104f551b0?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x140002f9688?, 0x1023e9768?, 0x90?, 0x10394f260?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x0?, 0x140002f96c4, 0x0?, 0x0?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x140003460f0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x140006a2000)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 44
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 114 [syscall]:
syscall.syscall6(0x104bb05b8?, 0x17?, 0x1039e4e28?, 0x103818ea0?, 0x1039e4e28?, 0x90000101014620?, 0x104f499a0?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x140002fd688?, 0x1023e9768?, 0x90?, 0x10394f260?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x1033eb16f?, 0x140002fd6c4, 0x1033d18f5?, 0xa?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x1400004a0f0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x1400020e2c0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 43
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 324 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x140003210e0)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 302
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 119 [syscall]:
syscall.syscall6(0x104bb05b8?, 0x1039c8317?, 0x1039bdb00?, 0x1039cb900?, 0x0?, 0x90000101010000?, 0x10b9ab290?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x14000020e88?, 0x1023e9768?, 0x90?, 0x10394f260?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x1039c4578?, 0x14000020ec4, 0x0?, 0x1039c4b90?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x140004962d0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x140005a4160)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 43
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 165 [syscall]:
syscall.syscall6(0x1400045ae00?, 0x10230c824?, 0x1038189a0?, 0x10230cb68?, 0xffffffffffffffff?, 0xffff00010001ffff?, 0x104f499a0?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x1400045ae88?, 0x1023e9768?, 0x90?, 0x10394f260?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x1400045aec0?, 0x1400045aec4, 0x1400045afa0?, 0x2?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x1400004a900)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x1400020eb00)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 45
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 141 [syscall]:
syscall.syscall6(0x14000088e00?, 0x10230c824?, 0x1038189a0?, 0x10230cb68?, 0xffffffffffffffff?, 0xffff00010001ffff?, 0x10b9ab290?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x14000088e88?, 0x1023e9768?, 0x90?, 0x10394f260?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x14000088ec0?, 0x14000088ec4, 0x14000088fa0?, 0x2?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x14000496930)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x140005a4840)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 46
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 134 [syscall]:
syscall.syscall6(0x104bb13c8?, 0x17?, 0x1038189a0?, 0x0?, 0x5ac?, 0x90000101017530?, 0x10b9ab290?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x140004c5688?, 0x1023e9768?, 0x90?, 0x10394f260?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x140004c56c0?, 0x140004c56c4, 0x140004c57a0?, 0x2?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x14000496450)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x140005a42c0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 44
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 322 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).Start(0x14000541400, {0x1039d5c90, 0x1400036b0e0})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/cache/internal/informers.go:211 +0x40
sigs.k8s.io/controller-runtime/pkg/cluster.(*cluster).Start(0x8?, {0x1039d5c90?, 0x1400036b0e0?})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/cluster/internal.go:104 +0x70
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1(0x14000202c00)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:223 +0xd0
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile in goroutine 273
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:207 +0x204

goroutine 327 [select]:
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch(0x140005180f0)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:126 +0x80
created by sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Start in goroutine 308
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:113 +0x1c4

goroutine 65 [IO wait]:
internal/poll.runtime_pollWait(0x10b9c4ca0, 0x72)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x140005f9f00?, 0x14000790000?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140005f9f00, {0x14000790000, 0xa000, 0xa000})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x140005f9f00, {0x14000790000?, 0x140006c3828?, 0x1026549cc?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x140004a4018, {0x14000790000?, 0x140006c3748?, 0x102313f0c?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/net.go:179 +0x34
crypto/tls.(*atLeastReader).Read(0x1400000fe48, {0x14000790000?, 0x1400000fe48?, 0x0?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:805 +0x40
bytes.(*Buffer).ReadFrom(0x1400053c2a8, {0x1039bd008, 0x1400000fe48})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/bytes/buffer.go:211 +0x90
crypto/tls.(*Conn).readFromUntil(0x1400053c000, {0x10bc0f658?, 0x140004a4018}, 0x9fb9?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:827 +0xd0
crypto/tls.(*Conn).readRecordOrCCS(0x1400053c000, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:625 +0x1e4
crypto/tls.(*Conn).readRecord(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0x1400053c000, {0x1400082b000, 0x1000, 0x1027df87c?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:1369 +0x168
bufio.(*Reader).Read(0x1400051c8a0, {0x14000236580, 0x9, 0x1045cb310?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/bufio/bufio.go:244 +0x1b4
io.ReadAtLeast({0x1039bca88, 0x1400051c8a0}, {0x14000236580, 0x9, 0x9}, 0x9)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/io/io.go:335 +0xa0
io.ReadFull(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0x14000236580, 0x9, 0x172604d7?}, {0x1039bca88?, 0x1400051c8a0?})
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/frame.go:237 +0x58
golang.org/x/net/http2.(*Framer).ReadFrame(0x14000236540)
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/frame.go:498 +0x78
golang.org/x/net/http2.(*clientConnReadLoop).run(0x140006c3f88)
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:2275 +0xf8
golang.org/x/net/http2.(*ClientConn).readLoop(0x14000828000)
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:2170 +0x5c
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 64
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:821 +0xabc

goroutine 302 [select]:
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start(0x140006a7380, {0x1039d5bb0, 0x10462bd80})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/internal.go:444 +0x768
github.com/fermyon/spin-operator/internal/webhook.startWebhookServer.func1()
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:110 +0x48
created by github.com/fermyon/spin-operator/internal/webhook.startWebhookServer in goroutine 44
	/Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:109 +0x2f8

goroutine 303 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x14000320ea0)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 302
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 304 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x14000320f30)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 302
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 305 [IO wait]:
internal/poll.runtime_pollWait(0x10b9c48c0, 0x72)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x1400032ee80?, 0xd0000001003420?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x1400032ee80)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:611 +0x250
net.(*netFD).accept(0x1400032ee80)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x14000202d80)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/tcpsock_posix.go:152 +0x28
net.(*TCPListener).Accept(0x14000202d80)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/tcpsock.go:315 +0x2c
crypto/tls.(*listener).Accept(0x140005e4de0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/tls.go:66 +0x30
net/http.(*Server).Serve(0x140002d0000, {0x1039c7fd0, 0x140005e4de0})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:3056 +0x2b8
sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start(0x14000225860, {0x1039d5c90?, 0x1400036b090})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:263 +0x774
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1(0x14000202520)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:223 +0xd0
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile in goroutine 304
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:207 +0x204

goroutine 273 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x14000320fc0)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 302
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 326 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x14000321050)
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 325
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 307 [syscall]:
syscall.syscall6(0x14000078bb8?, 0x10b9b23c8?, 0x88?, 0x104bb0130?, 0x14000078c28?, 0x1023729b4?, 0x104bb0108?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
golang.org/x/sys/unix.kevent(0x2000?, 0x14000026000?, 0x14000078c68?, 0x0?, 0x14000078c58?, 0x10240e63c?)
	/Users/caleb/go/pkg/mod/golang.org/x/sys@v0.16.0/unix/zsyscall_darwin_arm64.go:275 +0x54
golang.org/x/sys/unix.Kevent(0x14000078d00?, {0x0?, 0x14000078c98?, 0x10240e74c?}, {0x14000078e60?, 0x102313624?, 0x14000078cb8?}, 0x102313638?)
	/Users/caleb/go/pkg/mod/golang.org/x/sys@v0.16.0/unix/syscall_bsd.go:397 +0x40
github.com/fsnotify/fsnotify.(*Watcher).read(0x14000078d48?, {0x14000078e60?, 0x14000078d68?, 0xa})
	/Users/caleb/go/pkg/mod/github.com/fsnotify/fsnotify@v1.7.0/backend_kqueue.go:777 +0x48
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0x14000204f50)
	/Users/caleb/go/pkg/mod/github.com/fsnotify/fsnotify@v1.7.0/backend_kqueue.go:547 +0x94
created by github.com/fsnotify/fsnotify.NewBufferedWatcher in goroutine 305
	/Users/caleb/go/pkg/mod/github.com/fsnotify/fsnotify@v1.7.0/backend_kqueue.go:184 +0x1fc

goroutine 308 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Start(0x140005180f0, {0x1039d5c90, 0x1400036b090})
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:118 +0x210
sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start.func1()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:214 +0x28
created by sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start in goroutine 305
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:213 +0x2ac

goroutine 309 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start.func2()
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:248 +0x48
created by sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start in goroutine 305
	/Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:247 +0x6ac

goroutine 365 [IO wait]:
internal/poll.runtime_pollWait(0x10b9c49b8, 0x72)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x140001c1380?, 0x14000520000?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140001c1380, {0x14000520000, 0xa000, 0xa000})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x140001c1380, {0x14000520000?, 0x1400001b828?, 0x1026549cc?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x140004a4058, {0x14000520000?, 0x1400001b748?, 0x102313d94?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/net.go:179 +0x34
crypto/tls.(*atLeastReader).Read(0x14000b12018, {0x14000520000?, 0x14000b12018?, 0x140005fdd40?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:805 +0x40
bytes.(*Buffer).ReadFrom(0x140004ce2a8, {0x1039bd008, 0x14000b12018})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/bytes/buffer.go:211 +0x90
crypto/tls.(*Conn).readFromUntil(0x140004ce000, {0x10bc0f658?, 0x140004a4058}, 0x9fb8?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:827 +0xd0
crypto/tls.(*Conn).readRecordOrCCS(0x140004ce000, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:625 +0x1e4
crypto/tls.(*Conn).readRecord(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0x140004ce000, {0x1400048f000, 0x1000, 0x1027df87c?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:1369 +0x168
bufio.(*Reader).Read(0x1400060d740, {0x1400038a2e0, 0x9, 0x1045cb310?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/bufio/bufio.go:244 +0x1b4
io.ReadAtLeast({0x1039bca88, 0x1400060d740}, {0x1400038a2e0, 0x9, 0x9}, 0x9)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/io/io.go:335 +0xa0
io.ReadFull(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0x1400038a2e0, 0x9, 0x18139bd4?}, {0x1039bca88?, 0x1400060d740?})
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/frame.go:237 +0x58
golang.org/x/net/http2.(*Framer).ReadFrame(0x1400038a2a0)
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/frame.go:498 +0x78
golang.org/x/net/http2.(*clientConnReadLoop).run(0x1400001bf88)
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:2275 +0xf8
golang.org/x/net/http2.(*ClientConn).readLoop(0x140008d2000)
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:2170 +0x5c
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 364
	/Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:821 +0xabc
FAIL	github.com/fermyon/spin-operator/internal/webhook	3.849s
FAIL
make: *** [test] Error 1
@rajatjindal
Copy link
Member

I tried running the tests a few times to reproduce this issue, but unable to. Is there a specific command that triggers this behavior?

@endocrimes
Copy link
Contributor

I think this is actually because we use the shared scheme in the testenv client:

https://github.com/spinkube/spin-operator/blob/main/internal/webhook/admission_test.go#L63-L71

(I wasn't super expecting parallel integration tests - we'll need to switch it to making a new scheme, add in default types, and our types)

@calebschoepp
Copy link
Contributor Author

I tried running the tests a few times to reproduce this issue, but unable to. Is there a specific command that triggers this behavior?

I was getting locally just running make test. Might be really rare or something that is specific to my machine.

@bacongobbler bacongobbler added the bug Something isn't working label Feb 26, 2024
@bacongobbler
Copy link
Contributor

Removing from the initial milestone - looks like something we can fix async. I don't think the bug actually affects anything in prod, but please correct me if I'm wrong on that.

@calebschoepp
Copy link
Contributor Author

Removing from the initial milestone - looks like something we can fix async. I don't think the bug actually affects anything in prod, but please correct me if I'm wrong on that.

Correct, this has no impact on prod deployments.

@calebschoepp
Copy link
Contributor Author

@endocrimes does #134 fix this?

@endocrimes
Copy link
Contributor

Yep

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rajatjindal @endocrimes @bacongobbler @calebschoepp