Creating changes seems to fail with a panic when checking metadata or annotations

[illegalnumbers]

What happened?

I ran pulumi preview with a kubernetes cluster backing it and got a panic stack trace exception that appears to be related to metadata or annotations on a resource. This has happened with two different resources, a NamespacePatch and a CustomResourceDefinition. See below for the stack trace.

Example

 panic: fatal: An assertion has failed: path: [metadata annotations]  |  v: <nil>  | old: <nil>  |  oldInput: <nil>
    goroutine 448 [running]:
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.74.0/go/common/util/contract/failfast.go:23
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf(0x20?, {0x1029cf9dc?, 0x2?}, {0x1400d4d6828?, 0xa?, 0x100f5c960?})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.74.0/go/common/util/contract/assert.go:35 +0xe0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0x1400d4d7258, {0x140097a4c20, 0x2, 0x2}, {0x0, 0x0}, {0x0, 0x0?}, {0x0?, 0x0?}, ...)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3267 +0x100
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0x14010a6de90?, {0x14010a6de90, 0x1, 0x1}, 0x1400d4d6cd8?, 0x100f5c298?, 0x102fbe1a0?, 0x14010a6de60?, 0x0)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3369 +0x288
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0x1400d4d7258, {0x14010a6de90, 0x1, 0x1}, {0x1030068c0, 0x14006124120}, {0x1030068c0, 0x14005a9b8f0?}, {0x1030068c0?, 0x14005a7bf80?}, ...)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3290 +0x2c8
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0x14010077185?, {0x0, 0x0, 0x0}, 0x1401a16200b?, 0x0?, 0x102951a3f?, 0x2?, 0x0)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3369 +0x288
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.convertPatchToDiff(0x140061240f0, 0x14005a9b8c0, 0x1030068c0?, 0x14000f60030?, {0x140097a4be0, 0x2, 0x2})
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3195 +0x104
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*kubeProvider).Diff(0x140003e3500, {0x1033b8a48, 0x14000f70cf0}, 0x1400c5e6080)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:1637 +0xcf0
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x1033b8a48, 0x14000f70cf0}, {0x103237cc0?, 0x1400c5e6080})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.74.0/proto/go/provider_grpc.pb.go:555 +0x74
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x1033b8a48, 0x1400555a510}, {0x103237cc0, 0x1400c5e6080}, 0x140075bc640, 0x14005d9e018)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x308
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x103337c80?, 0x140003e3500}, {0x1033b8a48, 0x1400555a510}, 0x1401a160000, 0x140008f7cc0)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.74.0/proto/go/provider_grpc.pb.go:557 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0x14000832780, {0x1033c4100, 0x14000003ba0}, 0x14005e05d40, 0x14000bd6a80, 0x104a6c588, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:1337 +0xc90
    google.golang.org/grpc.(*Server).handleStream(0x14000832780, {0x1033c4100, 0x14000003ba0}, 0x14005e05d40, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:1714 +0x82c
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:959 +0x84
    created by google.golang.org/grpc.(*Server).serveStreams.func1
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:957 +0x16c

Output of pulumi about

CLI
Version 3.132.0
Go Version go1.23.1
Go Compiler gc

Plugins
KIND NAME VERSION
resource auth0 3.5.1
resource gcp 6.67.1
resource gcp 5.26.0
resource kubernetes 4.18.0
resource kubernetes 3.30.2
language nodejs 3.132.0-dev.0
resource tls 4.11.1

Host
OS darwin
Version 13.6.7
Arch arm64

This project is written in nodejs: executable='/Users/illegalnumbers/.asdf/shims/node' version='v20.11.0'

resources omitted for privacy

Dependencies:
NAME VERSION
@pulumi/auth0 3.5.1
@types/node 8.10.66
auth0 4.2.0
tslint 6.1.3
@pulumi/gcp 5.26.0
@pulumi/kubernetes 4.18.0
@pulumi/pulumi 3.103.1
@streamnative/pulumi-common 4.2.3
@types/auth0 2.35.11
typescript 3.9.10

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[illegalnumbers]

We also tried with the latest 3.x version which received the same problem.

[illegalnumbers]

Other things tried - removing ~/.pulumi, removing node_modules and reinstalling, relogging in to gcp where the cluster is located at, etc.

[blampe]

@illegalnumbers could you give a code example of a resource where you've seen this?

[illegalnumbers]

@illegalnumbers could you give a code example of a resource where you've seen this?

yea i can try to get a better repro in a bit, i need to get my stacks fixed first :) but afterward i will work on having something better here

[mjeffryes]

@illegalnumbers did you end up constructing a code example? We can't offer much help without a reproduction.