Add exported function to retrieve attribute from state for use during tests

[bendbennett]

SDK version

v2.20.0

Use-cases

Occasionally, when writing acceptance tests it is useful to be able to extract and store a value from state in order to use in subsequent test steps. For instance, when using the Taint field in a TestStep it is useful to be able to verify that a resource has been destroyed and recreated.

Attempted Solutions

Adding a function along the following lines to each provider which needs to be able to compare attributes before and after test steps.

func testExtractResourceAttr(resourceName string, attributeName string, attributeValue *string) r.TestCheckFunc {
	return func(s *terraform.State) error {
		rs, ok := s.RootModule().Resources[resourceName]

		if !ok {
			return fmt.Errorf("resource name %s not found in state", resourceName)
		}

		attrValue, ok := rs.Primary.Attributes[attributeName]

		if !ok {
			return fmt.Errorf("attribute %s not found in resource %s state", attributeName, resourceName)
		}

		*attributeValue = attrValue

		return nil
	}
}

Proposal

Add an exported function to /helper/resource that exposes this functionality to avoid code duplication in providers.

References

[bflad]

We've certainly seen providers that use this class of attribute value extraction across TestStep. 😄 I'm wondering if its worth investigating this enhancement against the context of these as well:

• Core FR: Check that only an 'update' occurred in a test #61
• Allow easier testing of resource's disappearance #73
• Add ability to perform assertions against the plan before and after TestStep Config is executed #64

They're all circling around verifying the resource under test was planned for and/or deleted/recreated properly. More easily enabling provider developers to extract values and perform their own equivalence testing certainly is interesting.

[marshallford]

Is there an official solution for this yet? I'm confused on how to best compare/test attributes across multiple test steps.

[austinvalle]

Hey there @marshallford 👋🏻,

If I'm understanding your use-case correctly, you may be interested in #295 and the upcoming PR #330 which resolves it

[marshallford]

Thanks for the help @austinvalle! I looked over the PR you linked and I have a couple of questions if you don't mind:

1. What is the difference between using state checks (which I have only used to test an output) and the various check functions recommended in the docs for testing resource attributes?

2. Regarding that PR specifically, will there be examples provided for comparing the same or different attribute across multiple test steps? I'm still not clear on how that might work. For context, here's the function that I grabbed from the random provider and its usage in my provider.

[austinvalle]

Absolutely!

1. What is the difference between using state checks (which I have only used to test an output) and the various check functions recommended in the docs for testing resource attributes?

So we're actually in the process of deprecating (originally #266) the (TestStep).Check field but #330 was the final piece of functionality we needed to add before we could reasonably do so. That documentation definitely should be updated to suggest using the new state check documentation though, so I'll submit a PR to make that suggestion, apologies for that confusion.

The reason we introduced the statecheck package to replace (TestStep).Check was because the underlying logic being exposed (via (TestStep).Check and numerous other terraform-plugin-testing features) relied on a legacy flatmap syntax to operate rather than the raw tfjson.Plan/tfjson.State representations. So now if you want to build a custom state check, you can use the actual JSON representation from Terraform (same reason we introduced plancheck).

2. Regarding that PR specifically, will there be examples provided for comparing the same or different attribute across multiple test steps? I'm still not clear on how that might work. For context, here's the function that I grabbed from the random provider and its usage in my provider.

Thanks for that feedback! I think you're right that what's in that PR right now mostly classifies as reference documentation. I'll make a note to add a better example, perhaps in our overview sections. (Our testing docs in general could probably use some TLC 😃 )

As for your example, here is what I believe the end state of that test would look like completely using statecheck and the new value comparers:

func TestAccNavigatorRun_skip_run(t *testing.T) {
    t.Parallel()

    // Here we set up the state check comparer, we tell it how we want state values to be compared (we want all state values to be equal)
    compareResourceCommand := statecheck.CompareValue(compare.ValuesSame())

    resource.Test(t, resource.TestCase{
        PreCheck:                 func() { testPreCheck(t) },
        ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
        Steps: []resource.TestStep{
            {
                Config:          testTerraformFile(t, filepath.Join("navigator_run", "skip_run")),
                ConfigVariables: testDefaultConfigVariables(t),
                ConfigStateChecks: []statecheck.StateCheck{
                    // Here are some equivalent checks to `TestCheckResourceAttrSet`
                    statecheck.ExpectKnownValue(navigatorRunResource, tfjsonpath.New("id"), knownvalue.NotNull()),
                    statecheck.ExpectKnownValue(navigatorRunResource, tfjsonpath.New("command"), knownvalue.NotNull()),
                    // Add the first state value "command" to the comparer. There is nothing to compare to at this point.
                    compareResourceCommand.AddStateValue(navigatorRunResource, tfjsonpath.New("command")),
                },
            },
            {
                Config: testTerraformFile(t, filepath.Join("navigator_run", "skip_run_update")),
                ConfigVariables: testConfigVariables(t, config.Variables{
                    "ansible_navigator_binary": config.StringVariable(acctest.RandString(8)),
                }),
                ConfigPlanChecks: resource.ConfigPlanChecks{
                    PreApply: []plancheck.PlanCheck{
                        plancheck.ExpectNonEmptyPlan(),
                    },
                },
                ConfigStateChecks: []statecheck.StateCheck{
                    // Here are some equivalent checks to `TestCheckResourceAttrSet`
                    statecheck.ExpectKnownValue(navigatorRunResource, tfjsonpath.New("id"), knownvalue.NotNull()),
                    statecheck.ExpectKnownValue(navigatorRunResource, tfjsonpath.New("command"), knownvalue.NotNull()),
                    // Add the second state value "command" to the same comparer. The second value will be compared with the first,
                    // if they aren't equal, the state check will fail here.
                    compareResourceCommand.AddStateValue(navigatorRunResource, tfjsonpath.New("command")),
                },
            },
        },
    })
}

Apologies if there are any compiler errors in that example, I didn't write it in a code editor 😆

[marshallford]

Thank you Thank you! That all tracks and the example you provided is very helpful. Looking forward to the doc updates as well.