Compatibility Testing Pulumi HCL | Pulumi BlogSkip to main content<br>Latest release: Full support for Terraform state, cross-language modules, and HCL as a first-class language.<br>See the release

25.6K<br>Contact us<br>Sign in<br>Dashboard<br>Get started<br>25.6K<br>Contact us<br>Sign in<br>Dashboard<br>Get started

Navigation

25.6K<br>Contact us<br>Sign in<br>Dashboard<br>Get started

Copy logo SVG<br>Download mark<br>Get other variants<br>Read brand guide

Blog

In this post<br>How we compatibility test Pulumi HCL<br>Writing tests with LLMs<br>Conclusion

Pulumi HCL has at its core a simple promise:<br>A program that works for tofu apply will also work for pulumi up.

This must be true to allow Terraform modules to be shared between tofu config and Pulumi programs. This property makes testing Pulumi HCL simple. Let me explain.<br>At the end of the day, Pulumi is a system to translate actual state & desired state into a series of imperative actions, so actual state can be reconciled to desired state. Terraform is a system to translate actual state & desired state into a series of imperative actions, so actual state can be reconciled to desired state. How desired state is expressed can be radically different, and the underlying reconciliation engine can be radically different, but at the end of the day, both tools do the same thing:<br>Executing a Terraform program looks like this:

flowchart LR<br>tf["*.tf files"]<br>current["current state"]<br>subgraph engine["reconciliation engine"]<br>direction LR<br>desired["desired state"] --> internal["provider steps"]<br>end<br>output["provider steps"]

tf --> desired<br>current --> engine<br>engine --> output<br>Executing a Pulumi program is more dynamic, because the reconciliation engine is in more active dialog with the user&rsquo;s program. That said, the diagram is the same shape. To match semantics, Pulumi HCL dynamically bridges any Terraform provider in the registry. This means that, for the subset of Pulumi programs that are valid OpenTofu programs, both programs take the same input (*.tf files) and produce the same step output (Terraform provider steps). Providers are the part of our model that generates user-observable behavior, which means if we match what providers see, we match what users see. This gives us a really nice definition of correctness for Pulumi HCL1:<br>Pulumi HCL correctly interprets an HCL program when it generates the same set of provider steps as tofu does.

If you are familiar with property-based testing, you might be thinking this looks like a testable property. You&rsquo;re right.

How we compatibility test Pulumi HCL<br>We have created a framework to assert on the property above for Pulumi HCL: tfcompat. Each tfcompat test has 2 components:<br>The files of the HCL program<br>The providers the program uses<br>I&rsquo;ll walk you through an example test case, then explain how the framework works.<br>An example tfcompat test<br>This Go test is the full code of TestL2SimpleResource:<br>// tests/tfcompat/l2_simple_resource_test.go

func TestL2SimpleResource(t *testing.T) {<br>t.Parallel()<br>tfcompat.RunCase(t, "l2_simple_resource", tfcompat.Case{<br>Providers: []tfcompat.Provider{<br>{Name: "simple", Factory: providers.SimpleProvider},<br>},<br>})

Factory is a function that produces a new in-memory Terraform provider called "simple". The "l2_simple_resource" in the test is the folder that contains the actual HCL program under test:<br># tests/tfcompat/testdata/cases/l2_simple_resource/main.tf

resource "simple_resource" "a_resource" {<br>input_one = "hello"<br>input_two = true

output "some_output" {<br>value = simple_resource.a_resource.result

This test asserts that Pulumi HCL & OpenTofu both:<br>ConfigureProvider the simple provider the same way.<br>Call the same plan RPC during pulumi preview & tofu plan.<br>Call the same ApplyResourceChange to create the resource.<br>Pulumi HCL or OpenTofu didn&rsquo;t call any other provider RPCs.<br>One really important takeaway is that nowhere in this test case do we write down what Pulumi HCL should do. tfcompat.RunCase takes a scenario, but it doesn&rsquo;t take accepted behavior. This will be important later. Before we get there, let me explain how tfcompat.RunCase works.<br>The anatomy of tfcompat.RunCase<br>Every tfcompat.RunCase runs 2 parallel processes, then compares the results:<br>The Terraform Side: tfcompat.RunCase runs each Terraform provider in-memory, then copies the files in its test directory to a temp dir and runs tofu plan, then tofu apply against the temp dir. We use TF_REATTACH_PROVIDERS to have tofu attach to our in-memory Terraform providers.

The Pulumi Side: tfcompat.RunCase runs each Terraform provider in-memory & copies the test files in its test directory to a separate test dir, and runs pulumi preview, then pulumi up against the temp dir. We use PULUMI_BRIDGE_REATTACH_PROVIDERS to instruct our dynamic bridge to attach to our in-memory provider.

For both Pulumi & Terraform, the test harness records each provider&rsquo;s gRPC calls for all providers and it records the stack outputs for both invocations.<br>After both runs have completed, the...