Terraform layout

The baby project

When a Terraform project is born it’s like a baby: small, pure and without bad habits. And you want to show to everyone:

├── Readme.md
├── main.tf
├── output.tf
├── variables.tf
├── terraform.tfstate

The commands you need to manage it are very small and nice (eat, sleep, poop, etc.):

$ terraform init
$ terraform plan 
-out=tfplan
$ terraform apply tfplan
Photo by Colin Maynard on Unsplash

But all the babies grow and your project does the same. It starts being more complex, nothing that you can not manage, of course, you are ready for this, you have read about it and you are able to improve it: you divide the responsibilities creating some modules (someday we will talk about them), you separate some components from the main files and you are (still) proud of it.

├── Readme.md
├── backend.tf
├── main.tf
├── output.tf
├── provider.tf
├── variables.tf
|
├── modules/
| ├── module1/
| ├── module2/
| ├── ...

The baby grows

There is a moment when you don’t know how but the cute baby project becomes an unmanageable and ugly teenager project, and you need to deal with it.
You don’t know what is doing in some environments: Is she taking drugs in development? Is he studying or partying in staging? You still love it but you are not so proud of it like in the past…

Photo by Parker Gibbons on Unsplash

You need to deal with it. And you deal with it.
You know that is not a good solution but you need to do it, so you start duplicating the code between environments. You copy your code between environments (you know this is not the best solution , but at least you try to do as well as possible) and you add some variables definitions to configure between envs.

├── Readme.md
|
├── development/
| ├── backend.tf
| ├── main.tf
| ├── output.tf
| ├── provider.tf
| ├── variables.tf
| ├── development.tfvars
| |
| ├── modules/
| | ├── module1/
| | ├── module2/
| | ├── ...
|
├── production/
| ├── backend.tf
| ├── main.tf
| ├── output.tf
| ├── provider.tf
| ├── variables.tf
| ├── production.tfvars
| |
| ├── modules/
| | ├── module1/
| | ├── module2/
| | ├── ...
|
├── staging/
| ├── backend.tf
| ├── main.tf
| ├── output.tf
| ├── provider.tf
| ├── variables.tf
| ├── staging.tfvars
| |
| ├── modules/
ssh-rsa
| | ├── module1/
| | ├── module2/
| | ├── ...
|
├── test/
| ├── backend.tf
| ├── main.tf
| ├── output.tf
| ├── provider.tf
| ├── variables.tf
| ├── test.tfvars
| |
| ├── modules
| | ├── module1/
| | ├── module2/
| | ├── ...

It’s not all bad, the way to manage have changed a bit but is still easy and nice to perform. Not everything needs to be a problem!

$ cd 
$ terraform init
$ terraform plan 
-out=tfplan
-var-file=.tfvars
$ terraform apply 
-var-file=.tfvars
tfplan
$ terraform outputcommon_network

But, there is a moment when you know that something is not going well.
When you change something in staging, you need to copy to test, production, development… It’s not a funny work, and someone someday (maybe a Friday after lunch?) totally forgot to propagate some change between environments. and this is chaos. You need to solve it, you want to keep it back and you start googling about it.

You will find lots of proposals like this basic one, or like this other one, or even, this one. If you search long enough you will be pointed to Terragrunt but none of this proposals satisfies our needs.

At this point, let me talk about our needs and what we were searching in a “relayout”.

Our requirements

  • We need something reusable between environments. Our specific situation is that the infrastructure described is exactly the same except for minimal changes (instances names and few things more).
  • We search something easy to read. We’ve discarded the use of Terragrunt or the Terraform Workspaces because add a point of complexity that we are trying to avoid.
  • We need something fast to implement. Reconfiguring all the current infrastructure as modules will add an extra time, that we don’t have. (Maybe we will talk another day about it).
  • We want something easy to maintain. We’ve faced the problem to repeat the same change in multiple environments by hand, updating the same file in each environment, and we are trying to avoid this situation.
  • We desire something easy to upgrade. We have the certainty that this infrastructure will evolve a lot, so it’s necessary to design it thinking in the future.
  • We desire a code easy to understand. We know that this code will be used by different teams with different knowledge on Terraform and on the infrastructure itself so we thought of them when we decided the solution.
  • We want a project as secure as possible. We have experienced the situation that for a human error, someone modifies the infrastructure in a wrong environment. This was an absolute chaos (and hours of pain and tears) that we don’t want to repeat.

Having all these points in mind we arrived at this strange point when our “teenager project” becomes an adult project.

Adult project

Keeping in mind all the points described below we finished with the code refactoring. During this point, the infrastructure itself has not changed at all. Again, the only thing that has changed is the layout, the file organization and the variables declaration.

First of all we moved all the manifests into a folder named (uah! an original name!) manifests. Using this approach we only need to write changes one time, not one time for each environment.

To configure the different environments we declared a backend (for the remote tfstate) and a variable file for each environment. To avoid mistakes of going to an undesired folder we decided to name all this backend and variables files with the name of the current environment (.auto.tfvars and .backend.tfvars).

├── Readme.md
|
├── environments/
| |
| ├── development/
| | ├── development.auto.tfvars
| | ├── development.backend.tfvars
| |
| ├── production/
| | ├── production.auto.tfvars
| | ├── production.backend.tfvars
| |
| ├── staging/
| | ├── staging.auto.tfvars
| | ├── staging.backend.tfvars
| |
├── manifests/
| ├── backend.tf
| ├── main.tf
| ├── output.tf
| ├── provider.tf
| ├── variables.tf
| ├── modules/
| | ├── module1/
| | ├── module2/
| | ├── ...
|

With this layout the way to manage it it’s a bit different. We need to access the desired environment and reference the manifests folder as plan/init folder (a small difference that simplifies our lives a lot).

$ cd environments
$ terraform init 
-backend-config=.backend.tfvars
../../manifests
$ terraform plan 
-out=tfplan
-var-file=.tfvars
../../manifests
$ terraform apply 
-var-file=.tfvars
../../manifests
tfplan

With this approach, the layout is simple, easy to maintain and clear to read. We achieved the desired state and we are able to work smoothly than before.

At this point, we reconfigured “a bit” the manifests. Now, having all of them in a single folder and shared between environments we have the ability to change them and test very fast in all the envs.

├── manifests/
| ├── backend.tf
| ├── backoffice.tf
| ├── database.tf
| ├── frontend.tf
| ├── output.tf
| ├── persistent_storage.tf
| ├── provider.tf
| ├── security.tf
| ├── modules/
| | ├── module1/
| | ├── module2/
| | ├── ...
|

We changed from the initial paradigm main.tf, output.tf and vars.tf to a set of files based on the “function”. If we need to change something of the frontend infrastructure we will go to the frontend.tf file. With this approach we gain two things:

  • Velocity detecting changes: If something changed in any part of the code we will see at which element of the architecture it applies, only taking a look into the file name.
  • Independence: If we need to delete some part of the infra we need to delete only one file. Or if we need to add something we don’t need to touch a 500 lines file, only create a new one.

Terraform output (the part we are not proud of )

With this approach “we faced only one problem”, terraform output doesn’t work because in the folders the provider is not defined. The fast way to solve this is having a provider.tf in each environment, as a copy of the one present in the manifests folder, with the same content:

provider "PROVIDER_NAME" { }

And, of course, configured via the .auto.tfvars files.

read original article here