Improve IaC code quality with pre-commit hooks for Terraform

Install and set up pre-commit

Why pre-commit and how to set it up locally?

It's where we're going to start. Pre-commit v2 release poped in early 2020. It's a fantastic tool to use in combination with your IDE OnSave formatting (if you have some) so that you make sure to always push clean commits (depending on the rules you've set up).

First, we'll see how to customize your workspace for you to be able to launch pre-commit scripts when you run the git commit command.

Requirements

• Python (python3 preferred) and pip (pip3 😉), and that's all.

  apt update && apt install -y python3 python3-pip
  
  pip3 install pre-commit --upgrade

The config file

• An example of .pre-commit-config.yaml

  # .pre-commit-config.yaml
  default_stages: [commit]
  repos:
      - repo: https://github.com/commitizen-tools/commitizen
        rev: v2.18.0
        hooks:
            - id: commitizen
              stages:
                  - commit-msg
      - repo: https://github.com/antonbabenko/pre-commit-terraform
        rev: v1.50.0
        hooks:
            - id: terraform_fmt
            - id: terraform_validate
            - id: terraform_tflint
            - id: terraform_tfsec
            - id: checkov
            - id: terraform_docs_replace
      - repo: https://github.com/pre-commit/pre-commit-hooks
        rev: v4.0.1
        hooks:
            - id: trailing-whitespace
            - id: end-of-file-fixer
            - id: check-yaml
            - id: check-added-large-files

In this key file, you'll choose hooks from a .pre-commit-hooks.yaml file defined in git repositories and default_stages on when to trigger them.

For this example, you may need to "install" a specific hook-type : commit-msg

pre-commit install --hook-type commit-msg

Run the scripts

First, add some terraform code

# main.tf
terraform {
  required_version = ">= 1.0.0, < 1.0.4"
}

resource "random_password" "password" {
  length = 16
  special = true
  override_special = "_%@"
}

output "test" {
  value = random_password.password.result
  sensitive = true
}

Manually trigger hooks to confirm that everything works pre-commit run -a

root@655088f99f35:/test# pre-commit run -a
[INFO] Initializing environment for https://github.com/commitizen-tools/commitizen.
[INFO] Initializing environment for https://github.com/antonbabenko/pre-commit-terraform.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Installing environment for https://github.com/antonbabenko/pre-commit-terraform.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Terraform fmt............................................................Failed
- hook id: terraform_fmt
- files were modified by this hook

main.tf
Terraform validate.......................................................Passed
Terraform validate with tflint...........................................Failed
- hook id: terraform_tflint
- exit code: 127

/root/.cache/pre-commit/reposnevwej6/terraform_tflint.sh: line 66: tflint: command not found

Terraform validate with tfsec............................................Failed
- hook id: terraform_tfsec
- exit code: 127

/root/.cache/pre-commit/reposnevwej6/terraform_tfsec.sh: line 25: tfsec: command not found

Checkov..................................................................Failed
- hook id: checkov
- exit code: 1

Executable `checkov` not found

Terraform docs (overwrite README.md).....................................Failed
- hook id: terraform_docs_replace
- exit code: 1

/bin/sh: 1: terraform-docs: not found
Command 'terraform-docs md ./ > .//README.md' returned non-zero exit status 127.

Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed

Exciting things get displayed, and that's because you have to install some of the tools as packages.

Hooks for Terraform

Now we'll be focusing on the tool Terraform. What may be the valuable hooks that we can install to increase your infrastructure as code quality? You've got some hints from the output above or from the config file section 🙂. After we've set them up, I'll be explaining their usefulness.

Which pre-commit to install for Terraform?

Install them (brew is also available for linux)

brew install tfsec tflint checkov terraform-docs

Now, you can rerun pre-commits and see them in action.

Terraform fmt............................................................Passed
Terraform validate.......................................................Passed
Terraform validate with tflint...........................................Passed
Terraform validate with tfsec............................................Passed
Checkov..................................................................Passed
Terraform docs (overwrite README.md).....................................Failed
- hook id: terraform_docs_replace
- files were modified by this hook
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
Check for added large files..............................................Passed

The majority show Passed, which is OK, and Terraform docs failed to indicate that files were modified by this hook which means that it created the doc as expected but didn't stage it with git yet. No more error then! Let's dive into the details of each hook.

Linting tools

They're here for code quality readability improvements:

• terraform fmt: The terraform fmt command is used to rewrite Terraform configuration files to a canonical format and style. It is the built-in terraform command to format code towards Hashicorp standards.
• terraform validate: The terraform validate command validates the configuration files in a directory, referring only to the configuration and not accessing any remote services such as remote state, provider APIs, etc. It is the built-in terraform command to check the integrity of the code towards Hashicorp standards.
• tflint: TFLint is a framework and each feature is provided by plugins. It finds possible errors (like illegal instance types) for Major Cloud providers (AWS/Azure/GCP), warns about deprecated syntax or unused declarations, enforces best practices and naming conventions.
• trailing-whitespace & end-of-file-fixer : Pretty straight forward. These come along with pre-commit installation.

Cloud Provider best practices

Now is a big part of this article: it's where terraform-specific pre-commit hooks improve your code quality drastically. Based on updated databases of best practices, the following 2 packages display warnings or errors in the providers' resources.

• tfsec A static analysis security scanner for your Terraform code
• checkov Checkov scans cloud infrastructure configurations to find misconfigurations before they're deployed.

As an example, let's take a look at the rule aws-kms-auto-rotate-keys from tfsec : it ensures that every resource "aws_kms_key" has the instruction enable_key_rotation = true

It is a best practice to enforce rotation on secrets but if for some reason you want to set key rotation to false, you can skip it using this instruction:

resource "aws_kms_key" "bad_example" {
	enable_key_rotation = false #tfsec:ignore:AWS004
}

You can do the same with checkov with #checkov:skip=CKV_AWS_20:The bucket is a public static content host

Collaborative best practices terraform-docs, commitizen, ...

• commitizen define a standard way of committing rules and communicating it (using the cli provided by commitizen). It's the one I recommend the most. It enforces the commit message to follow the standard of feat(scope): message. It is handy for generating changelogs and incrementing correctly the semantic version
• terraform-docs Generate Terraform modules documentation in various formats. This tool frees you from writing the objects you're creating with your code as it does it for you. It is customizable through a .terraform-docs.yml file.

Others check-yaml, check-added-large-files, ...

I showed you the essentials, but you definitely should check by yourself which tool best suits your needs. It's an open world; go explore!

Run it in a CI

Pre-commit hooks are great locally, but you'll contribute to a codebase shared with many other DevOps. So what if they don't have pre-commit installed?

Well, the side effect will be that others' changes may block you if they didn't run the pre-commit scripts for a moment. For instance, when you'll pull updates that weren't lint from the repo and then try to commit again. As a workaround, you may want to check somewhere if commits are san. Running pre-commit tools in a CI is one solution.

How to check if pre-commit scripts are run before git push?

You can set up pre-commit to run in gitlab-ci. My colleague and friend Stan made this simple with its Docker image and this usage example. Once set, the job will fail on any pre-commit hook failure like below.

We've seen how to set up pre-commit hooks on a Terraform project, which tools to install, and how to keep your repo checked even with new collaborators. I've witnessed a better global code quality over my different terraform repos since using pre-commit hooks.

As a DevOps, what may be your needs for code quality checking? What other tech or language do you need to hook (like Ansible or Python)? Do the benefits of CICD code quality outweigh the time it takes to run and the resources it uses? What do you think?