Operable

Grafana is an open-source dashboarding platform for all your cloud environments.

Grafana is an open-source dashboarding tool created by the eponymous company, Grafana Labs. It enables users to create dynamic, customizable dashboards to monitor and analyze metrics related to your infrastructure.

A wide range of data sources can be connected to Grafana: temporal databases such as InfluxDB, Prometheus, ElasticSearch, and even the native monitoring services of your preferred Cloud Provider. Grafana's intuitive user interface then lets you group the various data into real-time graphs, gauges and bar charts, to name but a few.

Grafana is an essential monitoring tool for your Kubernetes clusters, easy to install and configure thanks to its Helm chart. What's more, you can define your dashboards using ConfigMap. If you don't want the responsibility of managing your visualization tool, don't worry, there's a Grafana Cloud SaaS offering.

With the rise of microservice architectures and the use of the cloud to create complete, complex environments, Grafana has become an essential tool for both operational and development teams.

Kustomize makes it easy to configure your complex deployments in Kubernetes.

Kustomize lets you manage resource configurations for Kubernetes using YAML files with easy-to-use syntax. It also enables you to manage complex configurations for multi-environment applications by applying serial configurations.

In fact, Kustomize applies patches and overlays to a basic configuration. This simplifies managing complex configurations more than Helm, where you must always redefine a value file. And all this while keeping your code as DRY as possible!

Moreover, Kustomize automatically generates new Kubernetes secrets whenever you modify a data field in the configuration file. This feature keeps secrets secure while simplifying your rollbacks.

However, Kustomize is more challenging to grasp for people unfamiliar with Kubernetes resources and their declaration in YAML.  It's also less flexible than Helm in terms of customization: it's impossible to integrate logic via templating, for example.

We believe that Kustomize and Helm are two complementary tools for managing deployment configurations for Kubernetes. While Kustomize is ideal for managing configurations modularly, Helm offers a convenient way of managing more complex application packages.

By using Kustomize with Helm, teams can benefit from the advantages of both tools. For example, Helm can be used to manage complete application packages, while Kustomize can be used to customize specific configurations for deployment environments.

Kustomize is a powerful tool for managing deployment configurations for Kubernetes, with a simple syntax that perfectly complements Helm.

Prometheus Operator makes it easy to deploy and manage an entire technical stack around Prometheus to monitor a Kubernetes cluster.

Prometheus is the benchmark tool for metrology on Kubernetes architectures. Deployment and management are made particularly simple with Prometheus Operator, while other ancillary, but no less necessary, components are added to improve the operability of your platform: 

• Alerting with Alerte Manager
• HTTP monitoring with Blackbox Exporter
• Visualize metrics with Grafana

Installation of the Prometheus suite is a single command, and after just a few minutes, you'll have access to all your cluster's metrics and much more. The operator will be able to manipulate Prometheus resources via Kube CRDs. So you won't need to configure your resources in Prometheus but simply declare them in the Kube API. 

We'll automate monitoring by adding these resources to our charts, and Prometheus will monitor each deployed application (metrics or HTTP monitoring, for example).

However, the operator does not solve the major problem with Prometheus: a consolidated, centralized view in multi-environment, multi-cluster architectures. You'll need to deploy components that bridge the gap between different deployments: a central Grafana and solutions like Thanos to increase data retention.

Even if other solutions exist, such as Datadog (for a fee), Prometheus remains Kubernetes's de facto community standard. It will always be a good choice for operating your clusters.

Automate patch management of external dependencies (libraries) for infrastructure and applications

Patch Management is a significant challenge for platform security. Our infrastructures, as much as the applications deployed on them, use external components (dependencies, very often open source) that must be updated regularly to correct security flaws and bugs. With the rapid pace of updates and the growing number of dependencies, it can be difficult to keep all our dependencies up to date.

Renovate is an open-source dependency management tool that automates updating packages in your projects. It analyzes your dependency configuration files (such as package.json, pom.xml, or build.gradle) and automatically generates pull requests for necessary package updates.

Renovate is compatible with various package managers, including npm, yarn, pip, Mavn, and NuGet, making it easily adaptable to different programming languages. It also allows you to analyze the dependencies of your infrastructure, supporting Helm charts, Docker images, and Terraform modules. You can use it with major Git Providers such as Gitlab, Github, BitBucket, and Azure DevOps.

Highly configurable, it adapts perfectly to the development workflows of our projects. It can be integrated via CI tasks or, our preference, as a cronjob in a Kubernetes cluster to optimize processing with Redis caching. This deployment mode will enable us to scale more easily by deploying several "instances" of Renovate (cronjob Kube) to spread the load and adapt its operation.

With daily execution and automatic merge of changes (when the CI is valid), we automate part of the correction of security flaws by automatically applying patches. 

Renovate also enables us to track the evolution of our dependencies by providing us with an overview of the changes (new minor or major versions) that need to be taken into account to keep our dependencies up to date (via the open Merge Request/Pull Request list). We aim to ensure that we don't fall behind on major releases and always benefit from security updates in the long term.

We've created hundreds of infrastructures on several cloud providers and used Terraform every time. An IAC tool is essential when launching into the cloud, as it facilitates collaboration and the operability of an infrastructure. 

Terraform shines through with a wide range of features:

• The use of modules to define sets of resources that meet a precise need and can be easily reused.
• State Terraform for tracking the life cycle of each resource
• Compatibility with multiple clouds and systems thanks to providers. For example: AWS, OVH, but also Github

We know that IAC tools are offered by cloud providers such as CDK AWS, cloud formation, and even ARM for Azure. But their vendor lock-in and lack of interoperability made us lean towards Terraform. 

Despite being a leader in IAC for managing infrastructure, it is necessary to have a framework for the code base. Padok has converged on a WYSIWYG (What You See Is What You Get) pattern that helps standardize code and collaboration. 

The points to remember are : 

• Organizing Terraform states according to business needs
• Create modules that meet a complex or reproducible need
• Don't hesitate to factorize code as your infrastructure evolves

Tips for use 💡

And don't forget the tools for syntax quality and maintainability: terraform fmt, tfllint, tfautomv, or terraform-docs. 

Terraform is today's benchmark tool for building and maintaining infrastructure in the cloud. Tools such as Terragrunt further enhance its ability to manage at-scale infrastructure by offering features to avoid code redundancy, known as DRY (Don't Repeat Yourself).

Terragrunt is a tool offered by Gruntwork to enhance Terraform and boost its ability to manage multi-module deployments.

Terraform is the current community standard for as-code deployment of cloud resources. It includes libraries (called "providers") for almost all the resources of the major Cloud Providers.

However, Terraform has its limitations, penalizing teams who need to manage a multi-module infrastructure or large infrastructures. Indeed, in such cases, it is often necessary to split Terraform deployment into several modules (sometimes also called layers) to simplify them or avoid collisions of Terraform states.

However, this can quickly become very complicated to manage as it becomes necessary :

• Use remote states to share outputs between states
• Duplicate or over-template backend configurations that are not natively configurable in Terraform
• Manage common or environment-specific variables, for example, via files and symbolic links

Terragrunt sits on top of this to create and manage auto-generated Terraform workspaces. Terragrunt's enhanced functionality can be used to link layers. Terragrunt provides a better link between layers while relying on Terraform's proven deployment capabilities.

What's more, Terragrunt is configured using the same language as Terraform, HashiCorp Language (HCL), which has been extended to add the necessary functionalities. This facilitates team training and reduces the feeling of having a new tool to master.

Today, other tools try to meet this need, but Terragrunt is our favorite because it achieves the result by adding only a very thin layer around Terraform.