Synchronize a directory's contents with a git repository

Our use case

One of the many challenges we, Padok SREs, often face is making sure everything we do is maintainable by our clients. We usually write tons of documentation to make sure of that, commonly by writing a lot of markdown files.

During my last project, we were working on an internal infrastructure used only to improve the developers’ daily experience. To make everything particularly user-friendly, we put all of our documentation on a static website using mkdocs. We made sure that the documentation was only accessible through a VPN, and was available with a custom URL.

We didn't add unnecessary complexity by adding another CI/CD pipeline to maintain. Instead, we looked for a way to directly synchronize our git repository containing the documentation to a Kubernetes volume. This would allow us to quickly deploy updates to the website inside our cluster. Such a method allows us to live reload the website each time we modify the documentation. This also directly puts the documentation website behind the VPN protection of our Kubernetes cluster and allows us to use our own DNS to choose the hostname.

Since we didn't find any existing tool, we decided to create our own. That is Padok’s first open-source project: the git-volume-reloader.

How does it work?

The git-volume-reloader is a small tool written in Go. It simply clones a specific branch of a git repository, inside a specified working directory. A webhook sent by the git service provider triggers synchronization.

In our specific use case, we deployed the git-volume-reloader as a sidecar container, next to the main one running an mkdocs server. The two containers share a volume, which is the working directory in which the git-volume-reloader synchronizes the git repository. A set of guidelines and manifest examples are available inside the repository if you would like to see how it is done and/or do the same for your documentation.

What comes next?

Currently, we still have a set of open issues that we will tackle as soon as possible.

• The tool will crash if you push force to the git branch it is synchronized to. To fix this, we can do a pull force at each synchronization. Another way around would be to make sure that the liveness fails if such an event occurs so that Kubernetes would restart the Pod, and the repository would be cloned from scratch.
• There is a possible race condition if you push twice at the same time.

There are also some improvements we have decided to work on.

• The only git provider you can synchronize to is Github. We should, at least, handle GitLab repositories.
• The only authentication method available is through SSH by using the Github deploy keys system. We need to handle username and password authentication.
• We currently do not have any CI in the project. We have to implement some unit and integration tests to run on each new Pull Request. Also, we need a release management system to handle versioning and publishing a public Docker image of the git-volume-reloader.

The last point, in particular, raises a few questions to which we do not have answers yet. For example, at the time I'm writing this article, we are still debating at Padok which container registry we should use. But the main concerns are our contributing and release workflows.

The git-volume-reloader is the very first open-source project of Padok. But many more are coming. Check out our Github Page regularly. Of course, feel free to contribute to the open issues, or start discussing with us about different contributing and release workflows you are familiar with!