Experience Feedback: Colizey AWS migration

Target of the migration on AWS

State before migration

Colizey is a marketplace specializing in sporting goods. It is a monolithic application mainly developed in PHP.

Although already hosted in the Cloud, it does not benefit from all its advantages because it is deployed on a single virtual machine, thus ensuring no scalability of the application or redundancy. In addition, Colizey is administered and deployed manually, which makes each update of the application risky because potentially error-prone.

Target architecture

The target architecture (see below) that we proposed to Colizey for its migration to AWS is centered around Kubernetes. This open source container orchestrator is today the reference for deploying robust and resilient containerized applications in the Cloud. Kubernetes (K8s) is available on AWS through Elastic Kubernetes Service (EKS) which offers a managed K8s service to its users (K8s is a complex component to manually install and maintain).

To continuously deploy the application and the tools that come with it on K8s, we rely on ArgoCD, an open source tool well known to the Kubernetes community, allowing GitOps based deployment: by versioning the desired state of its K8s cluster in Git, ArgoCD takes care of constantly reconciling the desired state with the current state of the cluster.

Deploying a new version of your application consists in changing a version number in a file, pushing the changes to Git, then ArgoCD takes care of the rest. In the contratry, a manual change made in the cluster is immediately corrected by ArgoCD.

This therefore allows for a fully automated deployment approach and makes Git the single source of truth about the state of your Kubernetes cluster.

Project progress

In order to measure the success of the project as it progresses, we have defined three successes to be achieved at the end of the project:

1. Colizey production and sandbox environments are migrated to AWS
2. Ephemeral preview environments are accessible on AWS
3. A CI/CD pipeline automatically deploys Colizey in production

After a preliminary and technical strategy phase (1 week), we have defined the following plan to successfully complete this project in 5 weeks:

1. Setting up the network topology / accounts in Hub & Spoke
2. Creation of 3 landing zones (Production, Sandbox and QA) and their tooling (RDS, EKS, ArgoCD, DNS, ELK, etc.)
3. Data migration over time with Amazon Data Migration Service
4. Containerization of the Colizey application
5. Setting up CI/CD with GitHub Actions
6. Creation of ephemeral preview environments
7. Final production deployment

In this article, we will focus on two central topics of this migration: data migration, absolutely critical to put Colizey into production on AWS, as well as the creation of ephemeral application environments, offering developers a flexible experience, in a pure DevOps spirit.

Technical focus: Data migration with Amazon DMS

Amazon Data Migration Service (DMS) is a service responsible for migrating your data to AWS. DMS supports multiple external data sources (PostgreSQL, MySQL, MS SQL, Oracle, S3, MongoDB, etc.) as well as multiple targets within AWS (RDS, S3, DynamoDB, Redis, DocumentDB, OpenSearch, etc.), it is also capable of transforming the data at the time of its transfer, which makes it a powerful tool.

As part of the Colizey migration, we focused on the migration from PostgreSQL to PostgreSQL, without intermediate transformation. The advantage of DMS is that it can continuously replicate changes from the source database as long as it is running, in order to minimize the delay to be caught between the source database and the target database at the final time of the migration.

Since DMS is connected inside a VPC (private network on your AWS account), it was necessary to set up a Site-to-Site VPN connectivity between GCP and AWS so that DMS could connect to the source database in order to start the data replication to the target RDS database on AWS.

We then fumbled and made several attempts to choose the right settings adapted to the type of SQL data to be transferred. Amazon Data Migration Service offers to configure the transfer in a very fine-grained way (chunking, parallelization, etc.).

Once the combination of settings was found, we waited about 30 hours to migrate 1 TB of data (the entire source database).

This quantity being transferred, DMS switched to continuous replication mode until we made the final switch to AWS when RDS became the production database.

Thanks to Amazon DMS, the downtime during the switch to AWS was reduced to a minimum (around 1 hour), the time to catch up on the delay between the two databases and to make the DNS switch.

Technical focus: Ephemeral environments with ArgoCD

With Colizey on AWS, we can take advantage of the automation potential of the cloud to speed up the work of developers, making their lives easier.

To allow Colizey developers to test their changes in an environment as close as possible to production, a solution that we see emerging more and more: ephemeral development environments.

When a developer pushes a new branch of code (containing for example a bugfix, or a new feature), an automated process will create an instance of the application in a dedicated environment and allow the developer to access it, for example with a unique URL.

To do this, we used a very useful feature of ArgoCD: the ApplicationSet. An ApplicationSet allows you to dynamically deploy applications in Kubernetes based on external events. We therefore used its Pull Request Generator capability to create an instance of the Colizey application for each Pull Request (code change request) opened by developers on GitHub.

ArgoCD therefore deploys an instance of the application corresponding to the branch pushed by the developer, and makes it accessible on the internet, on preview.example.com for example. For constraints relating to DNS and TLS certificates, we did not expose each environment on a unique URL (e.g. feat-new-button-123abcd.preview.example.com) but on a unique URL for each environment, accessible throug a header-based routing.

Depending on the value of the Environment header, the developer was routed to the ephemeral environment of his choice (see diagram below).

To reduce the costs of this feature by about 25%, we set up a recurring job that turned off all ephemeral environments (by disabling the Cluster Autoscaler). This also reduces the carbon footprint of the feature!

Conclusion

Last June, Theodo Cloud supported Colizey in its migration to AWS in 5 weeks. From containerization to automation of production deployments, including ephemeral preview environments, this project allowed Colizey to reduce the time of production deployments by 75% (5 min vs. 20 min) while reducing the risk of error thanks to their automation. On the AWS cloud, Colizey has increased availability and resilience, being deployed in several distinct zones.

We help dozens of companies like Colizey to successfully migrate to the Cloud and to containerization. Why not you?
Do not hesitate to contact us!