Multi-regional Infrastructure deployment on Kubernetes with Docker, Jenkins, and GitHub

I am working for a tech company for the last 2 years and our team goal was to make our whole infrastructure handle a minimum of 200–400 billion requests per month across multiple countries and multiple data centers. I’m learning a lot about how to handle this insane amount of requests with smart and painless deployment. This is a blog about how you should handle multi-regional deployment with minimal complexity. I won’t go into how to set up things from scratch in this blog.

Problem:

First Lets discuss the situation here. Say, we have 6 micro-services and every microservice is stateless and it scales infinitely but Not all regions need all the services and also you need to control the deployment of services to specific regions for specific microservices. There is also a twist for every Kubernetes cluster you have to maintain multiple namespaces for like testing, staging, and production. They all have different configurations how will you handle that? and the last problem is while testing on namespaces other than production or running JMeter application to test you shouldn’t mess up with production allocated CPU/RAM.

Manage multiple environment configurations in Github: You can create a repository to store multiple microservice environments from here. Every microservice config will be a file and every namespace can be a config file name.

Building Docker image by pulling configuration from GitHub raw on the fly: When you’re building a docker image for a particular microservice on Jenkins you can pass for which namespace you’re building the image by passing argument using the ARG feature of docker. While pulling the configuration you can use single line fetch using GitHub raw. For example, let's say we have a Dockerfile like this.

You need to change few things like ARG, install curl to fetch the required config and save it as .env and work with it. You can build your docker image by passing an argument on shell script like this:-

docker build -t org/service:$BRANCH_NAME . --build-arg BRANCH_NAME=$BRANCH_NAME

Now you have a docker container ready to ship to all the Kubernetes clusters available for a particular namespace.

Push new images and replace previous ones in dockerhub.io: You can push to specific tags every time if you don’t need the previous one.

Ping to Kubernetes clusters: It’s like shouting to everyone in the room “Hey! Food is ready!”. So when you ping Kubernetes clusters using this command by using configuration from Github. You need to store different cluster access files to another repository which helps you to access the clusters that you want to ping.

kubectl --kubeconfig="cluser-key.yaml" -n staging rollout restart deployment.apps/service-name

You can write a Jenkins file for every service and specify which region you want to push this service into. Your Jenkins file should look like this.

For controlling resources on a cluster level you should use Kubernetes in case you put too much load while testing using the ResourceQuota feature from Kubernetes which helps to limit resources for a particular namespace.

And you’re done! You can use ansible in a deployment process and there are more complex and cooler ways you can deploy on a multi-regional level which we’re working on it but let’s discuss in another blog.