Kubernetes – Deployments

In the previous chapter, we explored some of the core concepts for application updates using the old rolling-update method. Starting with version 1.2, Kubernetes added the Deployment construct, which improves on the basic mechanisms of rolling-update and ReplicationControllers. As the name suggests, it gives us finer control over the code deployment itself. Deployments allow us to pause and resume application rollouts via declarative definitions and updates to pods and ReplicaSets. Additionally, they keep a history of past deployments and allow the user to easily roll back to previous versions.

It is no longer recommended to use ReplicationControllers. Instead, use a Deployment that configures a ReplicaSet in order to set up application availability for your stateless services or applications. Furthermore, do not directly manage the ReplicaSets that are created by your deployments; only do so through the Deployment API.

Deployment use cases

We’ll explore a number of typical scenarios for deployments in more detail:

  • Roll out a ReplicaSet
  • Update the state of a set of Pods
  • Roll back to an earlier version of a Deployment
  • Scale up to accommodate cluster load
  • Pause and use Deployment status in order to make changes or indicate a stuck deployment
  • Clean up a deployment

In the following code of the node-js-deploy.yaml file, we can see that the definition is very similar to a ReplicationController. The main difference is that we now have an ability to make changes and updates to the deployment objects and let Kubernetes manage updating the underlying pods and replicas for us:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: node-js-deploy
  labels:
    name: node-js-deploy
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: node-js-deploy
    spec:
      containers:
      - name: node-js-deploy
        image: jonbaier/pod-scaling:0.1
        ports:
        - containerPort: 80

In this example, we’ve created a Deployment named node-js-deploy via the name field under metadata. We’re creating a single pod that will be managed by the selector field, which is going to help the Deployment understand which pods to manage. The spec tells the pod to run the jobbaier/pod-scaling container and directs traffic through port 80 via the containerPort.

We can run the familiar create command with the optional --record flag so that the creation of the Deployment is recorded in the rollout history. Otherwise, we will only see subsequent changes in the rollout history using the $ kubectl create -f node-js-deploy.yaml --record command.

You may need to add --validate=false if this beta type is not enabled on your cluster.

We should see a message about the deployment being successfully created. After a few moments, it will finish creating our pod, which we can check for ourselves with a get pods command. We add the -l flag to only see the pods relevant to this deployment:

$ kubectl get pods -l name=node-js-deploy

If you’d like to get the state of the deployment, you can issue the following:

$ kubectl get deployments

You can also see the state of a rollout, which will be more useful in the future when we update our Deployments. You can use kubectl rollout status deployment/node-js-deploy to see what’s going on.

We create a service just as we did with ReplicationControllers. The following is a Service definition for the Deployment we just created. Notice that it is almost identical to the Services we created in the past. Save the following code in node-js-deploy-service.yaml file:

apiVersion: v1
kind: Service
metadata:
  name: node-js-deploy
  labels:
    name: node-js-deploy
spec:
  type: LoadBalancer
  ports:
  - port: 80
  sessionAffinity: ClientIP
  selector:
    name: node-js-deploy

Once this service is created using kubectl, you’ll be able to access the deployment pods through the service IP or the service name if you are inside a pod on this namespace. 

Scaling

The scale command works the same way as it did in our ReplicationController. To scale up, we simply use the deployment name and specify the new number of replicas, as shown here:

$ kubectl scale deployment node-js-deploy --replicas 3

If all goes well, we’ll simply see a message about the deployment being scaled in the output of our Terminal window. We can check the number of running pods using the get pods command from earlier. In the latest versions of Kubernetes, you’re also able to set up pod scaling for your cluster, which allows you to do horizontal autoscaling so you can scale up pods based on the CPU utilization of your cluster. You’ll need to set a maximum and minimum number of pods in order to get this going.

Here’s what that command would look like with this example:

$ kubectl autoscale deployment node-js-deploy --min=25 --max=30 --cpu-percent=75
deployment "node-js-deploy" autoscaled
Read more about horizontal pod scaling in this walkthrough: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/.

There’s also a concept of proportional scaling, which allows you to run multiple version of your application at the same time. This implementation would be useful when incrementing a backward-compatible version of an API-based microservice, for example. When doing this type of deployment, you’ll use .spec.strategy.rollingUpdate.maxUnavailable and .spec.strategy.rollingUpdate.maxSurge to limit the maximum number of pods that can be down during an update to the deployment, or the maximum number of pods that can be created that exceed the desired number of pods, respectively.

Updates and rollouts

Deployments allow for updating in a few different ways. First, there is thkubectl set command, which allows us to change the deployment configuration without redeploying manually. Currently, it only allows for updating the image, but as new versions of our application or container image are processed, we will need to do this quite often. 

Let’s take a look using our deployment from the previous section. We should have three replicas running right now. Verify this by running the get pods command with a filter for our deployment:

$ kubectl get pods -l name=node-js-deploy

We should see three pods similar to those listed in the following screenshot:

Deployment pod listing

Take one of the pods listed on our setup, replace it in the following command where it says {POD_NAME_FROM_YOUR_LISTING}, and run this command:

$ kubectl describe pod/{POD_NAME_FROM_YOUR_LISTING} | grep Image:

We should see an output like the following screenshot with the current image version of 0.1

Current pod image

Now that we know what our current deployment is running, let’s try to update to the next version. This can be achieved easily using thkubectl set command and specifying the new version, as shown here:

$ kubectl set image deployment/node-js-deploy node-js-deploy=jonbaier/pod-scaling:0.2
$ deployment "node-js-deploy" image updated

If all goes well, we should see text that says deployment "node-js-deploy" image updated displayed on the screen.

We can double–check the status using the following rollout status command:

$ kubectl rollout status deployment/node-js-deploy

Alternatively, we can directly edit the deployment in an editor window with kubectl edit deployment/node-js-deploy and change .spec.template.spec.containers[0].image from jonbaier/pod-scaling:0.1 to jonbaier/pod-scaling:0.2. Either of these methods will work to update your deployment, and as a reminder you can check the status of your update with the kubectl status command:

$ kubectl rollout status deployment/node-js-deployment
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
deployment "node-js-deployment" successfully rolled out

We should see some text saying that the deployment successfully rolled out. If you see any text about waiting for the rollout to finish, you may need to wait a moment for it to finish, or alternatively check the logs for issues.

Once it’s finished, run the get pods command as earlier. This time, we will see new pods listed:

Deployment pod listing after update

Once again, plug one of your pod names into the describe command we ran earlier. This time, we should see the image has been updated to 0.2.

What happened behind the scenes is that Kubernetes has rolled out a new version for us. It basically creates a new ReplicaSet with the new version. Once this pod is online and healthy, it kills one of the older versions. It continues this behavior, scaling out the new version and scaling down the old versions, until only the new pods are left. Another way to observe this behavior indirectly is to investigate the ReplicaSet that the Deployment object is using to update your desired application state.

Remember, you don’t interact directly with ReplicaSet, but rather give Kubernetes directives in the form of Deployment elements and let Kubernetes make the required changes to the cluster object store and state. Take a look at the ReplicaSets quickly after running your image update command, and you’ll see how multiple ReplicaSets are used to effect the image change without application downtime:

$ kubectl get rs
NAME                               DESIRED CURRENT READY AGE
node-js-deploy-1556879905          3       3       3     46s
node-js-deploy-4657899444          0       0       0     85s

The following diagram describes the workflow for your reference:

Deployment life cycle

It’s worth noting that the rollback definition allows us to control the pod replace method in our deployment definition. There is a strategy.type field that defaults to RollingUpdate and the preceding behavior. Optionally, we can also specify Recreate as the replacement strategy and it will kill all the old pods first before creating the new versions.

History and rollbacks

One of the useful features of the rollout API is the ability to track the deployment history. Let’s do one more update before we check the history. Run the kubectl set command once more and specify version 0.3:

$ kubectl set image deployment/node-js-deploy node-js-deploy=jonbaier/pod-scaling:0.3
$ deployment "node-js-deploy" image updated

Once again, we’ll see text that says deployment "node-js-deploy" image updated displayed on the screen. Now, run the get pods command once more:

$ kubectl get pods -l name=node-js-deploy

Let’s also take a look at our deployment history. Run the rollout history command:

$ kubectl rollout history deployment/node-js-deploy

We should see an output similar to the following:

Rollout history

As we can see, the history shows us the initial deployment creation, our first update to 0.2, and then our final update to 0.3. In addition to status and history, throllout command also supports the pause, resume, and undo sub-commands. The rollout pause command allows us to pause a command while the rollout is still in progress. This can be useful for troubleshooting and also helpful for canary-type launches, where we wish to do final testing of the new version before rolling out to the entire user base. When we are ready to continue the rollout, we can simply use the rollout resume command. 

But what if something goes wrong? That is where the rollout undo command and the rollout history itself are really handy. Let’s simulate this by trying to update to a version of our pod that is not yet available. We will set the image to version 42.0, which does not exist:

$ kubectl set image deployment/node-js-deploy node-js-deploy=jonbaier/pod-scaling:42.0

We should still see the text that says deployment "node-js-deploy" image updated displayed on the screen. But if we check the status, we will see that it is still waiting:

$ kubectl rollout status deployment/node-js-deploy
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...

Here, we see that the deployment has been paused after updating two of the three pods, but Kubernetes knows enough to stop there in order to prevent the entire application from going offline due to the mistake in the container image name. We can press Ctrl + C to kill the status command and then run the get pods command once more:

$ kubectl get pods -l name=node-js-deploy

We should now see an ErrImagePull, as in the following screenshot:

Image pull error

As we expected, it can’t pull the 42.0 version of the image because it doesn’t exist. This error refers to a container that’s stuck in an image pull loop, which is noted as ImagePullBackoff in the latest versions of Kubernetes. We may also have issues with deployments if we run out of resources on the cluster or hit limits that are set for our namespace. Additionally, the deployment can fail for a number of application-related causes, such as health check failure, permission issues, and application bugs, of course.

It’s entirely possible to create deployments that are wholly unavailable if you don’t change maxUnavailable and spec.replicas to different numbers, as the default for each is 1!

Whenever a failure to roll out happens, we can easily roll back to a previous version using the rollout undo command. This command will take our deployment back to the previous version:

$ kubectl rollout undo deployment/node-js-deploy

After that, we can run a rollout status command once more and we should see everything rolled out successfully. Run the kubectl rollout history deployment/node-js-deploy command again and we’ll see both our attempt to roll out version 42.0 and revert to 0.3:

Rollout history after rollback
We can also specify the --to-revision flag when running an undo to roll back to a specific version. This can be handy for times when our rollout succeeds, but we discover logical errors down the road.

Autoscaling

As you can see, Deployments are a great improvement over ReplicationControllers, allowing us to seamlessly update our applications, while integrating with the other resources of Kubernetes in much the same way.

Another area that we saw in the previous chapter, and also supported for Deployments, is Horizontal Pod Autoscalers (HPAs). HPAs help you manage cluster utilization by scaling the number of  pods based on CPU utilization. There are three objects that can scale using HPAs, DaemonSets not included:

  • Deployment (the recommended method)
  • ReplicaSet
  • ReplicationController (not recommended)

The HPA is implemented as a control loop similar to other controllers that we’ve discussed, and you can adjust the sensitivity of the controller manager by adjusting its sync period via --horizontal-pod-autoscaler-sync-period (default 30 seconds).

We will walk through a quick remake of the HPAs from the previous chapter, this time using the Deployments we have created so far. Save the following code in node-js-deploy-hpa.yaml file: 

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: node-js-deploy
spec:
  minReplicas: 3
  maxReplicas: 6
  scaleTargetRef:
    apiVersion: v1
    kind: Deployment
    name: node-js-deploy
  targetCPUUtilizationPercentage: 10
The API is changing quickly with these tools as they’re in beta, so take careful note of the apiVersion element, which used to be autoscaling/v1, but is now autoscalingv2beta1.

We have lowered the CPU threshold to 10% and changed our minimum and maximum pods to 3 and 6, respectively. Create the preceding HPA with our trusty kubectl create -f command. After this is completed, we can check that it’s available with the kubectl get hpa command:

Horizontal pod autoscaler 

We can also check that we have only 3 pods running with the kubectl get deploy command. Now, let’s add some load to trigger the autoscaler:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: boomload-deploy
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: loadgenerator-deploy
    spec:
      containers:
      - image: williamyeh/boom
        name: boom-deploy
        command: ["/bin/sh","-c"]
        args: ["while true ; do boom http://node-js-deploy/ -c 100 -n 500 ; sleep 1 ; done"]

Create boomload-deploy.yaml file as usual. Now, monitor the HPA with the alternating kubectl get hpa and kubectl get deploy commands. After a few moments, we should see the load jump above 10%. After a few more moments, we should also see the number of pods increase all the way up to 6 replicas:

HPA increase and pod scale up

Again, we can clean this up by removing our load generation pod and waiting a few moments:

$ kubectl delete deploy boomload-deploy

Again, if we watch the HPA, we’ll start to see the CPU usage drop. After a few minutes, we will go back down to 0% CPU load and then the Deployment will scale back to 3 replicas.

Comments are closed.