loading...

Kubernetes – Advanced services

Let’s explore the IP strategy as it relates to services and communication between containers. If you recall, in the Services section of  Chapter 2, Pods, Services, Replication Controllers, and Labels, you learned that Kubernetes is using kube-proxy to determine the proper pod IP address and port serving each request. Behind the scenes, kube-proxy is actually using virtual IPs and iptables to make all this magic work.

kube-proxy now has two modes—userspace and iptables. As of now, 1.2 iptables is the default mode. In both modes, kube-proxy is running on every host. Its first duty is to monitor the API from the Kubernetes master. Any updates to services will trigger an update to iptables from kube-proxy. For example, when a new service is created, a virtual IP address is chosen and a rule in iptables is set, which will direct its traffic to kube-proxy via a random port. Thus, we now have a way to capture service-destined traffic on this node. Since kube-proxy is running on all nodes, we have cluster-wide resolution for the service VIP (short for virtual IP). Additionally, DNS records can point to this VIP as well.

In the userspace mode, we have a hook created in iptables, but the proxying of traffic is still handled by kube-proxy. The iptables rule is only sending traffic to the service entry in kube-proxy at this point. Once kube-proxy receives the traffic for a particular service, it must then forward it to a pod in the service’s pool of candidates. It does this using a random port that was selected during service creation.

Refer to the following diagram for an overview of the flow:

Kube-proxy communication
It is also possible to always forward traffic from the same client IP to the same backend pod/container using the sessionAffinity element in your service definition.

In the iptables mode, the pods are coded directly in the iptable rules. This removes the dependency on kube-proxy for actually proxying the traffic. The request will go straight to iptables and then on to the pod. This is faster and removes a possible point of failure. Readiness probe, as we discussed in the Health Check section of  Chapter 2, Pods, Services, Replication Controllers, and Labels, is your friend here as this mode also loses the ability to retry pods.

External services

In the previous chapter, we saw a few service examples. For testing and demonstration purposes, we wanted all the services to be externally accessible. This was configured by the type: LoadBalancer element in our service definition. The LoadBalancer type creates an external load balancer on the cloud provider. We should note that support for external load balancers varies by provider, as does the implementation. In our case, we are using GCE, so integration is pretty smooth. The only additional setup needed is to open firewall rules for the external service ports.

Let’s dig a little deeper and do a describe command on one of the services from the More on labels section in Chapter 2, Pods, Services, Replication Controllers, and Labels:

$ kubectl describe service/node-js-labels

The following screenshot is the result of the preceding command:

Service description

In the output of the preceding screenshot, you’ll note several key elements. Our Namespace: is set to default, the Type: is LoadBalancer, and we have the external IP listed under LoadBalancer Ingress:. Furthermore, we can see Endpoints:, which shows us the IPs of the pods that are available to answer service requests.

Internal services

Let’s explore the other types of services that we can deploy. First, by default, services are only internally facing. You can specify a type of clusterIP to achieve this, but, if no type is defined, clusterIP is the assumed type. Let’s take a look at an example, nodejs-service-internal.yaml; note the lack of the type element:

apiVersion: v1 
kind: Service 
metadata: 
  name: node-js-internal 
  labels: 
    name: node-js-internal 
spec: 
  ports: 
  - port: 80 
  selector: 
    name: node-js 

Use this listing to create the service definition file. You’ll need a healthy version of the node-js RC (Listing nodejs-health-controller-2.yaml). As you can see, the selector matches on the pods named node-js that our RC launched in the previous chapter. We will create the service and then list the currently running services with a filter as follows:

$ kubectl create -f nodejs-service-internal.yaml
$ kubectl get services -l name=node-js-internal

The following screenshot is the result of the preceding command:

Internal service listing

As you can see, we have a new service, but only one IP. Furthermore, the IP address is not externally accessible. We won’t be able to test the service from a web browser this time. However, we can use the handy kubectl exec command and attempt to connect from one of the other pods. You will need node-js-pod (nodejs-pod.yaml) running. Then, you can execute the following command:

$ kubectl exec node-js-pod -- curl <node-js-internal IP>

This allows us to run a docker exec command as if we had a shell in the node-js-pod container. It then hits the internal service URL, which forwards to any pods with the node-js label.

If all is well, you should get the raw HTML output back. You have successfully created an internal-only service. This can be useful for backend services that you want to make available to other containers running in your cluster, but not open to the world at large.

Custom load balancing

A third type of service that K8s allows is the NodePort type. This type allows us to expose a service through the host or node (minion) on a specific port. In this way, we can use the IP address of any node (minion) and access our service on the assigned node port. Kubernetes will assign a node port by default in the range of 300032767, but you can also specify your own custom port. In the example in the following listing nodejs-service-nodeport.yaml, we choose port 30001, as follows:

apiVersion: v1 
kind: Service 
metadata: 
  name: node-js-nodeport 
  labels: 
    name: node-js-nodeport 
spec: 
  ports: 
  - port: 80 
    nodePort: 30001 
  selector: 
    name: node-js 
  type: NodePort 

Once again, create this YAML definition file and create your service, as follows:

$ kubectl create -f nodejs-service-nodeport.yaml

The output should have a message like this:

New GCP firewall rule

Note message about opening firewall ports. Similar to the external load balancer type, NodePort is exposing your service externally using ports on the nodes. This could be useful if, for example, you want to use your own load balancer in front of the nodes. Let’s make sure that we open those ports on GCP before we test our new service.

From the GCE VM instance console, click on the details for any of your nodes (minions). Then, click on the network, which is usually the default unless otherwise specified during creation. In  Firewall rules, we can add a rule by clicking on  Add firewall rule.

Create a rule like the one shown in the following screenshot (tcp:30001 on the 0.0.0.0/0 IP range):

Create a new firewall rule page

We can now test our new service by opening a browser and using an IP address of any node (minion) in your cluster. The format to test the new service is as follows:

http://<Minoion IP Address>:<NodePort>/

Finally, the latest version has added an ExternalName type, which maps a CNAME to the service. 

Cross-node proxy

Remember that kube-proxy is running on all the nodes, so even if the pod is not running there, the traffic will be given a proxy to the appropriate host. Refer to the Cross-node traffic screenshot for a visual on how the traffic flows. A user makes a request to an external IP or URL. The request is serviced by Node in this case. However, the pod does not happen to run on this node. This is not a problem because the pod IP addresses are routable. So, kube-proxy or iptables simply passes traffic onto the pod IP for this service. The network routing then completes on Node 2, where the requested application lives:

Cross-node traffic

Custom ports

Services also allow you to map your traffic to different ports; then, the containers and pods expose themselves. We will create a service that exposes port 90 and forwards traffic to port 80 on the pods. We will call the node-js-90 pod to reflect the custom port number. Create the following two definition files, nodejs-customPort-controller.yaml and nodejs-customPort-service.yaml:

apiVersion: v1 
kind: ReplicationController 
metadata: 
  name: node-js-90 
  labels: 
    name: node-js-90 
spec: 
  replicas: 3 
  selector: 
    name: node-js-90 
  template: 
    metadata: 
      labels: 
        name: node-js-90 
    spec: 
      containers: 
      - name: node-js-90 
        image: jonbaier/node-express-info:latest 
        ports: 
        - containerPort: 80 
apiVersion: v1 
kind: Service 
metadata: 
  name: node-js-90 
  labels: 
    name: node-js-90 
spec: 
  type: LoadBalancer 
  ports: 
  - port: 90 
    targetPort: 80 
  selector: 
    name: node-js-90
If you are using the free trial for Google Cloud Platform, you may have issues with the LoadBalancer type services. This type creates multiple external IP addresses, but trial accounts are limited to only one static address.

You’ll note that in the service definition, we have a targetPort element. This element tells the service the port to use for pods/containers in the pool. As we saw in previous examples, if you do not specify targetPort, it assumes that it’s the same port as the service. This port is still used as the service port, but, in this case, we are going to expose the service on port 90 while the containers serve content on port 80.

Create this RC and service and open the appropriate firewall rules, as we did in the last example. It may take a moment for the external load balancer IP to propagate to the get service command. Once it does, you should be able to open and see our familiar web application in a browser using the following format:

http://<external service IP>:90/

Multiple ports

Another custom port use case is that of multiple ports. Many applications expose multiple ports, such as HTTP on port 80 and port 8888 for web servers. The following example shows our app responding on both ports. Once again, we’ll also need to add a firewall rule for this port, as we did for the list nodejs-service-nodeport.yaml previously. Save the listing as nodejs-multi-controller.yaml and nodejs-multi-service.yaml:

apiVersion: v1 
kind: ReplicationController 
metadata: 
  name: node-js-multi 
  labels: 
    name: node-js-multi 
spec: 
  replicas: 3 
  selector: 
    name: node-js-multi 
  template: 
    metadata: 
      labels: 
        name: node-js-multi 
    spec:
      containers: 
      - name: node-js-multi 
        image: jonbaier/node-express-multi:latest 
        ports: 
        - containerPort: 80 
        - containerPort: 8888
apiVersion: v1 
kind: Service 
metadata: 
  name: node-js-multi 
  labels: 
    name: node-js-multi 
spec: 
  type: LoadBalancer 
  ports: 
  - name: http 
    protocol: TCP 
    port: 80 
  - name: fake-admin-http 
    protocol: TCP 
    port: 8888 
  selector: 
    name: node-js-multi 
The application and container itself must be listening on both ports for this to work. In this example, port 8888 is used to represent a fake admin interface. If, for example, you want to listen on port 443, you would need a proper SSL socket listening on the server.

Ingress

We previously discussed how Kubernetes uses the service abstract as a means to proxy traffic to a backing pod that’s distributed throughout our cluster. While this is helpful in both scaling and pod recovery, there are more advanced routing scenarios that are not addressed by this design.

To that end, Kubernetes has added an ingress resource, which allows for custom proxying and load balancing to a back service. Think of it as an extra layer or hop in the routing path before traffic hits our service. Just as an application has a service and backing pods, the ingress resource needs both an Ingress entry point and an ingress controller that perform the custom logic. The entry point defines the routes and the controller actually handles the routing. This is helpful for picking up traffic that would normally be dropped by an edge router or forwarded elsewhere outside of the cluster.

Ingress itself can be configured to offer externally addressable URLs for internal services, to terminate SSL, offer name-based virtual hosting as you’d see in a traditional web server, or load balance traffic. Ingress on its own cannot service requests, but requires an additional ingress controller to fulfill the capabilities outlined in the object. You’ll see nginx and other load balancing or proxying technology involved as part of the controller framework.  In the following examples, we’ll be using GCE, but you’ll need to deploy a controller yourself in order to take advantage of this feature. A popular option at the moment is the nginx-based ingress-nginx controller.

You can check it out here: https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#glbc-beta-limitations.

An ingress controller is deployed as a pod which runs a daemon. This pod watches the Kubernetes apiserver/ingresses endpoint for changes to the ingress resource. For our examples, we will use the default GCE backend.

Types of ingress

There are a couple different types of ingress, such as the following:

  • Single service ingress: This strategy exposes a single service via creating an ingress with a default backend that has no rules. You can alternatively use Service.Type=LoadBalancer or Service.Type=NodePort, or a port proxy to accomplish something similar.
  • Fanout: Given that od IP addressing is only available internally to the Kubernetes network, you’ll need to use a simple fanout strategy in order to accommodate edge traffic and provide ingress to the correct endpoints in your cluster. This will resemble a load balancer in practice.
  • Name-based hosting: This approach is similar to service name indication (SNI), which allows a web server to present multiple HTTPS websites with different certificates on the same TCP port and IP address.

Kubernetes uses host headers to route requests with this approach. The following example snippet ingress-example.yaml shows what name-based virtual hosting would look like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: name-based-hosting
spec:
  rules:
  - host: example01.foo.com
    http:
      paths:
      - backend:
          serviceName: sevice01
          servicePort: 8080
  - host: example02.foo.com
    http:
      paths:
      - backend:
          serviceName: sevice02
          servicePort: 8080

As you may recall, in Chapter 1, Introduction to Kubernetes, we saw that a GCE cluster comes with a default back which provides Layer 7 load balancing capability. We can see this controller running if we look at the kube-system namespace:

$ kubectl get rc --namespace=kube-system

We should see an RC listed with the l7-default-backend-v1.0 name, as shown here:

GCE Layer 7 Ingress controller

This provides the ingress controller piece that actually routes the traffic defined in our ingress entry points. Let’s create some resources for an Ingress.

First, we will create a few new replication controllers with the httpwhalesay image. This is a remix of the original whalesay that was displayed in a browser. The following listing, whale-rcs.yaml, shows the YAML. Note the three dashes that let us combine several resources into one YAML file:

apiVersion: v1
kind: ReplicationController
metadata:
  name: whale-ingress-a
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: whale-ingress-a
    spec:
      containers:
      - name: sayhey
        image: jonbaier/httpwhalesay:0.1
        command: ["node", "index.js", "Whale Type A, Here."]
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: whale-ingress-b
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: whale-ingress-b
    spec:
      containers:
      - name: sayhey
        image: jonbaier/httpwhalesay:0.1
        command: ["node", "index.js", "Hey man, It's Whale B, Just
        Chillin'."]
        ports:
        - containerPort: 80

Note that we are creating pods with the same container, but different startup parameters. Take note of these parameters for later. We will also create Service endpoints for each of these RCs as shown in the whale-svcs.yaml listing:

apiVersion: v1
kind: Service
metadata:
  name: whale-svc-a
  labels:
    app: whale-ingress-a
spec:
  type: NodePort
  ports:
  - port: 80
    nodePort: 30301
    protocol: TCP
    name: http
  selector:
    app: whale-ingress-a
---
apiVersion: v1
kind: Service
metadata:
  name: whale-svc-b
  labels:
    app: whale-ingress-b
spec:
  type: NodePort
  ports:
  - port: 80
    nodePort: 30284
    protocol: TCP
    name: http
  selector:
    app: whale-ingress-b
---
apiVersion: v1
kind: Service
metadata:
 name: whale-svc-default
 labels:
   app: whale-ingress-a
spec:
  type: NodePort
  ports:
  - port: 80
    nodePort: 30302
    protocol: TCP
    name: http
  selector:
    app: whale-ingress-a

Again, create these with the kubectl create -f command, as follows:

$ kubectl create -f whale-rcs.yaml
$ kubectl create -f whale-svcs.yaml

We should see messages about the successful creation of the RCs and Services. Next, we need to define the Ingress entry point. We will use http://a.whale.hey and http://b.whale.hey as our demo entry points as shown in the following listing whale-ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: whale-ingress
spec:
  rules:
  - host: a.whale.hey
    http:
      paths:
      - path: /
        backend:
          serviceName: whale-svc-a
          servicePort: 80
  - host: b.whale.hey
    http:
      paths:
      - path: /
        backend:
          serviceName: whale-svc-b
          servicePort: 80

Again, use kubectl create -f to create this ingress. Once this is successfully created, we will need to wait a few moments for GCE to give the ingress a static IP address. Use the following command to watch the Ingress resource:

$ kubectl get ingress

Once the Ingress has an IP, we should see an entry in ADDRESS, like the one shown here:

Ingress description

Since this is not a registered domain name, we will need to specify the resolution in the curl command, like this:

$ curl --resolve a.whale.hey:80:130.211.24.177 http://a.whale.hey/

This should display the following:

Whalesay A

We can also try the second URL. Doing this, we will get our second RC:

$ curl --resolve b.whale.hey:80:130.211.24.177 http://b.whale.hey/
Whalesay B

Note that the images are almost the same, except that the words from each whale reflect the startup parameters from each RC we started earlier. Thus, our two Ingress points are directing traffic to different backends.

In this example, we used the default GCE backend for an Ingress controller. Kubernetes allows us to build our own, and nginx actually has a few versions available as well.

Migrations, multicluster, and more

As we’ve already seen so far, Kubernetes offers a high level of flexibility and customization to create a service abstraction around your containers running in the cluster. However, there may be times where you want to point to something outside your cluster.

An example of this would be working with legacy systems or even applications running on another cluster. In the case of the former, this is a perfectly good strategy in order to migrate to Kubernetes and containers in general. We can begin by managing the service endpoints in Kubernetes while stitching the stack together using the K8s orchestration concepts. Additionally, we can even start bringing over pieces of the stack, as the frontend, one at a time as the organization refactors applications for microservices and/or containerization.

To allow access to non pod-based applications, the services construct allows you to use endpoints that are outside the cluster. Kubernetes is actually creating an endpoint resource every time you create a service that uses selectors. The endpoints object keeps track of the pod IPs in the load balancing pool. You can see this by running the get endpoints command, as follows:

$ kubectl get endpoints

You should see something similar to the following:

NAME               ENDPOINTS
http-pd            10.244.2.29:80,10.244.2.30:80,10.244.3.16:80
kubernetes         10.240.0.2:443
node-js            10.244.0.12:80,10.244.2.24:80,10.244.3.13:80

You’ll note the entry for all the services we currently have running on our cluster. For most services, the endpoints are just the IP of each pod running in an RC. As I mentioned previously, Kubernetes does this automatically based on the selector. As we scale the replicas in a controller with matching labels, Kubernetes will update the endpoints automatically.

If we want to create a service for something that is not a pod and therefore has no labels to select, we can easily do this with both a service definition nodejs-custom-service.yaml and endpoint definition nodejs-custom-endpoint.yaml, as follows:

apiVersion: v1 
kind: Service 
metadata: 
  name: custom-service 
spec: 
  type: LoadBalancer 
  ports: 
  - name: http 
    protocol: TCP 
    port: 80

apiVersion: v1 
kind: Endpoints 
metadata: 
  name: custom-service 
subsets: 
- addresses: 
  - ip: <X.X.X.X> 
  ports: 
    - name: http 
      port: 80 
      protocol: TCP 

In the preceding example, you’ll need to replace <X.X.X.X> with a real IP address, where the new service can point to. In my case, I used the public load balancer IP from the node-js-multi service we created earlier in listing ingress-example.yaml. Go ahead and create these resources now.

If we now run a get endpoints command, we will see this IP address at port 80, which is associated with the custom-service endpoint. Furthermore, if we look at the service details, we will see the IP listed in the Endpoints section:

$ kubectl describe service/custom-service

We can test out this new service by opening the custom-service external IP from a browser.

Custom addressing

Another option to customize services is with the clusterIP element. In our examples so far, we’ve not specified an IP address, which means that it chooses the internal address of the service for us. However, we can add this element and choose the IP address in advance with something like clusterip: 10.0.125.105.

There may be times when you don’t want to load balance and would rather have DNS with A records for each pod. For example, software that needs to replicate data evenly to all nodes may rely on A records to distribute data. In this case, we can use an example like the following one and set clusterip to None.

Kubernetes will not assign an IP address and instead only assign A records in DNS for each of the pods. If you are using DNS, the service should be available at node-js-none or node-js-none.default.cluster.local from within the cluster. For this, we  will use the following listing nodejs-headless-service.yaml:

apiVersion: v1 
kind: Service 
metadata: 
  name: node-js-none 
  labels: 
    name: node-js-none 
spec: 
  clusterIP: None 
  ports: 
  - port: 80 
  selector: 
    name: node-js 

Test it out after you create this service with the trusty exec command:

$ kubectl exec node-js-pod -- curl node-js-none

Comments are closed.

loading...