loading...

Kubernetes – Why federation?

There are several major advantages to taking on Kubernetes cluster federation. As mentioned previously, federation allows you increase the availability and tenancy capabilities of your Kubernetes clusters. By scaling across availability zones or regions of a single cloud service provider (CSP), or by scaling across multiple CSPs, federation takes the concept of high availability to the next level. Some term this global scheduling, which will could enable you to direct traffic in order to maximize an inexpensive CSP resource that becomes available in the spot market. You could also use global scheduling to relocate workloads cluster to end use populations, improving the performance of your applications.

There is also the opportunity to treat entire clusters as if they were Kubernetes objects, and deal with failure on a per-cluster basis instead of per machine. Cluster federation could allow operators to automatically recover from entire clusters failing by routing traffic to redundant, available clusters.

It should be noted that, while federation increases the potential for high availability on your cluster, it’s clear that the significant increase in complexity also lowers your potential reliability if your clusters aren’t managed well. You can manage some of this complexity by using a hosted PaaS version of Kubernetes such as GKE, where leaving the cluster management to GCP will drastically lower the operational load on your teams.

Federation can also enable your team to support a hybrid environment, with on-premises clusters pairing with your resources in the cloud. Depending on your traffic routing requirements, this may require additional engineering in the form of a service mesh.

There’s a number of technical features that federation supplies, which enable higher potential availability.

The building blocks of federation

Federation makes it easy to manage resources across clusters by providing two distinct types of building blocks. The first is resources and the second is service discovery:

  • Resource synchronization across clusters: Federation is the glue that allows you to keep track of the many resources needed to run sets of applications. When you’re running a lot of applications, with many resources and object types, across many clusters, federation is key to keeping your clusters organized and managed well. You may find yourself needing to keep an application deployment running in multiple clusters with a single pane of glass view.
  • Multi-cluster service discovery: There are a number of resources that share well between clusters such as DNS, load balancers, object storage, and ingress. Federation gives you the ability to automatically configure those services with multi-cluster awareness, so you can route application traffic and manage the control plane across several clusters.

As we’ll learn next, Kubernetes federation is managed by a tool named kubefed, which has a number of command-line flags that allow you to manage many clusters and the building blocks we discussed previously. The major building blocks of kubefed that we’ll use are as follows:

  • kubefed init: Initialize a federation control plane
  • kubefed join: Join a cluster to a federation
  • kubefed options: Print the list of flags inherited by all commands
  • kubefed unjoin: Unjoin a cluster from a federation
  • kubefed version: Print the client and server version information

Here’s a handy list of the options that can be used:

      --alsologtostderr                              log to standard error as well as files
      --as string                                    Username to impersonate for the operation
      --as-group stringArray                         Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
      --cache-dir string                             Default HTTP cache directory (default "/Users/jrondeau/.kube/http-cache")
      --certificate-authority string                 Path to a cert file for the certificate authority
      --client-certificate string                    Path to a client certificate file for TLS
      --client-key string                            Path to a client key file for TLS
      --cloud-provider-gce-lb-src-cidrs cidrs        CIDRs opened in GCE firewall for LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16)
      --cluster string                               The name of the kubeconfig cluster to use
      --context string                               The name of the kubeconfig context to use
      --default-not-ready-toleration-seconds int     Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300)
      --default-unreachable-toleration-seconds int   Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300)
  -h, --help                                         help for kubefed
      --insecure-skip-tls-verify                     If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
      --ir-data-source string                        Data source used by InitialResources. Supported options: influxdb, gcm. (default "influxdb")
      --ir-dbname string                             InfluxDB database name which contains metrics required by InitialResources (default "k8s")
      --ir-hawkular string                           Hawkular configuration URL
      --ir-influxdb-host string                      Address of InfluxDB which contains metrics required by InitialResources (default "localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb:api/proxy")
      --ir-namespace-only                            Whether the estimation should be made only based on data from the same namespace.
      --ir-password string                           Password used for connecting to InfluxDB (default "root")
      --ir-percentile int                            Which percentile of samples should InitialResources use when estimating resources. For experiment purposes. (default 90)
      --ir-user string                               User used for connecting to InfluxDB (default "root")
      --kubeconfig string                            Path to the kubeconfig file to use for CLI requests.
      --log-backtrace-at traceLocation               when logging hits line file:N, emit a stack trace (default :0)
      --log-dir string                               If non-empty, write log files in this directory
      --log-flush-frequency duration                 Maximum number of seconds between log flushes (default 5s)
      --logtostderr                                  log to standard error instead of files (default true)
      --match-server-version                         Require server version to match client version
  -n, --namespace string                             If present, the namespace scope for this CLI request
      --password string                              Password for basic authentication to the API server
      --request-timeout string                       The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
  -s, --server string                                The address and port of the Kubernetes API server
      --stderrthreshold severity                     logs at or above this threshold go to stderr (default 2)
      --token string                                 Bearer token for authentication to the API server
      --user string                                  The name of the kubeconfig user to use
      --username string                              Username for basic authentication to the API server
  -v, --v Level                                      log level for V logs
      --vmodule moduleSpec                           comma-separated list of pattern=N settings for file-filtered logging

Here’s a high-level diagram that shows what all of these pieces look like when strung together:

Key components

There are two key components to the federation capability within Kubernetes. These components make up the federation control plane.

The first is federation-controller-manager, which embeds the core control loops required to operate federation. federation-controller-manager watches the state of your clusters via apiserver and makes changes in order to reach a desired state.

The second is federation-apiserver, which validates and configures Kubernetes objects such as pods, services, and controllers. federation-apiserver is the frontend for the cluster through which all other components interact.

Federated services

Now that we have the building blocks of federation conceptualized in our mind, let’s review one more facet of this before setting up federation. How exactly does a common service, deployed across multiple clusters, work?

Federated services are created in a very similar fashion to regular services: first, by sending the desired state and properties of the service to an API endpoint, which is then brought to bear by the Kubernetes architecture. There are two main differences:

  • A non-federated service will make an API call directly to a cluster API endpoint
  • A federated service will make the call to the Federated API endpoint at federation/v1beta1, which will then redirect the API call to all of the individual clusters within the federation control plane

This second type of service allows us to extend such things as DNS service discovery across cluster boundaries. The DNS resolv chain is able to leverage service federation and public DNS records to resolve names across multiple clusters.

The API for a federated service is 100% compatible with regular services.

When a service is created, federation takes care of several things. First, it creates matching services in all clusters where kubefed specifies they reside. The health of those services is monitored so that traffic can be routed or re-routed to them. Lastly, federation ensure that there’s a definitive set of public DNS records available through providers such as Route 53 or Google Cloud DNS.

Microservices residing on different pods within your Kubernetes clusters will use all of this machinery in order to locate the federated service either within their own cluster or navigate to the nearest healthy example within your federation map.

Comments are closed.

loading...