Kubernetes – FluentD and Google Cloud Logging

Looking back at the System pod listing screenshot at the beginning of the chapter, you may have noted a number of pods starting with the words fluentd-cloud-logging-kubernetes. These pods appear when using the GCE provider for your K8s cluster.

A pod like this exists on every node in our cluster, and its sole purpose is to handle the processing of Kubernetes logs. If we log in to our Google Cloud Platform account, we can see some of the logs processed there. Simply use the left side, and under Stackdriver, select  Logging. This will take us to a log listing page with a number of drop-down menus on the top. If this is your first time visiting the page, the first drop-down will likely be set to Cloud HTTP Load Balancer

In this drop-down menu, we’ll see a number of GCE types of entries. Select GCE VM instances and then the Kubernetes master or one of the nodes. In the second drop-down, we can choose various log groups, including kubelet. We can also filter by the event log level and date. Additionally, we can use the play button to watch events stream in live shown as follows:

The Google Cloud Logging filter

FluentD

Now we know that the fluentd-cloud-logging-kubernetes pods are sending the data to the Google Cloud, but why do we need FluentD? Simply put, FluentD is a collector.

It can be configured to have multiple sources to collect and tag logs, which are then sent to various output points for analysis, alerting, or archiving. We can even transform data using plugins before it is passed on to its destination.

Not all provider setups have FluentD installed by default, but it is one of the recommended approaches to give us greater flexibility for future monitoring operations. The AWS Kubernetes setup also uses FluentD, but instead forwards events to Elasticsearch.

Exploring FluentD: If you are curious about the inner workings of the FluentD setup or just want to customize the log collection, we can explore quite easily using the kubectl exec command and one of the pod names from the command we ran earlier in the chapter. First, let’s see if we can find the FluentD config file: $ kubectl exec fluentd-cloud-logging-kubernetes-minion-group-r4qt --namespace=kube-system -- ls /etc/td-agent.
 We will look in the etc folder and then td-agent, which is the fluent sub folder. While searching in this directory, we should see a td-agent.conf file. We can view that file with a simple cat command, as follows: $ kubectl exec fluentd-cloud-logging-kubernetes-minion-group-r4qt --namespace=kube-system -- cat /etc/td-agent/td-agent.conf.

We should see a number of sources, including the various Kubernetes components, Docker, and some GCP elements. While we can make changes here, remember that it is a running container and our changes won’t be saved if the pod dies or is restarted. If we really want to customize, it’s best to use this container as a base and build a new container, which we can push to a repository for later use.

Comments are closed.