Log Aggregation with Fluentd

Log Aggregation with Fluentd

We can use Fluentd to aggregate logs from our apps. These are the steps to build and deploy Fluentd using OpenShift templates and CLI commands. Run these commands with a MINGW64 or Unix command line tool, from inside the directory containing your templates.

OpenShift Templates:

• Build Config - fluentd.bc.yaml
• Config Map - fluent.conf
• Deployment Config - fluentd.dc.yaml
• Network Policy - allow-openshift-ingress-to-fluentd-master-app.yaml

Build the Fluentd image

# set some environment variables in your CLI session
export APP_NAME=fluentd
export INSTANCE=master
export NAMESPACE=example123-dev

# setup Docker Hub credentials (if necessary)
oc create secret docker-registry csst-dockerhub-secret \
    --docker-server=docker.io \
    --docker-username=<Docker Hub username> \
    --docker-password=<Docker Hub personal access token> \
    --docker-email=<email>

# Create the Image Stream:
oc process -n $NAMESPACE -f fluentd.bc.yaml -p INSTANCE=$INSTANCE -o yaml | oc apply -n $NAMESPACE -f -

# start the build:
oc start-build -n $NAMESPACE $APP_NAME-$INSTANCE

Fluentd Config Map

This configuration of the Fluentd container receives logs from a fluent-bit sidecar that runs on someof our other applications, and outputs error notifications to the Showcase Team's Discord channel.

Create a Config Map from the file /fluent.conf note: config map Key for this file must be named fluent.conf

oc create -n $NAMESPACE configmap $APP_NAME-$INSTANCE-config --from-file=fluent.conf

Deployment

Create the deployment in OpenShift by referencing the Deployment Config template in a local file

# set an environment variable in your CLI session for the http route to the Fluentd service
export HOST_ROUTE=fluentd-csst.apps.silver.devops.gov.bc.ca

oc process -n $NAMESPACE -f fluentd.dc.yaml -p INSTANCE=$INSTANCE -p NAMESPACE=$NAMESPACE -p APP_NAME=$APP_NAME -p HOST_ROUTE=$HOST_ROUTE -o yaml | oc apply -n $NAMESPACE -f -

Network Policy

Apply the network policy with the following command:

oc apply -n $NAMESPACE -f allow-openshift-ingress-to-fluentd-master-app.yaml