Viewing System Logs in Grafana

Grafana is an open-source analytics application. It provides charts, graphs, and alerts when connected to supported data sources.

You can configure TOS to make logs available in Grafana. In this way, Grafana can access all system logs, both from standard output and files, and can search logs using labels. For information about working with Grafana, see https://grafana.com.

Access to Grafana

The connection is established in the Grafana Loki data-source automatically. Loki is a component set that indexes only metadata about logs. By default, Loki stores data for no more than 14 days and no more than 40 Gb. We recommend that you do not change these default values.

Access to Grafana

By default, only the TOS admin user can access Grafana. However, the TOS admin can give permission to other users with the Super Administrator role.

  • From TOS 5.2, all administrators and super administrators (new and existing) will have access to Grafana.

To add a user to the monitoring group:

[<ADMIN> ~]# kubectl exec -it deploy/keycloak-service -c keycloak-service -- manage_keycloak -r groups -a add_user_to_group -g monitoring -u <username>
kubectl exec -it deploy/keycloak-service -c keycloak-service -- manage_keycloak -r groups -a add_user_to_group -g monitoring -u <username>

To remove a user from the monitoring group:

[<ADMIN> ~]# kubectl exec -it deploy/keycloak-service -c keycloak-service -- manage_keycloak -r groups -a remove_user_from_group -g monitoring -u <user name>
kubectl exec -it deploy/keycloak-service -c keycloak-service -- manage_keycloak -r groups -a remove_user_from_group -g monitoring -u <user name>

Viewing logs in Grafana

  1. Go to:

    https://<IP>/v2/monitor/grafana/

  2. Select Explore > Loki.

  3. In Log Labels, search for logs of a specific pod, container, or filename.

Exporting log CLI

The Log CLI allows you to share a log with Tufin Support or other administrators who need to analyze the logs.

curl -LO https://github.com/grafana/loki/releases/download/v2.1.0/logcli-linux-amd64.zip
unzip logcli-linux-amd64.zip
mv logcli-linux-amd64 logcli

export LOKI_ADDR="http://$(kubectl get svc loki-querier -ojsonpath='{.spec.clusterIP}'):3100"
./logcli labels container
./logcli query '{container="tss"}'
./logcli query '{container="tss"} |~ ".*error.*"'
./logcli series -q --match='{instance="172.16.0.1",container="tss"} |= "error"

For further information, see grafana.com/docs/loki/latest/getting-started/logcli

Disabling access to Grafana

When a user no longer needs access to Grafana, you should remove their user permissions:

[<ADMIN> ~]# kubectl exec -it deploy/keycloak-service -c keycloak-service -- manage_keycloak -r groups -a remove_user_from_group -g monitoring -u <user name>
kubectl exec -it deploy/keycloak-service -c keycloak-service -- manage_keycloak -r groups -a remove_user_from_group -g monitoring -u <user name>