Skip to main content

Kubelet

Plugin: go.d.plugin Module: k8s_kubelet

Overview

This collector monitors Kubelet instances.

This collector is supported on all platforms.

This collector supports collecting metrics from multiple instances of this integration, including remote instances.

Default Behavior

Auto-Detection

This integration doesn't support auto-detection.

Limits

The default configuration for this integration does not impose any limits on data collection.

Performance Impact

The default configuration for this integration is not expected to impose a significant performance impact on the system.

Metrics

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

Per Kubelet instance

These metrics refer to the entire monitored application.

This scope has no labels.

Metrics:

MetricDimensionsUnit
k8s_kubelet.apiserver_audit_requests_rejectedrejectedrequests/s
k8s_kubelet.apiserver_storage_data_key_generation_failuresfailuresevents/s
k8s_kubelet.apiserver_storage_data_key_generation_latencies5µs, 10µs, 20µs, 40µs, 80µs, 160µs, 320µs, 640µs, 1280µs, 2560µs, 5120µs, 10240µs, 20480µs, 40960µs, +Infobserves/s
k8s_kubelet.apiserver_storage_data_key_generation_latencies_percent5µs, 10µs, 20µs, 40µs, 80µs, 160µs, 320µs, 640µs, 1280µs, 2560µs, 5120µs, 10240µs, 20480µs, 40960µs, +Infpercentage
k8s_kubelet.apiserver_storage_envelope_transformation_cache_missescache missesevents/s
k8s_kubelet.kubelet_containers_runningtotalrunning_containers
k8s_kubelet.kubelet_pods_runningtotalrunning_pods
k8s_kubelet.kubelet_pods_log_filesystem_used_bytesa dimension per namespace and podB
k8s_kubelet.kubelet_runtime_operationsa dimension per operation typeoperations/s
k8s_kubelet.kubelet_runtime_operations_errorsa dimension per operation typeerrors/s
k8s_kubelet.kubelet_docker_operationsa dimension per operation typeoperations/s
k8s_kubelet.kubelet_docker_operations_errorsa dimension per operation typeerrors/s
k8s_kubelet.kubelet_node_config_errorexperiencing_errorbool
k8s_kubelet.kubelet_pleg_relist_interval_microseconds0.5, 0.9, 0.99microseconds
k8s_kubelet.kubelet_pleg_relist_latency_microseconds0.5, 0.9, 0.99microseconds
k8s_kubelet.kubelet_token_requeststotal, failedtoken_requests/s
k8s_kubelet.rest_client_requests_by_codea dimension per HTTP status coderequests/s
k8s_kubelet.rest_client_requests_by_methoda dimension per HTTP methodrequests/s

Per volume manager

These metrics refer to the Volume Manager.

This scope has no labels.

Metrics:

MetricDimensionsUnit
k8s_kubelet.volume_manager_total_volumesactual, desiredstate

Alerts

The following alerts are available:

Alert nameOn metricDescription
kubelet_node_config_error k8s_kubelet.kubelet_node_config_errorthe node is experiencing a configuration-related error (0: false, 1: true)
kubelet_token_requests k8s_kubelet.kubelet_token_requestsnumber of failed Token() requests to the alternate token source
kubelet_token_requests k8s_kubelet.kubelet_token_requestsnumber of failed Token() requests to the alternate token source
kubelet_operations_error k8s_kubelet.kubelet_operations_errorsnumber of Docker or runtime operation errors
kubelet_operations_error k8s_kubelet.kubelet_operations_errorsnumber of Docker or runtime operation errors

Setup

Prerequisites

No action required.

Configuration

File

The configuration file name for this integration is go.d/k8s_kubelet.conf.

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/k8s_kubelet.conf

Options

The following options can be defined globally: update_every, autodetection_retry.

Config options
NameDescriptionDefaultRequired
update_everyData collection frequency.1no
autodetection_retryRecheck interval in seconds. Zero means no recheck will be scheduled.0no
urlServer URL.http://127.0.0.1:10255/metricsyes
timeoutHTTP request timeout.1no
usernameUsername for basic HTTP authentication.no
passwordPassword for basic HTTP authentication.no
proxy_urlProxy URL.no
proxy_usernameUsername for proxy basic HTTP authentication.no
proxy_passwordPassword for proxy basic HTTP authentication.no
methodHTTP request method.GETno
bodyHTTP request body.no
headersHTTP request headers.no
not_follow_redirectsRedirect handling policy. Controls whether the client follows redirects.nono
tls_skip_verifyServer certificate chain and hostname validation policy. Controls whether the client performs this check.nono
tls_caCertification authority that the client uses when verifying the server's certificates.no
tls_certClient TLS certificate.no
tls_keyClient TLS key.no

Examples

Basic

A basic example configuration.

jobs:
- name: local
url: http://127.0.0.1:10255/metrics

HTTPS with self-signed certificate

Do not validate server certificate chain and hostname.

Config
jobs:
- name: local
url: http://127.0.0.1:10250/metrics
tls_skip_verify: yes

Troubleshooting

Debug Mode

Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.

To troubleshoot issues with the k8s_kubelet collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

  • Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

    cd /usr/libexec/netdata/plugins.d/
  • Switch to the netdata user.

    sudo -u netdata -s
  • Run the go.d.plugin to debug the collector:

    ./go.d.plugin -d -m k8s_kubelet

Getting Logs

If you're encountering problems with the k8s_kubelet collector, follow these steps to retrieve logs and identify potential issues:

  • Run the command specific to your system (systemd, non-systemd, or Docker container).
  • Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.

System with systemd

Use the following command to view logs generated since the last Netdata service restart:

journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep k8s_kubelet

System without systemd

Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector's name:

grep k8s_kubelet /var/log/netdata/collector.log

Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.

Docker Container

If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:

docker logs netdata 2>&1 | grep k8s_kubelet

Do you have any feedback for this page? If so, you can open a new issue on our netdata/learn repository.