Skip to content

Error "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]" in K3s or RKE2

Article Number: 000022297

Environment

A Rancher-provisioned or standalone RKE2 or K3s cluster

Situation

The control plane nodes (API Server) in a K3s or RKE2 cluster report the following error in the K3s logs or RKE2 kube-apiserver Pod logs: "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"

These errors may appear intermittently and often do not coincide with any observable functional degradation of the Kubernetes cluster.

Cause

This error indicates that a client (a user, a service account, or an automated process) is attempting to communicate with the Kubernetes API using a token that the API server no longer recognizes as valid. Common triggers include:

  • Stale Secrets: A resource or controller in the cluster may be holding a reference to a secret containing an old token.
  • Post-Maintenance Artifacts: This is frequently observed after etcd operations or restoring a cluster from an etcd snapshot, where previous session tokens are invalidated.
  • Token Accumulation: An accumulation of expired or unused tokens from past sessions or integrations.
  • External Clients: Individual users or CI/CD pipelines using outdated kubeconfig files.

Resolution

If there is no functional impact on cluster operations (e.g., pods are deploying correctly, and the UI is responsive), this message is considered informational and safe to ignore. It simply records a rejected authentication attempt.

To identify and silence the source of these errors, you can investigate the origin of the requests:

  1. Enable API Audit Logging: Audit logs provide details on the source IP, user agent, and specific identity attempting the connection. Refer to the documentation to enable this:

  2. Enable API audit log in RKE2

  3. Collect Logs: Use a logging aggregator to analyze the frequency and source of the 401 Unauthorized responses.

  4. Collect Audit API logs using Rancher Logging

  5. Remediation: Once the source IP or User-Agent is identified, update the kubeconfig or restart the offending pod/service to refresh its service account token.