Skip to content

Failed to handling tunnel request from remote address x.x.x.x:42412 (X-Forwarded-For: x.x.x.x): response 400: cluster not found

Article Number: 000020950

Environment

  • Rancher v2.x

Situation

There are a lot of errors observed in the Rancher pods:

[ERROR] Failed to handling tunnel request from remote address x.x.x.x:42412 (X-Forwarded-For: x.x.x.x): response 400: cluster not found

Cause

Remaining nodes from deleted clusters trying to connect to Rancher.

Resolution

The error message "Failed to handling tunnel request from remote address x.x.x.x:42412 (X-Forwarded-For: x.x.x.x): response 400: cluster not found"  indicates that on some hosts from now-deleted clusters, there are Rancher agent containers/Pods that are still running and attempting to connect to Rancher.

To locate these instances and stop the running containers or hosts to prevent these messages:

Suppose your load-balancer is performing Layer-7 load-balancing and setting the X-Forwarded-For header itself. In that case, you could enable use-forwarded-headers on the ingress-nginx controller in the Rancher local cluster (https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers).

This would pass through the X-Forwarded-For header from the loadbalancer, enabling you to identify the hosts from which these requests originate within the Rancher Pod logs.