S3 backups failing in a Rancher-provisioned RKE2 or K3s cluster with "failed to test for existence of bucket: HEAD Forbidden"

Article Number: 000022081

Environment

A Rancher-provisioned RKE2 or K3s cluster, with etcd snapshots configured to an S3 bucket, and a HTTP Proxy configuration

Situation

The automated etcd snapshots for a Rancher-provisioned RKE2 or K3s cluster to an S3 bucket are failing. The error message listed next to the Failed snapshots is of the following format:

failed to initialize S3 client: failed to test for existence of bucket etcd-backups: Head "https://s3.example.com/etcd-backups/": Forbidden

In addition, in Rancher < v2.11 you may experience some UI slowness as a result of the repeated failing snapshots in the affected cluster(s).

Cause

The cause for this case is determined to be a missing entry in the affected cluster's NO_PROXY environment variable, which can result in a "HEAD <S3-ENDPOINT:PORT> Forbidden" error, as a result of S3 requests being incorrectly sent to the HTTP Proxy.

Resolution

The recommended approach to resolve this is to double check the proxy configuration of the downstream RKE2/K3s cluster, specifically the NO_PROXY setting. Ensure this contains an entry that matches the full hostname of the S3 endpoint, e.g. s3.example.com in the example above.