Skip to content

Newly created namespaces not visible due to malformed ResourceQuota annotations

Article Number: 000021931

Environment

A Rancher v2.x managed cluster, using Project Resource Quotas

Situation

When a non-admin user creates a new namespace within an affected project, the namespace is not visible to them; however, the namespace object is successfully created and can be viewed by an admin user.

Inspecting the manifest of the affected namespace, after creation, the following annotation is missing:

lifecycle.cattle.io/create.namespace-auth: "true"

Another important symptom is that the Rancher logs show errors similar to the following:

[ERROR] error syncing 'test-suse': handler namespace-auth: invalid character 'm' after object key:value pair, handler resourceQuotaSyncController: invalid character 'm' after object key:value pair, requeuing

Cause

The problem is not with the newly created namespace, but with an existing namespace within the same project that has a malformed JSON string in the field.cattle.io/resourceQuota annotation.

When a new namespace is created, Rancher controllers process resources within that project. If these controllers encounter a namespace with a syntax error in the field.cattle.io/resourceQuota annotation, this process will fail. This behaviour is related to an induced failure in Go's json.Unmarshal function within Rancher's resource quota controllers. As a result, the expected RBAC is not configured on the new namespace, indicated by the absence of the lifecycle.cattle.io/create.namespace-auth: "true" annotation, and it remains hidden to non-admin users.

This issue is not introduced by Rancher itself and namespaces where the resource quota is only defined via the Rancher UI will contain correctly formatted JSON. The issue occurs where the field.cattle.io/resourceQuota annotation is modified manually via kubectl edit or through GitOps tools (such as ArgoCD), and a syntax error is introduced.

Examples of malformed JSON:

  • Incorrect quoting: Using triple single quotes instead of one.

field.cattle.io/resourceQuota: '''{"limit":{"pods":"30", ... }}'''
- Missing characters: A missing double quote on a value.

field.cattle.io/resourceQuota: '{"limit":{ ... "limitsCpu":1600m", ... }}'

(Note the missing quote before 1600m)

Resolution

To resolve this issue, you must identify and correct the malformed field.cattle.io/resourceQuota annotation on the problematic namespace(s) within the affected project. Look for syntax errors such as incorrect or extra quotes, missing commas, or other typos, and once you've found the namespace with the invalid annotation, use kubectl edit to correct the JSON syntax.

For example, correct this annotation with extra single quotes: field.cattle.io/resourceQuota: '''{"limit":...}'''

To this: field.cattle.io/resourceQuota: '{"limit":...}'