Skip to content

How to remove Kubernetes namespaces stuck in a Terminating state

This document (000021065) is provided subject to the disclaimer at the end of this document.

Environment

A Kubernetes cluster with a Namespace stuck in a Terminating state

Situation

When attempting to delete a Namespace within a Kubernetes cluster, the Namespace is stuck in a Terminating state and is not being successfully removed.

Some online guides for this situation instruct users to remove the kubernetes finalizer from affected Namespaces, to force their deletion from the cluster. However, the kubernetes finalizer should not be manually removed from a Namespace. Whilst its removal will permit the deletion of the Namespace resource, it may leave orphaned resources within it, which will be accessible again via the Kubernetes API if a Namespace with the same name is created again, and may cause other cluster issues.

This article details how to correctly identify the issue preventing deletion of a Namespace and how to safely remedy this.

Resolution

A Namespace stuck in a Terminating state is most frequently caused by one of two issues:

  1. An unavailable APIService, e.g. the v1beta1.custom.metrics.k8s.io APIService was created within the cluster, but the workload backing the service that implements it has subsequently been deleted.
  2. A resource present within the Namespace, which has a finalizer that can no longer be satisfied, i.e. a custom resource controlled by an operator, with a finalizer referencing the operator, but where the operator has already been removed from the cluster.

To identify and resolve these issues, follow the instructions below.

Check if an APIService is unavailable

  1. Check if any APIService is unavailable by running kubectl get apiservice | grep False, per the following example:
kubectl get apiservices | grep False
v1beta1.custom.metrics.k8s.io          cattle-monitoring-system/rancher-monitoring-prometheus-adapter   False (ServiceNotFound)   55m
  1. If an unavailable APIService is identified, such as v1beta1.custom.metrics.k8s.io in the example above, then take steps to resolve this:
  2. If the workload serving the API has been removed from the cluster, and the APIService is no longer expected to work, then delete it with kubectl delete apiservice <name>, e.g. kubectl delete apiservice v1beta1.custom.metrics.k8s.io
  3. If the APIService is expected to work, then investigate why the Kubernetes Service and/or workload backing it are not working, and fix them to resolve the unavailable APIService.

Identify and remove those resources, within the Namespace, which have a finalizer that cannot be satisfied

  1. Find all resources that still exist where is the Namespace stuck in a terminating state:
kubectl api-resources --verbs=list --namespaced -o name | xargs -I {} sh -c 'echo "Checking for {} resources"; kubectl get {} -n <namespace>;'
  1. Remove the finalizers from any resources within the Namespace identified from the output of the command above:
kubectl -n <namespace> patch <kind> <name> -p '{"metadata":{"finalizers":null}}' --type=merge

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.