Skip to content

Unable to delete any Projects in Rancher >= 2.9 if the Legacy Feature Flag was active when the project was created in Rancher < 2.9

Article Number: 000021871

Environment

  • Rancher 2.9+
  • Rancher projects created for the local or Downstream cluster, on a previous version of Rancher
  • The "Legacy feature flag" is enabled under Global Settings in Rancher.

Situation

  • A project with the finalizer "clusterscoped.controller.cattle.io/project-precan-alert-controller" cannot be removed, either from Rancher UI or executing this kubectl conmand in the Rancher local cluster:  'kubectl delete projects.management.cattle.io p-xxxx' and it remains in "terminating" status due to the finalizer.
  • This project has been created on a Rancher version <2.9, with the "legacy" feature flag enabled under Global Settings=> Feature Flags => Legacy

Cause

  • The "legacy" feature flag is a remanent from Rancher 2.5.x and per the docs: https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/installation-references/feature-flags , this flag is disabled by default on new Rancher installations.
  • It is meant for backwards compatibility of clusters that started in the 2.5 era and needed time to remove usage of those features.
  • The finalizer "clusterscoped.controller.cattle.io/project-precan-alert-controller" is from the deprecated and removed Monitoring/Logging/Alerting V1.
  • In Rancher 2.9 and later, none of the legacy code related to this exists. It is advisable not to enable "Legacy feature flag", unless strictly required.

Resolution

  • You can manually edit the project hanging in Terminating status, and remove the finalizer clusterscoped.controller.cattle.io/project-precan-alert-controller.
  • There is a sample script that can help locating projects with the finalizer clusterscoped.controller.cattle.io/project-precan-alert-controller, and delete them (to be executed in the  Rancher local cluster), but please use it with caution: 
namespaces=$(kubectl get ns --no-headers -o custom-columns=NAME:.metadata.name | grep "c-" | grep -v cluster-fleet)

for n in $namespaces; do
  echo "Namespace: $n"
  projects=$(kubectl get project -n $n --no-headers -o name)

  for p in $projects; do
    echo "Project: $p"
    finalizers=$(kubectl get $p -n $n -o json | jq -r '.metadata.finalizers[]')

    for i in $finalizers; do
      if [[ $i == clusterscoped.controller.cattle.io/project-precan-alert-controller* ]]; then
        echo "Removing finalizer from $p: $i"
        index=$(kubectl get $p -n $n -o json | jq -r ".metadata.finalizers | index(\"$i\")")
        kubectl patch $p -n $n --type='json' -p="[{\"op\": \"remove\", \"path\": \"/metadata/finalizers/$index\"}]"
      fi
    done
  done
done