Managing terminating states
Your project can get stuck in Terminating states when multiple projects are being managed by RedHat OpenShift Operators. As operators manage the life-cycle of applications, invariably, the stuck-in-terminating problem is often one that you may encounter.
This problem persists even after you confirm that you deleted all resources found by using the oc get all command in the project.
To resolve this issue we need to modify an OpenShift Resource called a namespace, which is synonymous with Red Hat project. We will modify the finalizer in OpenShift, which is a mechanism to inform the Kubernetes control plane that an action needs to take place so the standard Kubernetes garbage logic is performed.
To modify any project/namespace, you need the cluster-admin Role-Based Access Control (RBAC) permission. If you do not have the permission, you get an error like this one:
# oc get namespace
Error from server (Forbidden): namespaces is forbidden: User "suyi" cannot list namespaces at the cluster scope: no RBAC policy matched
Where suyi is your OpenShift username.
Make sure you have the cluster-admin privilege
Ask your OpenShift administrator to grant you the cluster-admin permission with a command like this:
# oc adm policy add-cluster-role-to-user cluster-admin suyi
or
# oc adm policy add-role-to-user admin suyi -n demo-suyi
Where demo-suyi is your namespace/project name.
If you want the admin privilege for the specific namespace that is visible to you, use the following command:
After you are granted a cluster-admin role, you can then run this command:
# oc get namespace
That command lists all the projects that are available in OpenShift. You will see your project in the Terminating state below:
Run the following command to display the content of the problematic namespace:
# oc get -o yaml namespace/demo-suyi
Where demo-suyi is your namespace/project name.
Next, modify that value under finalizers. Remove the line - kubernetes.
However, if you try to edit and save the file directly with the oc edit namespace/demo-suyi command, your finalizer is not updated. Direct editing might not reflect the changes in the namespace.
Instead, you can extract the YAML with the following command:
# oc get namespace demo-suyi -o yaml > demo-suyi-yaml-bkp.yaml
Where demo-suyi is the namespace/project name that you want to delete, and demo-suyi-yaml-bkp is the file name where you want to back up your namespace.
vi the file and remove the value under finalizers.
Delete the line with - kubernetes
After deleting the file, the result looks like this. Save the file and exit.
Next, start a temporary local proxy server to let the local server talk to the remote OpenShift cluster. Running the command below starts the temporary proxy process in the background.
# oc proxy &
Load the new YAML via the API using curl.
# curl -k -H "Content-Type: application/yaml" -X PUT --data-binary @demo-suyi-yaml-bkp.yaml http://127.0.0.1:8001/api/v1/namespaces/demo-suyi/finalize
Where demo-suyi is the namespace/project name you want to delete, and demo-suyi-yaml-bkp is the file name for your namespace backup. Note that http://127.0.0.1:8001 is the proxy server that we started above.
Update the namespace object in the OpenShift cluster with HTTP POST
The command above gives a successful response, indicating that you updated the namespace by sending the update request with the file. Alternatively, you can try the oc apply -f demo-suyi-yaml-bkp.yaml command, but this does not always work.
You should no longer see the namespace/project.
# oc get namespace demo-suyi
Great! You have managed to remove the offending project.
To finish off, kill the proxy server with the following command:
# kill -9 %%
Congratulations!! 😶🌫️🫅🏽🪅🧆🚂🕳️
You have managed to remove the offending project.
Ref: https://www.redhat.com/sysadmin/openshift-terminating-state
Top comments (0)