How to delete Kubernetes resources for good

Kubernetes is ruling the tech industry now for a while. Either it’s really credible or it’s just hype – it doesn’t matter. People are using this more than ever.

I’ll show how a Kubernetes resource can be deleted. Probably you already know about it, or maybe not. Let’s see.

The basic way of deleting a K8s resource is to execute delete command in kubectl. So if you want to delete a pod my-pod, you run,

$ kubectl delete pod my-pod 
pod "my-pod-3984774" deleted

This works most of the time. Unless …, it doesn’t. You’ll delete a pod and after few seconds you check if your pod is deleted.

$ kubectl get pods
NAME         READY STATUS  RESTARTS AGE
my-pod-fc2wc 2/2   Running 0        43s

What? I just deleted my pod. Why it started again? someone is doing something behind.

This is pretty common if you have a deployment that manages these pods. So the next step could be just deleting the deployment. This could go on and on. You delete pod, deployment, service, replication controllers, replica sets, namespaces. Sometimes, it’s enough to delete some resources. Then they will not come back. Other times, (sigh!), they keep coming back.

When you are experimenting K8s installation for few weeks you probably know if you delete a Pod, the Deployment/ReplicationController/Replicaset will create a new Pod right away. This way Pods cannot be deleted for good. We need to delete those parent resources that are controlling Pods.

It’s possible to see which resource is responsible for this. We just need to check the metadata.ownerReferences attribute of the resource we want to delete. The following command will give access to that information for pod my-pod-fc2wc.

kubectl get pod my-pod-fc2wc -o yaml

The output will be very verbose. But if you browse you’ll see a line containing ownerReferences. That’ll contain a name and kind. Get those and execute

kubectl get <kind> <name> -o yaml

Keep doing that as long as you don’t see ownerReference. And then delete only That resource.

For example, this is how it looks like for my-pod-fc2wc

ownerReferences:
 apiVersion: apps/v1
 blockOwnerDeletion: true
 controller: true
 kind: ReplicaSet
 name: ml-pipeline-ui-artifact-5d68fc6566
 uid: df457e76-2569-492b-88e8-5375afc154a3
 resourceVersion: "11868965"
 selfLink: /api/v1/namespaces/MY-NAMESPACE/pods/ml-pipeline-ui-artifact-5d68fc6566-fc2wc
 uid: bcc9e531-ce0b-4462-81b3-00d588a5bcc3 

If you have jq installed, then this look-up can be done pretty easily without scrolling 100s of YAML lines.

kubectl get pod my-pod-fc2wc -o json | jq .metadata.ownerReferences