Metrics-server CrashLoopBackOff with NEW install by rke - General Discussions

Hello all,
I have installed at least 10 times last one days, but its same every time
Everything runs fine but metrics-server is CrashLoopBackOff

what I understand below section are missing from the pods YAML file and needs to add into Deployment.

I am new to Kubernetes, I have 2 question

I am using rke to install rancher cluster, so why bellow setting are missing from pods to start metric-server?

Command:
/metrics-server
–cert-dir=/tmp
–secure-port=4443
–kubelet-preferred-address-types=InternalIP
–kubelet-insecure-tls

2) what is the best way to add those lines ### Cluster information: Kubernetes version: [rke@rke19-master1 ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.0.56 Ready controlplane,etcd 17m v1.19.10 192.168.0.57 Ready controlplane,etcd 17m v1.19.10 192.168.0.58 Ready controlplane,etcd 17m v1.19.10 192.168.0.59 Ready worker 17m v1.19.10 192.168.0.60 Ready worker 17m v1.19.10 [rke@rke19-master1 ~]$ > [rke@rke19-master1 ~]$ kubectl get pods -n kube-system > NAME READY STATUS RESTARTS AGE > calico-kube-controllers-5cc4f54c4f-8wfsq 1/1 Running 0 9m10s > canal-dxklh 2/2 Running 0 9m10s > canal-fgp2j 2/2 Running 0 9m11s > canal-fzg2f 2/2 Running 0 9m11s > canal-nxgvl 2/2 Running 0 9m11s > canal-r59gp 2/2 Running 0 9m10s > coredns-67bbc84b8f-2cs8m 1/1 Running 0 7m6s > coredns-67bbc84b8f-dbm29 1/1 Running 0 6m51s > coredns-autoscaler-fd9d4557c-nkbk9 1/1 Running 0 7m7s > metrics-server-5b6d79d4f4-ggl57 0/1 CrashLoopBackOff 6 6m46s > rke-coredns-addon-deploy-job-4rp28 0/1 Completed 0 9m3s > rke-ingress-controller-deploy-job-p2tll 0/1 Completed 0 6m39s > rke-metrics-addon-deploy-job-rtf7c 0/1 Completed 0 7m2s > rke-network-plugin-deploy-job-tkkz5 0/1 Completed 0 9m35s > [rke@rke19-master1 ~]$ kubectl get pods metrics-server-5b6d79d4f4-ggl57 -n kube-system -o yaml > apiVersion: v1 > kind: Pod > metadata: > annotations: > cni.projectcalico.org/podIP: 10.42.4.3/32 > cni.projectcalico.org/podIPs: 10.42.4.3/32 > creationTimestamp: "2021-08-16T23:00:42Z" > generateName: metrics-server-5b6d79d4f4- > labels: > k8s-app: metrics-server > pod-template-hash: 5b6d79d4f4 > managedFields: > - apiVersion: v1 > fieldsType: FieldsV1 > fieldsV1: > f:metadata: > f:generateName: {} > f:labels: > .: {} > f:k8s-app: {} > f:pod-template-hash: {} > f:ownerReferences: > .: {} > k:{"uid":"fb15b257-4a9d-478b-b461-8b61c165e3db"}: > .: {} > f:apiVersion: {} > f:blockOwnerDeletion: {} > f:controller: {} > f:kind: {} > f:name: {} > f:uid: {} > f:spec: > f:affinity: > .: {} > f:nodeAffinity: > .: {} > f:requiredDuringSchedulingIgnoredDuringExecution: > .: {} > f:nodeSelectorTerms: {} > f:containers: > k:{"name":"metrics-server"}: > .: {} > f:args: {} > f:image: {} > f:imagePullPolicy: {} > f:livenessProbe: > .: {} > f:failureThreshold: {} > f:httpGet: > .: {} > f:path: {} > f:port: {} > f:scheme: {} > f:periodSeconds: {} > f:successThreshold: {} > f:timeoutSeconds: {} > f:name: {} > f:ports: > .: {} > k:{"containerPort":4443,"protocol":"TCP"}: > .: {} > f:containerPort: {} > f:name: {} > f:protocol: {} > f:readinessProbe: > .: {} > f:failureThreshold: {} > f:httpGet: > .: {} > f:path: {} > f:port: {} > f:scheme: {} > f:periodSeconds: {} > f:successThreshold: {} > f:timeoutSeconds: {} > f:resources: {} > f:securityContext: > .: {} > f:readOnlyRootFilesystem: {} > f:runAsNonRoot: {} > f:runAsUser: {} > f:terminationMessagePath: {} > f:terminationMessagePolicy: {} > f:volumeMounts: > .: {} > k:{"mountPath":"/tmp"}: > .: {} > f:mountPath: {} > f:name: {} > f:dnsPolicy: {} > f:enableServiceLinks: {} > f:priorityClassName: {} > f:restartPolicy: {} > f:schedulerName: {} > f:securityContext: {} > f:serviceAccount: {} > f:serviceAccountName: {} > f:terminationGracePeriodSeconds: {} > f:tolerations: {} > f:volumes: > .: {} > k:{"name":"tmp-dir"}: > .: {} > f:emptyDir: {} > f:name: {} > manager: kube-controller-manager > operation: Update > time: "2021-08-16T23:00:42Z" > - apiVersion: v1 > fieldsType: FieldsV1 > fieldsV1: > f:metadata: > f:annotations: > .: {} > f:cni.projectcalico.org/podIP: {} > f:cni.projectcalico.org/podIPs: {} > manager: calico > operation: Update > time: "2021-08-16T23:00:47Z" > - apiVersion: v1 > fieldsType: FieldsV1 > fieldsV1: > f:status: > f:conditions: > k:{"type":"ContainersReady"}: > .: {} > f:lastProbeTime: {} > f:lastTransitionTime: {} > f:message: {} > f:reason: {} > f:status: {} > f:type: {} > k:{"type":"Initialized"}: > .: {} > f:lastProbeTime: {} > f:lastTransitionTime: {} > f:status: {} > f:type: {} > k:{"type":"Ready"}: > .: {} > f:lastProbeTime: {} > f:lastTransitionTime: {} > f:message: {} > f:reason: {} > f:status: {} > f:type: {} > f:containerStatuses: {} > f:hostIP: {} > f:phase: {} > f:podIP: {} > f:podIPs: > .: {} > k:{"ip":"10.42.4.3"}: > .: {} > f:ip: {} > f:startTime: {} > manager: kubelet > operation: Update > time: "2021-08-16T23:00:54Z" > name: metrics-server-5b6d79d4f4-ggl57 > namespace: kube-system > ownerReferences: > - apiVersion: apps/v1 > blockOwnerDeletion: true > controller: true > kind: ReplicaSet > name: metrics-server-5b6d79d4f4 > uid: fb15b257-4a9d-478b-b461-8b61c165e3db > resourceVersion: "3887" > selfLink: /api/v1/namespaces/kube-system/pods/metrics-server-5b6d79d4f4-ggl57 > uid: af8d4e07-aa3f-4efe-8169-feb37cfd97df > spec: > affinity: > nodeAffinity: > requiredDuringSchedulingIgnoredDuringExecution: > nodeSelectorTerms: > - matchExpressions: > - key: beta.kubernetes.io/os > operator: NotIn > values: > - windows > - key: node-role.kubernetes.io/worker > operator: Exists > containers: > - args: > - --cert-dir=/tmp > - --secure-port=4443 > - --kubelet-insecure-tls > - --kubelet-preferred-address-types=InternalIP > - --logtostderr > image: 192.168.0.35:5000/rancher/metrics-server:v0.3.6 > imagePullPolicy: IfNotPresent > livenessProbe: > failureThreshold: 3 > httpGet: > path: /livez > port: https > scheme: HTTPS > periodSeconds: 10 > successThreshold: 1 > timeoutSeconds: 1 > name: metrics-server > ports: > - containerPort: 4443 > name: https > protocol: TCP > readinessProbe: > failureThreshold: 3 > httpGet: > path: /readyz > port: https > scheme: HTTPS > periodSeconds: 10 > successThreshold: 1 > timeoutSeconds: 1 > resources: {} > securityContext: > readOnlyRootFilesystem: true > runAsNonRoot: true > runAsUser: 1000 > terminationMessagePath: /dev/termination-log > terminationMessagePolicy: File > volumeMounts: > - mountPath: /tmp > name: tmp-dir > - mountPath: /var/run/secrets/kubernetes.io/serviceaccount > name: metrics-server-token-78b6h > readOnly: true > dnsPolicy: ClusterFirst > enableServiceLinks: true > nodeName: 192.168.0.59 > preemptionPolicy: PreemptLowerPriority > priority: 2000000000 > priorityClassName: system-cluster-critical > restartPolicy: Always > schedulerName: default-scheduler > securityContext: {} > serviceAccount: metrics-server > serviceAccountName: metrics-server > terminationGracePeriodSeconds: 30 > tolerations: > - effect: NoExecute > operator: Exists > - effect: NoSchedule > operator: Exists > volumes: > - emptyDir: {} > name: tmp-dir > - name: metrics-server-token-78b6h > secret: > defaultMode: 420 > secretName: metrics-server-token-78b6h > status: > conditions: > - lastProbeTime: null > lastTransitionTime: "2021-08-16T23:00:43Z" > status: "True" > type: Initialized > - lastProbeTime: null > lastTransitionTime: "2021-08-16T23:00:43Z" > message: 'containers with unready status: [metrics-server]' > reason: ContainersNotReady > status: "False" > type: Ready > - lastProbeTime: null > lastTransitionTime: "2021-08-16T23:00:43Z" > message: 'containers with unready status: [metrics-server]' > reason: ContainersNotReady > status: "False" > type: ContainersReady > - lastProbeTime: null > lastTransitionTime: "2021-08-16T23:00:43Z" > status: "True" > type: PodScheduled > containerStatuses: > - containerID: docker://abdfe11278348f3481565a380f5290e4eb7894c6bbcf6373a87ee6bf8d769fd6 > image: 192.168.0.35:5000/rancher/metrics-server:v0.3.6 > imageID: docker-pullable://192.168.0.35:5000/rancher/metrics-server@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b > lastState: > terminated: > containerID: docker://abdfe11278348f3481565a380f5290e4eb7894c6bbcf6373a87ee6bf8d769fd6 > exitCode: 2 > finishedAt: "2021-08-16T23:15:47Z" > reason: Error > startedAt: "2021-08-16T23:15:18Z" > name: metrics-server > ready: false > restartCount: 9 > started: false > state: > waiting: > message: back-off 5m0s restarting failed container=metrics-server pod=metrics-server-5b6d79d4f4-ggl57_kube-system(af8d4e07-aa3f-4efe-8169-feb37cfd97df) > reason: CrashLoopBackOff > hostIP: 192.168.0.59 > phase: Running > podIP: 10.42.4.3 > podIPs: > - ip: 10.42.4.3 > qosClass: BestEffort > startTime: "2021-08-16T23:00:43Z" [rke@rke19-master1 ~]$ kubectl logs metrics-server-5b6d79d4f4-ggl57 -n kube-system I0816 23:21:20.156497 1 secure_serving.go:116] Serving securely on [::]:4443 [rke@rke19-master1 ~]$ [rke@rke19-master1 ~]$ kubectl get pods metrics-server-5b6d79d4f4-ggl57 -n kube-system -o yaml apiVersion: v1 kind: Pod metadata: annotations: cni.projectcalico.org/podIP: 10.42.4.3/32 cni.projectcalico.org/podIPs: 10.42.4.3/32 creationTimestamp: "2021-08-16T23:00:42Z" generateName: metrics-server-5b6d79d4f4- labels: k8s-app: metrics-server pod-template-hash: 5b6d79d4f4 managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:k8s-app: {} f:pod-template-hash: {} f:ownerReferences: .: {} k:{"uid":"fb15b257-4a9d-478b-b461-8b61c165e3db"}: .: {} f:apiVersion: {} f:blockOwnerDeletion: {} f:controller: {} f:kind: {} f:name: {} f:uid: {} f:spec: f:affinity: .: {} f:nodeAffinity: .: {} f:requiredDuringSchedulingIgnoredDuringExecution: .: {} f:nodeSelectorTerms: {} f:containers: k:{"name":"metrics-server"}: .: {} f:args: {} f:image: {} f:imagePullPolicy: {} f:livenessProbe: .: {} f:failureThreshold: {} f:httpGet: .: {} f:path: {} f:port: {} f:scheme: {} f:periodSeconds: {} f:successThreshold: {} f:timeoutSeconds: {} f:name: {} f:ports: .: {} k:{"containerPort":4443,"protocol":"TCP"}: .: {} f:containerPort: {} f:name: {} f:protocol: {} f:readinessProbe: .: {} f:failureThreshold: {} f:httpGet: .: {} f:path: {} f:port: {} f:scheme: {} f:periodSeconds: {} f:successThreshold: {} f:timeoutSeconds: {} f:resources: {} f:securityContext: .: {} f:readOnlyRootFilesystem: {} f:runAsNonRoot: {} f:runAsUser: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/tmp"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:priorityClassName: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: {} f:serviceAccount: {} f:serviceAccountName: {} f:terminationGracePeriodSeconds: {} f:tolerations: {} f:volumes: .: {} k:{"name":"tmp-dir"}: .: {} f:emptyDir: {} f:name: {} manager: kube-controller-manager operation: Update time: "2021-08-16T23:00:42Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:cni.projectcalico.org/podIP: {} f:cni.projectcalico.org/podIPs: {} manager: calico operation: Update time: "2021-08-16T23:00:47Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:phase: {} f:podIP: {} f:podIPs: .: {} k:{"ip":"10.42.4.3"}: .: {} f:ip: {} f:startTime: {} manager: kubelet operation: Update time: "2021-08-16T23:00:54Z" name: metrics-server-5b6d79d4f4-ggl57 namespace: kube-system ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: ReplicaSet name: metrics-server-5b6d79d4f4 uid: fb15b257-4a9d-478b-b461-8b61c165e3db resourceVersion: "3887" selfLink: /api/v1/namespaces/kube-system/pods/metrics-server-5b6d79d4f4-ggl57 uid: af8d4e07-aa3f-4efe-8169-feb37cfd97df spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/os operator: NotIn values: - windows - key: node-role.kubernetes.io/worker operator: Exists containers: - args: - --cert-dir=/tmp - --secure-port=4443 - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP - --logtostderr image: 192.168.0.35:5000/rancher/metrics-server:v0.3.6 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: metrics-server ports: - containerPort: 4443 name: https protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /readyz port: https scheme: HTTPS periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: {} securityContext: readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /tmp name: tmp-dir - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: metrics-server-token-78b6h readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: 192.168.0.59 preemptionPolicy: PreemptLowerPriority priority: 2000000000 priorityClassName: system-cluster-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: metrics-server serviceAccountName: metrics-server terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute operator: Exists - effect: NoSchedule operator: Exists volumes: - emptyDir: {} name: tmp-dir - name: metrics-server-token-78b6h secret: defaultMode: 420 secretName: metrics-server-token-78b6h status: conditions: - lastProbeTime: null lastTransitionTime: "2021-08-16T23:00:43Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2021-08-16T23:00:43Z" message: 'containers with unready status: [metrics-server]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2021-08-16T23:00:43Z" message: 'containers with unready status: [metrics-server]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2021-08-16T23:00:43Z" status: "True" type: PodScheduled containerStatuses: - containerID: docker://abdfe11278348f3481565a380f5290e4eb7894c6bbcf6373a87ee6bf8d769fd6 image: 192.168.0.35:5000/rancher/metrics-server:v0.3.6 imageID: docker-pullable://192.168.0.35:5000/rancher/metrics-server@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b lastState: terminated: containerID: docker://abdfe11278348f3481565a380f5290e4eb7894c6bbcf6373a87ee6bf8d769fd6 exitCode: 2 finishedAt: "2021-08-16T23:15:47Z" reason: Error startedAt: "2021-08-16T23:15:18Z" name: metrics-server ready: false restartCount: 9 started: false state: waiting: message: back-off 5m0s restarting failed container=metrics-server pod=metrics-server-5b6d79d4f4-ggl57_kube-system(af8d4e07-aa3f-4efe-8169-feb37cfd97df) reason: CrashLoopBackOff hostIP: 192.168.0.59 phase: Running podIP: 10.42.4.3 podIPs: - ip: 10.42.4.3 qosClass: BestEffort startTime: "2021-08-16T23:00:43Z" [rke@rke19-master1 ~]$ ^C [rke@rke19-master1 ~]$ kubectl logs metrics-server-5b6d79d4f4-ggl57 -n kube-system I0816 23:21:20.156497 1 secure_serving.go:116] Serving securely on [::]:4443 [rke@rke19-master1 ~]$ ^C [rke@rke19-master1 ~]$ kubectl describe pods metrics-server-5b6d79d4f4-ggl57 -n kube-system Name: metrics-server-5b6d79d4f4-ggl57 Namespace: kube-system Priority: 2000000000 Priority Class Name: system-cluster-critical Node: 192.168.0.59/192.168.0.59 Start Time: Tue, 17 Aug 2021 00:00:43 +0100 Labels: k8s-app=metrics-server pod-template-hash=5b6d79d4f4 Annotations: cni.projectcalico.org/podIP: 10.42.4.3/32 cni.projectcalico.org/podIPs: 10.42.4.3/32 Status: Running IP: 10.42.4.3 IP: 10.42.4.3 Controlled By: ReplicaSet/metrics-server-5b6d79d4f4 Containers: metrics-server: Container ID: docker://e28b6812965786cd2f520a20dd2adf6cbe9c6a720de905ce16992ed0f4cd7c9e Image: 192.168.0.35:5000/rancher/metrics-server:v0.3.6 Image ID: docker-pullable://192.168.0.35:5000/rancher/metrics-server@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b Port: 4443/TCP Host Port: 0/TCP Args: --cert-dir=/tmp --secure-port=4443 --kubelet-insecure-tls --kubelet-preferred-address-types=InternalIP --logtostderr State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Tue, 17 Aug 2021 00:21:18 +0100 Finished: Tue, 17 Aug 2021 00:21:47 +0100 Ready: False Restart Count: 11 Liveness: http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get https://:https/readyz delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: <none> Mounts: /tmp from tmp-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-78b6h (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: tmp-dir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> metrics-server-token-78b6h: Type: Secret (a volume populated by a Secret) SecretName: metrics-server-token-78b6h Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: :NoExecuteop=Exists :NoScheduleop=Exists Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 21m default-scheduler Successfully assigned kube-system/metrics-server-5b6d79d4f4-ggl57 to 192.168.0.59 Normal Pulling 21m kubelet Pulling image "192.168.0.35:5000/rancher/metrics-server:v0.3.6" Normal Pulled 21m kubelet Successfully pulled image "192.168.0.35:5000/rancher/metrics-server:v0.3.6" in 4.687484656s Warning Unhealthy 21m kubelet Readiness probe failed: Get "https://10.42.4.3:4443/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 21m kubelet Liveness probe failed: Get "https://10.42.4.3:4443/livez": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 20m kubelet Readiness probe failed: Get "https://10.42.4.3:4443/readyz": dial tcp 10.42.4.3:4443: connect: connection refused Warning Unhealthy 20m (x5 over 21m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 404 Warning Unhealthy 20m (x5 over 21m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 404 Normal Killing 20m (x2 over 21m) kubelet Container metrics-server failed liveness probe, will be restarted Normal Created 20m (x3 over 21m) kubelet Created container metrics-server Normal Started 20m (x3 over 21m) kubelet Started container metrics-server Warning BackOff 6m29s (x52 over 18m) kubelet Back-off restarting failed container Normal Pulled 83s (x10 over 20m) kubelet Container image "192.168.0.35:5000/rancher/metrics-server:v0.3.6" already present on machine [rke@rke19-master1 ~]$

Configure Quality of Service for Pods

This page shows how to configure Pods so that they will be assigned particular Quality of Service (QoS) classes. Kubernetes uses QoS classes to make decisions about scheduling and evicting Pods. Before you begin You need to have a Kubernetes cluster,...