ubuntu安装rancher2.6.3出错

docker version:20.10.12
rancher:2.6.3

docker logs rancher

2022/01/04 09:23:09 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6443/version?timeout=15m0s": dial tcp 127.0.0.1:6443: connect: connection refused
2022/01/04 09:23:11 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6443/version?timeout=15m0s": dial tcp 127.0.0.1:6443: connect: connection refused
2022/01/04 09:23:13 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6443/version?timeout=15m0s": dial tcp 127.0.0.1:6443: connect: connection refused
2022/01/04 09:23:15 [INFO] Waiting for server to become available: an error on the server ("apiserver not ready") has prevented the request from succeeding
2022/01/04 09:23:17 [INFO] Waiting for server to become available: an error on the server ("apiserver not ready") has prevented the request from succeeding
2022/01/04 09:23:19 [INFO] Waiting for server to become available: an error on the server ("apiserver not ready") has prevented the request from succeeding
2022/01/04 09:23:21 [INFO] Waiting for server to become available: an error on the server ("apiserver not ready") has prevented the request from succeeding
2022/01/04 09:23:33 [INFO] Running in single server mode, will not peer connections
2022/01/04 09:23:33 [INFO] Applying CRD features.management.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD navlinks.ui.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD clusters.management.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD apiservices.management.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD settings.management.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD preferences.management.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD features.management.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD operations.catalog.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD apps.catalog.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD managedcharts.management.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD clusters.provisioning.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD clusters.provisioning.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD rkeclusters.rke.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD custommachines.rke.cattle.io
2022/01/04 09:23:33 [INFO] Applying CRD clusters.cluster.x-k8s.io
2022/01/04 09:23:33 [INFO] Applying CRD machinedeployments.cluster.x-k8s.io
2022/01/04 09:23:33 [INFO] Applying CRD machinehealthchecks.cluster.x-k8s.io
2022/01/04 09:23:33 [INFO] Applying CRD machines.cluster.x-k8s.io
2022/01/04 09:23:33 [INFO] Applying CRD machinesets.cluster.x-k8s.io
2022/01/04 09:23:36 [FATAL] k3s exited with: exit status 1
2022/01/04 09:23:53 [INFO] Rancher version v2.6.3 (3c1d5fac3) is starting
2022/01/04 09:23:53 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2022/01/04 09:23:53 [INFO] Listening on /tmp/log.sock
2022/01/04 09:23:53 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6443/version?timeout=15m0s": dial tcp 127.0.0.1:6443: connect: connection refused
2022/01/04 09:23:55 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6443/version?timeout=15m0s": dial tcp 127.0.0.1:6443: connect: connection refused
2022/01/04 09:23:57 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6443/version?timeout=15m0s": dial tcp 127.0.0.1:6443: connect: connection refused
2022/01/04 09:23:59 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6443/version?timeout=15m0s": dial tcp 127.0.0.1:6443: connect: connection refused
2022/01/04 09:24:01 [INFO] Waiting for server to become available: an error on the server ("apiserver not ready") has prevented the request from succeeding
2022/01/04 09:24:03 [INFO] Waiting for server to become available: an error on the server ("apiserver not ready") has prevented the request from succeeding
2022/01/04 09:24:05 [INFO] Waiting for server to become available: an error on the server ("apiserver not ready") has prevented the request from succeeding
2022/01/04 09:24:17 [FATAL] k3s exited with: exit status 1
已邀请:

主机的CPU、内存分别是多少?

2.6里面,内置k3s的日志保存到一个文件中,不会输出到std out/err。


获取内置k3s的日志,进入rancher-server容器中:/var/lib/rancher/k3s.log


可以根据这里面的日志信息,找到进一步troubleshooting的线索。

W0105 07:22:18.255472     111 proxier.go:663] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0105 07:22:18.255771 111 proxier.go:663] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0105 07:22:18.255993 111 proxier.go:663] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0105 07:22:18.256178 111 proxier.go:663] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
time="2022-01-05T07:22:18.256627326Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: can't change directory to '5.13.0-21-generic': No such file or directory`, error: exit status 1"
W0105 07:22:18.271102 111 manager.go:159] Cannot detect current cgroup on cgroup v2
I0105 07:22:18.271297 111 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
E0105 07:22:18.272194 111 node.go:161] Failed to retrieve node info: nodes "local-node" not found
time="2022-01-05T07:22:18.908402355Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
E0105 07:22:19.398497 111 node.go:161] Failed to retrieve node info: nodes "local-node" not found
time="2022-01-05T07:22:19.915114636Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
time="2022-01-05T07:22:20.920962707Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
time="2022-01-05T07:22:21.249894989Z" level=info msg="Stopped tunnel to 127.0.0.1:6443"
time="2022-01-05T07:22:21.249971453Z" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
time="2022-01-05T07:22:21.249953097Z" level=info msg="Connecting to proxy" url="wss://172.17.0.6:6443/v1-k3s/connect"
time="2022-01-05T07:22:21.250168675Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
time="2022-01-05T07:22:21.255770246Z" level=info msg="Handling backend connection request [local-node]"
E0105 07:22:21.444435 111 node.go:161] Failed to retrieve node info: nodes "local-node" not found
time="2022-01-05T07:22:21.927579093Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:22.436Z","caller":"grpclog/grpclog.go:60","msg":"grpc: addrConn.createTransport failed to connect to {http://127.0.0.1:2399 <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2399: connect: connection refused\". Reconnecting..."}
time="2022-01-05T07:22:22.932488636Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
W0105 07:22:23.301368 111 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
I0105 07:22:23.320293 111 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
I0105 07:22:23.320632 111 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I0105 07:22:23.320773 111 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
I0105 07:22:23.320834 111 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
I0105 07:22:23.320863 111 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
I0105 07:22:23.320881 111 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
I0105 07:22:23.321151 111 kubelet.go:404] "Attempting to sync node with API server"
I0105 07:22:23.321199 111 kubelet.go:272] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
I0105 07:22:23.321242 111 kubelet.go:283] "Adding apiserver pod source"
I0105 07:22:23.321275 111 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
I0105 07:22:23.322623 111 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="containerd" version="v1.4.12-k3s1" apiVersion="v1alpha2"
I0105 07:22:23.323404 111 server.go:1191] "Started kubelet"
I0105 07:22:23.323557 111 server.go:149] "Starting to listen" address="0.0.0.0" port=10250
E0105 07:22:23.324480 111 cri_stats_provider.go:369] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
E0105 07:22:23.324563 111 kubelet.go:1306] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
I0105 07:22:23.325244 111 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
I0105 07:22:23.325397 111 volume_manager.go:271] "Starting Kubelet Volume Manager"
I0105 07:22:23.325514 111 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
I0105 07:22:23.327018 111 server.go:409] "Adding debug handlers to kubelet server"
I0105 07:22:23.337102 111 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
I0105 07:22:23.346390 111 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv6
I0105 07:22:23.346404 111 status_manager.go:157] "Starting to sync pod status with apiserver"
I0105 07:22:23.346411 111 kubelet.go:1846] "Starting kubelet main sync loop"
E0105 07:22:23.346435 111 kubelet.go:1870] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
E0105 07:22:23.425656 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:23.447470 111 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
{"level":"warn","ts":"2022-01-05T07:22:23.473Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"129.562146ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/minions/local-node\" ","response":"range_response_count:0 size:5"}
{"level":"info","ts":"2022-01-05T07:22:23.473Z","caller":"traceutil/trace.go:145","msg":"trace[747994718] range","detail":"{range_begin:/registry/minions/local-node; range_end:; response_count:0; response_revision:376; }","duration":"129.689383ms","start":"2022-01-05T07:22:23.343Z","end":"2022-01-05T07:22:23.473Z","steps":["trace[747994718] 'agreement among raft nodes before linearized reading' (duration: 79.852674ms)","trace[747994718] 'range keys from in-memory index tree' (duration: 49.666504ms)"]}
E0105 07:22:23.474999 111 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"local-node\" not found" node="local-node"
E0105 07:22:23.526166 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:23.626950 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:23.647984 111 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
E0105 07:22:23.728113 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:23.828238 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:23.927855 111 csi_plugin.go:295] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "local-node" not found
E0105 07:22:23.928694 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
time="2022-01-05T07:22:23.937136131Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
E0105 07:22:24.029893 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:24.049060 111 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
E0105 07:22:24.130488 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:24.231085 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
I0105 07:22:24.322387 111 apiserver.go:52] "Watching apiserver"
E0105 07:22:24.331912 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:24.432268 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:24.533327 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:24.634235 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:24.665Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"101.46605ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/management.cattle.io/features/embedded-cluster-api\" ","response":"range_response_count:0 size:5"}
{"level":"info","ts":"2022-01-05T07:22:24.665Z","caller":"traceutil/trace.go:145","msg":"trace[1923196392] range","detail":"{range_begin:/registry/management.cattle.io/features/embedded-cluster-api; range_end:; response_count:0; response_revision:379; }","duration":"101.594948ms","start":"2022-01-05T07:22:24.564Z","end":"2022-01-05T07:22:24.665Z","steps":["trace[1923196392] 'agreement among raft nodes before linearized reading' (duration: 27.222987ms)","trace[1923196392] 'range keys from in-memory index tree' (duration: 74.198883ms)"]}
E0105 07:22:24.734950 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:24.835773 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:24.849642 111 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
{"level":"warn","ts":"2022-01-05T07:22:24.860Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"103.440153ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/management.cattle.io/features/multi-cluster-management-agent\" ","response":"range_response_count:0 size:5"}
{"level":"warn","ts":"2022-01-05T07:22:24.860Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"104.492673ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/leases/kube-system/cloud-controller-manager\" ","response":"range_response_count:1 size:490"}
{"level":"info","ts":"2022-01-05T07:22:24.860Z","caller":"traceutil/trace.go:145","msg":"trace[507055350] range","detail":"{range_begin:/registry/management.cattle.io/features/multi-cluster-management-agent; range_end:; response_count:0; response_revision:382; }","duration":"103.54992ms","start":"2022-01-05T07:22:24.757Z","end":"2022-01-05T07:22:24.860Z","steps":["trace[507055350] 'agreement among raft nodes before linearized reading' (duration: 29.226837ms)","trace[507055350] 'range keys from in-memory index tree' (duration: 74.180092ms)"]}
{"level":"info","ts":"2022-01-05T07:22:24.860Z","caller":"traceutil/trace.go:145","msg":"trace[326728882] range","detail":"{range_begin:/registry/leases/kube-system/cloud-controller-manager; range_end:; response_count:1; response_revision:382; }","duration":"104.580072ms","start":"2022-01-05T07:22:24.756Z","end":"2022-01-05T07:22:24.860Z","steps":["trace[326728882] 'agreement among raft nodes before linearized reading' (duration: 30.196146ms)","trace[326728882] 'range keys from in-memory index tree' (duration: 74.252211ms)"]}
E0105 07:22:24.936626 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
time="2022-01-05T07:22:24.947248505Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
E0105 07:22:25.037310 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:25.137775 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:25.238683 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:25.263Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"103.518167ms","expected-duration":"100ms","prefix":"","request":"header:<ID:4428865998624453566 username:\"etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/validatingwebhookconfigurations/validating-webhook-configuration\" mod_revision:0 > success:<request_put:<key:\"/registry/validatingwebhookconfigurations/validating-webhook-configuration\" value_size:5606 >> failure:<>>","response":"size:16"}
E0105 07:22:25.339494 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:25.439968 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:25.492Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"166.360724ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/minions/local-node\" ","response":"range_response_count:0 size:5"}
{"level":"info","ts":"2022-01-05T07:22:25.492Z","caller":"traceutil/trace.go:145","msg":"trace[360493669] range","detail":"{range_begin:/registry/minions/local-node; range_end:; response_count:0; response_revision:391; }","duration":"166.479844ms","start":"2022-01-05T07:22:25.326Z","end":"2022-01-05T07:22:25.492Z","steps":["trace[360493669] 'agreement among raft nodes before linearized reading' (duration: 67.286391ms)","trace[360493669] 'range keys from in-memory index tree' (duration: 99.027195ms)"]}
{"level":"warn","ts":"2022-01-05T07:22:25.493Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"142.766294ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" ","response":"range_response_count:5 size:14778"}
{"level":"info","ts":"2022-01-05T07:22:25.493Z","caller":"traceutil/trace.go:145","msg":"trace[1277063742] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/; range_end:/registry/apiextensions.k8s.io/customresourcedefinitions0; response_count:5; response_revision:392; }","duration":"142.877506ms","start":"2022-01-05T07:22:25.350Z","end":"2022-01-05T07:22:25.493Z","steps":["trace[1277063742] 'agreement among raft nodes before linearized reading' (duration: 142.57337ms)"]}
E0105 07:22:25.494670 111 node.go:161] Failed to retrieve node info: nodes "local-node" not found
E0105 07:22:25.540282 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:25.641482 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:25.701Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"141.724815ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" ","response":"range_response_count:6 size:18125"}
{"level":"info","ts":"2022-01-05T07:22:25.701Z","caller":"traceutil/trace.go:145","msg":"trace[950787893] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/; range_end:/registry/apiextensions.k8s.io/customresourcedefinitions0; response_count:6; response_revision:395; }","duration":"141.856175ms","start":"2022-01-05T07:22:25.559Z","end":"2022-01-05T07:22:25.701Z","steps":["trace[950787893] 'agreement among raft nodes before linearized reading' (duration: 141.577576ms)"]}
E0105 07:22:25.742495 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:25.842851 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:25.884Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"117.990821ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/csinodes/local-node\" ","response":"range_response_count:0 size:5"}
{"level":"info","ts":"2022-01-05T07:22:25.884Z","caller":"traceutil/trace.go:145","msg":"trace[1758534523] range","detail":"{range_begin:/registry/csinodes/local-node; range_end:; response_count:0; response_revision:397; }","duration":"118.034091ms","start":"2022-01-05T07:22:25.766Z","end":"2022-01-05T07:22:25.884Z","steps":["trace[1758534523] 'agreement among raft nodes before linearized reading' (duration: 43.990754ms)","trace[1758534523] 'range keys from in-memory index tree' (duration: 73.980067ms)"]}
{"level":"warn","ts":"2022-01-05T07:22:25.884Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"115.280169ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" ","response":"range_response_count:7 size:21989"}
{"level":"info","ts":"2022-01-05T07:22:25.884Z","caller":"traceutil/trace.go:145","msg":"trace[446934280] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/; range_end:/registry/apiextensions.k8s.io/customresourcedefinitions0; response_count:7; response_revision:398; }","duration":"115.31071ms","start":"2022-01-05T07:22:25.769Z","end":"2022-01-05T07:22:25.884Z","steps":["trace[446934280] 'agreement among raft nodes before linearized reading' (duration: 115.217726ms)"]}
E0105 07:22:25.886558 111 csi_plugin.go:295] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "local-node" not found
E0105 07:22:25.943364 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:26.043850 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:26.107Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"156.900846ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/minions/local-node\" ","response":"range_response_count:0 size:5"}
{"level":"warn","ts":"2022-01-05T07:22:26.107Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"193.91192ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/clusterregistrationtokens.management.cattle.io\" ","response":"range_response_count:1 size:4069"}
{"level":"info","ts":"2022-01-05T07:22:26.107Z","caller":"traceutil/trace.go:145","msg":"trace[2089120919] range","detail":"{range_begin:/registry/minions/local-node; range_end:; response_count:0; response_revision:401; }","duration":"157.023864ms","start":"2022-01-05T07:22:25.950Z","end":"2022-01-05T07:22:26.107Z","steps":["trace[2089120919] 'agreement among raft nodes before linearized reading' (duration: 57.818709ms)","trace[2089120919] 'range keys from in-memory index tree' (duration: 99.031894ms)"]}
{"level":"info","ts":"2022-01-05T07:22:26.107Z","caller":"traceutil/trace.go:145","msg":"trace[2131808485] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/clusterregistrationtokens.management.cattle.io; range_end:; response_count:1; response_revision:401; }","duration":"194.026311ms","start":"2022-01-05T07:22:25.913Z","end":"2022-01-05T07:22:26.107Z","steps":["trace[2131808485] 'agreement among raft nodes before linearized reading' (duration: 94.784569ms)","trace[2131808485] 'range keys from in-memory index tree' (duration: 99.061185ms)"]}
{"level":"warn","ts":"2022-01-05T07:22:26.107Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"146.488122ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/masterleases/\" range_end:\"/registry/masterleases0\" ","response":"range_response_count:1 size:129"}
{"level":"info","ts":"2022-01-05T07:22:26.107Z","caller":"traceutil/trace.go:145","msg":"trace[2051841665] range","detail":"{range_begin:/registry/masterleases/; range_end:/registry/masterleases0; response_count:1; response_revision:402; }","duration":"146.600511ms","start":"2022-01-05T07:22:25.961Z","end":"2022-01-05T07:22:26.107Z","steps":["trace[2051841665] 'agreement among raft nodes before linearized reading' (duration: 146.352045ms)"]}
{"level":"warn","ts":"2022-01-05T07:22:26.108Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"125.211452ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/csinodes/local-node\" ","response":"range_response_count:0 size:5"}
{"level":"info","ts":"2022-01-05T07:22:26.108Z","caller":"traceutil/trace.go:145","msg":"trace[505912812] range","detail":"{range_begin:/registry/csinodes/local-node; range_end:; response_count:0; response_revision:402; }","duration":"125.311345ms","start":"2022-01-05T07:22:25.982Z","end":"2022-01-05T07:22:26.108Z","steps":["trace[505912812] 'agreement among raft nodes before linearized reading' (duration: 125.068438ms)"]}
time="2022-01-05T07:22:26.109589730Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
E0105 07:22:26.144426 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:26.245377 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:26.316Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"190.394755ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/minions/local-node\" ","response":"range_response_count:0 size:5"}
{"level":"info","ts":"2022-01-05T07:22:26.316Z","caller":"traceutil/trace.go:145","msg":"trace[238708033] range","detail":"{range_begin:/registry/minions/local-node; range_end:; response_count:0; response_revision:404; }","duration":"190.517821ms","start":"2022-01-05T07:22:26.125Z","end":"2022-01-05T07:22:26.316Z","steps":["trace[238708033] 'agreement among raft nodes before linearized reading' (duration: 91.481635ms)","trace[238708033] 'range keys from in-memory index tree' (duration: 98.858285ms)"]}
{"level":"warn","ts":"2022-01-05T07:22:26.316Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"143.440687ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" ","response":"range_response_count:9 size:29614"}
{"level":"info","ts":"2022-01-05T07:22:26.316Z","caller":"traceutil/trace.go:145","msg":"trace[1344793129] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/; range_end:/registry/apiextensions.k8s.io/customresourcedefinitions0; response_count:9; response_revision:405; }","duration":"143.550158ms","start":"2022-01-05T07:22:26.173Z","end":"2022-01-05T07:22:26.316Z","steps":["trace[1344793129] 'agreement among raft nodes before linearized reading' (duration: 143.070866ms)"]}
E0105 07:22:26.346221 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:26.447306 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:26.450183 111 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
{"level":"warn","ts":"2022-01-05T07:22:26.524Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"143.822975ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" ","response":"range_response_count:10 size:33051"}
{"level":"info","ts":"2022-01-05T07:22:26.524Z","caller":"traceutil/trace.go:145","msg":"trace[459551804] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/; range_end:/registry/apiextensions.k8s.io/customresourcedefinitions0; response_count:10; response_revision:408; }","duration":"143.945327ms","start":"2022-01-05T07:22:26.380Z","end":"2022-01-05T07:22:26.524Z","steps":["trace[459551804] 'agreement among raft nodes before linearized reading' (duration: 69.605812ms)","trace[459551804] 'range keys from in-memory index tree' (duration: 74.036901ms)"]}
E0105 07:22:26.547929 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:26.648938 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:26.743Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"152.694184ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" ","response":"range_response_count:11 size:38448"}
{"level":"info","ts":"2022-01-05T07:22:26.743Z","caller":"traceutil/trace.go:145","msg":"trace[143616330] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/; range_end:/registry/apiextensions.k8s.io/customresourcedefinitions0; response_count:11; response_revision:411; }","duration":"152.829654ms","start":"2022-01-05T07:22:26.590Z","end":"2022-01-05T07:22:26.743Z","steps":["trace[143616330] 'agreement among raft nodes before linearized reading' (duration: 43.30783ms)","trace[143616330] 'range keys from in-memory index tree' (duration: 109.244388ms)"]}
{"level":"warn","ts":"2022-01-05T07:22:26.743Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"109.322039ms","expected-duration":"100ms","prefix":"","request":"header:<ID:4428865998624453626 username:\"etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/clusterrepos.catalog.cattle.io\" mod_revision:411 > success:<request_put:<key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/clusterrepos.catalog.cattle.io\" value_size:5303 >> failure:<request_range:<key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/clusterrepos.catalog.cattle.io\" > >>","response":"size:16"}
E0105 07:22:26.749138 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:26.849907 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:26.943Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"190.611655ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/operations.catalog.cattle.io\" ","response":"range_response_count:1 size:3885"}
{"level":"info","ts":"2022-01-05T07:22:26.943Z","caller":"traceutil/trace.go:145","msg":"trace[1856980711] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/operations.catalog.cattle.io; range_end:; response_count:1; response_revision:413; }","duration":"190.651253ms","start":"2022-01-05T07:22:26.752Z","end":"2022-01-05T07:22:26.943Z","steps":["trace[1856980711] 'agreement among raft nodes before linearized reading' (duration: 116.697533ms)","trace[1856980711] 'range keys from in-memory index tree' (duration: 73.879411ms)"]}
E0105 07:22:26.950885 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:27.051426 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:27.151970 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:27.165Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"141.454552ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/apps.catalog.cattle.io\" ","response":"range_response_count:1 size:5693"}
{"level":"info","ts":"2022-01-05T07:22:27.165Z","caller":"traceutil/trace.go:145","msg":"trace[725350926] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/apps.catalog.cattle.io; range_end:; response_count:1; response_revision:417; }","duration":"141.605204ms","start":"2022-01-05T07:22:27.024Z","end":"2022-01-05T07:22:27.165Z","steps":["trace[725350926] 'agreement among raft nodes before linearized reading' (duration: 42.65527ms)","trace[725350926] 'range keys from in-memory index tree' (duration: 98.725348ms)"]}
time="2022-01-05T07:22:27.167343721Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
E0105 07:22:27.253121 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:27.349Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"117.871809ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/fleetworkspaces.management.cattle.io\" ","response":"range_response_count:1 size:2450"}
{"level":"info","ts":"2022-01-05T07:22:27.349Z","caller":"traceutil/trace.go:145","msg":"trace[2029177786] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/fleetworkspaces.management.cattle.io; range_end:; response_count:1; response_revision:420; }","duration":"117.994087ms","start":"2022-01-05T07:22:27.231Z","end":"2022-01-05T07:22:27.349Z","steps":["trace[2029177786] 'agreement among raft nodes before linearized reading' (duration: 43.765359ms)","trace[2029177786] 'range keys from in-memory index tree' (duration: 74.03381ms)"]}
E0105 07:22:27.351102 111 csi_plugin.go:295] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "local-node" not found
E0105 07:22:27.357176 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:27.457787 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:27.558527 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:27.588Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"195.166817ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/clusters.fleet.cattle.io\" ","response":"range_response_count:1 size:8709"}
{"level":"info","ts":"2022-01-05T07:22:27.589Z","caller":"traceutil/trace.go:145","msg":"trace[1942019702] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/clusters.fleet.cattle.io; range_end:; response_count:1; response_revision:425; }","duration":"195.327864ms","start":"2022-01-05T07:22:27.393Z","end":"2022-01-05T07:22:27.589Z","steps":["trace[1942019702] 'agreement among raft nodes before linearized reading' (duration: 195.055509ms)"]}
E0105 07:22:27.658651 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:27.758775 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:27.797Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"140.951142ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/bundles.fleet.cattle.io\" ","response":"range_response_count:1 size:16153"}
{"level":"info","ts":"2022-01-05T07:22:27.797Z","caller":"traceutil/trace.go:145","msg":"trace[1402366736] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/bundles.fleet.cattle.io; range_end:; response_count:1; response_revision:427; }","duration":"141.070416ms","start":"2022-01-05T07:22:27.656Z","end":"2022-01-05T07:22:27.797Z","steps":["trace[1402366736] 'agreement among raft nodes before linearized reading' (duration: 41.869024ms)","trace[1402366736] 'range keys from in-memory index tree' (duration: 98.969479ms)"]}
E0105 07:22:27.859361 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:27.960367 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:28.016Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"138.767525ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/managedcharts.management.cattle.io\" ","response":"range_response_count:1 size:13454"}
{"level":"info","ts":"2022-01-05T07:22:28.016Z","caller":"traceutil/trace.go:145","msg":"trace[1445586739] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/managedcharts.management.cattle.io; range_end:; response_count:1; response_revision:430; }","duration":"138.815017ms","start":"2022-01-05T07:22:27.878Z","end":"2022-01-05T07:22:28.016Z","steps":["trace[1445586739] 'agreement among raft nodes before linearized reading' (duration: 39.954985ms)","trace[1445586739] 'range keys from in-memory index tree' (duration: 98.771132ms)"]}
E0105 07:22:28.061084 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:28.161953 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
{"level":"warn","ts":"2022-01-05T07:22:28.236Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"146.038214ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/clusters.provisioning.cattle.io\" ","response":"range_response_count:1 size:13119"}
{"level":"info","ts":"2022-01-05T07:22:28.236Z","caller":"traceutil/trace.go:145","msg":"trace[1143837590] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/clusters.provisioning.cattle.io; range_end:; response_count:1; response_revision:433; }","duration":"146.191119ms","start":"2022-01-05T07:22:28.090Z","end":"2022-01-05T07:22:28.236Z","steps":["trace[1143837590] 'agreement among raft nodes before linearized reading' (duration: 46.835814ms)","trace[1143837590] 'range keys from in-memory index tree' (duration: 99.061732ms)"]}
{"level":"warn","ts":"2022-01-05T07:22:28.237Z","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"113.693138ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/minions/local-node\" ","response":"range_response_count:0 size:5"}
{"level":"info","ts":"2022-01-05T07:22:28.237Z","caller":"traceutil/trace.go:145","msg":"trace[179354569] range","detail":"{range_begin:/registry/minions/local-node; range_end:; response_count:0; response_revision:435; }","duration":"113.823064ms","start":"2022-01-05T07:22:28.123Z","end":"2022-01-05T07:22:28.237Z","steps":["trace[179354569] 'agreement among raft nodes before linearized reading' (duration: 113.581902ms)"]}
time="2022-01-05T07:22:28.239391744Z" level=info msg="Waiting for control-plane node local-node startup: nodes \"local-node\" not found"
E0105 07:22:28.262626 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
I0105 07:22:28.337665 111 kubelet_node_status.go:71] "Attempting to register node" node="local-node"
I0105 07:22:28.338557 111 cpu_manager.go:199] "Starting CPU manager" policy="none"
I0105 07:22:28.338598 111 cpu_manager.go:200] "Reconciling" reconcilePeriod="10s"
I0105 07:22:28.338646 111 state_mem.go:36] "Initialized new in-memory state store"
I0105 07:22:28.339596 111 state_mem.go:88] "Updated default CPUSet" cpuSet=""
I0105 07:22:28.339651 111 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
I0105 07:22:28.339674 111 policy_none.go:44] "None policy: Start"
E0105 07:22:28.362978 111 kubelet.go:2291] "Error getting node" err="node \"local-node\" not found"
E0105 07:22:28.363080 111 node_container_manager_linux.go:57] "Failed to create cgroup" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state" cgroupName=[kubepods]
E0105 07:22:28.363092 111 kubelet.go:1384] "Failed to start ContainerManager" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state"

降级使用cgroupv1就行,有些OS比较激进,直接默认v2。实际上从容器角度,v2还属于experimental。

要回复问题请先登录注册