runtime: failed to create new OS thread

[Kry1702]

What happened?

[root@m01 log]# kubectl get pod -A
runtime: failed to create new OS thread (have 3 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

runtime stack:
runtime.throw(0x1c1d651, 0x9)
/usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.newosproc(0xc0000b6400)
/usr/local/go/src/runtime/os_linux.go:161 +0x1c5
runtime.newm1(0xc0000b6400)
/usr/local/go/src/runtime/proc.go:1843 +0xdd
runtime.newm(0x1d07a08, 0xc00004a800, 0x2)
/usr/local/go/src/runtime/proc.go:1822 +0x9b
runtime.startm(0x0, 0xc000048001)
/usr/local/go/src/runtime/proc.go:1979 +0xc9
runtime.wakep()
/usr/local/go/src/runtime/proc.go:2067 +0x66
runtime.newproc.func1()
/usr/local/go/src/runtime/proc.go:3561 +0x97
runtime.systemstack(0x46c414)
/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1116

goroutine 1 [running, locked to thread]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0000b26f0 sp=0xc0000b26e8 pc=0x46c540
runtime.newproc(0x0, 0x1d078c8)
/usr/local/go/src/runtime/proc.go:3554 +0x6e fp=0xc0000b2738 sp=0xc0000b26f0 pc=0x44214e
runtime.init.6()
/usr/local/go/src/runtime/proc.go:243 +0x35 fp=0xc0000b2758 sp=0xc0000b2738 pc=0x439e35
runtime.doInit(0x2b17e80)
/usr/local/go/src/runtime/proc.go:5652 +0x8a fp=0xc0000b2788 sp=0xc0000b2758 pc=0x446fca
runtime.main()
/usr/local/go/src/runtime/proc.go:151 +0xd9 fp=0xc0000b27e0 sp=0xc0000b2788 pc=0x439b59
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0000b27e8 sp=0xc0000b27e0 pc=0x46e181

What did you expect to happen?

Output Pod information.

How can we reproduce it (as minimally and precisely as possible)?

I don't know how to repeat the problem.

Anything else we need to know?

$ ulimit -u
655350
$ ulimit -n
655350

systemctl edit kubelet.service
[Service]
LimitNOFILE=infinity
LimitNPROC=infinity

Kubernetes version

$ kubectl version
v 1.19.16

Cloud provider

Physical server

OS version

# On Linux:
$ cat /etc/os-release
CentOS Linux 7 (Core)

$ uname -a
linux m01 4.20.3-1.el7.elrepo.x86_64  x86_64  GNU/Linux

Install tools

kubeadm

Container runtime (CRI) and version (if applicable)

Docker v24.0.6

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Kry1702]

此问题没有签名标签。请使用以下命令之一添加适当的标签：

• /sig <group-name>
• /wg <group-name>
• /committee <group-name>

请参阅小组列表以获取可用的 SIG、工作组和委员会的列表。

使用 PR 评论与我互动的说明可在此处获得。如果您对我的行为有疑问或建议，请针对kubernetes-sigs/prow存储库提交问题。

• /sig apps
• /wg serving
• /committee steering

[neolit123]

/kind support
/sig cli

kubectl version
v 1.19.16

this is a very old version. try 1.30.

[Kry1702]

/kind support /sig cli

kubectl version
v 1.19.16

this is a very old version. try 1.30.

The version is too new,v1.19.16 This version has been running stably for 200 days.
The problem popped up twice this week, and the solution was to restart Docker.

The system level is set to be large enough (ulimit -n and ulimit -u), this problem will still occur. Since I installed the kubernetes cluster based on kubeadm, the service runs through the container, is it possible that ulimit -n and ulimit -u are restricted in the container system, i can't prove my suspicions.

[dims]

/close

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.