踩到了 docker 一个 open 了 3 年+， 350+回复的 kennel 级神坑，如果是生产环境各位运维童鞋会怎么处理？
fzinfz · fzinfz · 2017-06-25 13:18:25 +08:00 · 16912 次点击这是一个创建于 2706 天前的主题，其中的信息可能已经有所发展或是发生改变。
unregister_netdevice: waiting for lo to become free. Usage count = #
https://github.com/moby/moby/issues/5618
貌似可能的原因很多，kernel 4.11 解决了一些：
https://github.com/coreos/bugs/issues/254
Another source of leakage has been fixed by torvalds/linux@f647821, which is part of 4.11.
torvalds/linux@b7c8487 is in the same batch of commits, is also in 4.11, and fixes another possible cause
https://github.com/moby/moby/issues/5618
貌似可能的原因很多，kernel 4.11 解决了一些：
https://github.com/coreos/bugs/issues/254
Another source of leakage has been fixed by torvalds/linux@f647821, which is part of 4.11.
torvalds/linux@b7c8487 is in the same batch of commits, is also in 4.11, and fixes another possible cause
 |
1
choury 2017-06-25 13:50:28 +08:00
我们的做法是甩锅给内核组，让他们修
|
 |
2
ryd994 2017-06-25 14:06:35 +08:00  2
几乎所有人都踩到的……
还好没影响我使用，除了 dmesg spamming 以外 既然有那么多内核大佬都在修，我们这些小菜只能等了 现在你知道运维找大师做法给服务器开光有啥用了吗？ 哦，还有个办法是买 RH 订阅然后甩锅给 RH |
 |
3
xss 2017-06-25 14:29:14 +08:00
我也遇到了, 然而具体有什么影响?
|
 |
4
cnnblike 2017-06-25 15:15:41 +08:00
我也遇到了，神烦
|
 |
5
mritd 2017-06-25 16:07:30 +08:00 via iPhone
很正常
|
 |
6
fuxkcsdn 2017-06-25 20:40:57 +08:00 via iPhone
遇到过+1
|
 |
7
freestyle 2017-06-25 20:45:14 +08:00 via iPhone
遇到过 但不清楚有什么影响
|
 |
8
htfy96 2017-06-25 21:34:56 +08:00
还有那个 userland-proxy，必须性能很差地在用户态搞 NAT，原因之一也是 kernel bug
|
 |
9
dusheng 2017-06-25 23:33:10 +08:00
刚遇到过
|
 |
10
SharkIng 2017-06-25 23:59:47 +08:00 via iPhone
这个问题把我们害惨了，不过据说最新 kernel 修复了？
|
 |
11
fzinfz OP @SharkIng 具体有啥影响？最新 kernel 有做了修复，不过不知是否完全修复。
这个错误我是刚重启的时候连不上 SSH 进 hyper-v console 才发现的，重启 VM 后 SSH 正常但是还是报了 2 条 unregister_netdevice，不知是否相关。 [以下有点乱，仅作个人记录] 看了下 dmesg，还伴随着很多“ docker0: port #(veth......) entered blocking/disabled state ”的 log，看了下 GCE/Vultr KVM 的 VM 也有 blocking/disabled log （但是没有 unregister_netdevice ），猜测是 packetbeat 抓包导致的。（没有 packetbeat 抓包的机子就没有这些 log。） PS1：所有 container 都是--net host 运行 PS2：报 unregister_netdevice 的 VM 用的是 B150 主板板载 Realtek 网卡... PS3：目前 kernel 4.10.0-19-generic，准备升 4.11 看下 贴下 unregister_netdevice 那台机器的 dmesg，当作记录： [ 8.263327] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. [ 8.265021] Bridge firewalling registered [ 8.270259] nf_conntrack version 0.5.0 (65536 buckets, 262144 max) [ 8.352304] Initializing XFRM netlink socket [ 8.395477] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready [ 8.934882] aufs au_opts_verify:1585:dockerd[1619]: dirperm1 breaks the protection by the permission bits on the lower branch [ 9.152098] docker0: port 1(veth41c7ef0) entered blocking state [ 9.152098] docker0: port 1(veth41c7ef0) entered disabled state [ 9.152267] device veth41c7ef0 entered promiscuous mode [ 9.152695] IPv6: ADDRCONF(NETDEV_UP): veth41c7ef0: link is not ready [ 9.152700] docker0: port 1(veth41c7ef0) entered blocking state [ 9.152701] docker0: port 1(veth41c7ef0) entered forwarding state [ 9.152771] docker0: port 1(veth41c7ef0) entered disabled state [ 11.332410] eth0: renamed from vethe136227 [ 11.348824] IPv6: ADDRCONF(NETDEV_CHANGE): veth41c7ef0: link becomes ready [ 11.348896] docker0: port 1(veth41c7ef0) entered blocking state [ 11.348897] docker0: port 1(veth41c7ef0) entered forwarding state [ 11.348985] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready [ 12.766843] docker0: port 2(veth49d7060) entered blocking state [ 12.766844] docker0: port 2(veth49d7060) entered disabled state [ 12.767074] device veth49d7060 entered promiscuous mode [ 12.767203] IPv6: ADDRCONF(NETDEV_UP): veth49d7060: link is not ready [ 12.767206] docker0: port 2(veth49d7060) entered blocking state [ 12.767207] docker0: port 2(veth49d7060) entered forwarding state [ 12.767329] docker0: port 2(veth49d7060) entered disabled state [ 13.780687] eth0: renamed from vethc063ec5 [ 13.792810] IPv6: ADDRCONF(NETDEV_CHANGE): veth49d7060: link becomes ready [ 13.792887] docker0: port 2(veth49d7060) entered blocking state [ 13.792888] docker0: port 2(veth49d7060) entered forwarding state [ 52.227899] hv_balloon: INFO_TYPE_MAX_PAGE_CNT = 3072000 [ 56.286544] hv_utils: KVP IC version 4.0 [ 910.342263] docker0: port 3(veth622db14) entered blocking state [ 910.342265] docker0: port 3(veth622db14) entered disabled state [ 910.342459] device veth622db14 entered promiscuous mode [ 910.342573] IPv6: ADDRCONF(NETDEV_UP): veth622db14: link is not ready [ 910.468949] eth0: renamed from vethf117385 [ 910.485201] IPv6: ADDRCONF(NETDEV_CHANGE): veth622db14: link becomes ready [ 910.485297] docker0: port 3(veth622db14) entered blocking state [ 910.485299] docker0: port 3(veth622db14) entered forwarding state [ 924.891826] docker0: port 3(veth622db14) entered disabled state [ 924.891832] vethf117385: renamed from eth0 [ 924.933207] docker0: port 3(veth622db14) entered disabled state [ 924.935302] device veth622db14 left promiscuous mode [ 924.935306] docker0: port 3(veth622db14) entered disabled state [ 1088.348123] unregister_netdevice: waiting for lo to become free. Usage count = 1 [ 2182.977980] docker0: port 3(vethe1afdfb) entered blocking state [ 2182.977981] docker0: port 3(vethe1afdfb) entered disabled state [ 2182.978204] device vethe1afdfb entered promiscuous mode [ 2182.978357] IPv6: ADDRCONF(NETDEV_UP): vethe1afdfb: link is not ready [ 2183.076738] eth0: renamed from vethfcb6742 [ 2183.094282] IPv6: ADDRCONF(NETDEV_CHANGE): vethe1afdfb: link becomes ready [ 2183.094481] docker0: port 3(vethe1afdfb) entered blocking state [ 2183.094482] docker0: port 3(vethe1afdfb) entered forwarding state [ 2194.980883] docker0: port 3(vethe1afdfb) entered disabled state [ 2194.981195] vethfcb6742: renamed from eth0 [ 2195.018586] docker0: port 3(vethe1afdfb) entered disabled state [ 2195.021027] device vethe1afdfb left promiscuous mode [ 2195.021031] docker0: port 3(vethe1afdfb) entered disabled state [ 3021.669760] docker0: port 3(vethb0d2857) entered blocking state [ 3021.669760] docker0: port 3(vethb0d2857) entered disabled state [ 3021.670990] device vethb0d2857 entered promiscuous mode [ 3021.671312] IPv6: ADDRCONF(NETDEV_UP): vethb0d2857: link is not ready [ 3021.784817] eth0: renamed from veth12e4511 [ 3021.804650] IPv6: ADDRCONF(NETDEV_CHANGE): vethb0d2857: link becomes ready [ 3021.804739] docker0: port 3(vethb0d2857) entered blocking state [ 3021.804740] docker0: port 3(vethb0d2857) entered forwarding state [ 3022.613543] docker0: port 3(vethb0d2857) entered disabled state [ 3022.613652] veth12e4511: renamed from eth0 [ 3022.651080] docker0: port 3(vethb0d2857) entered disabled state [ 3022.656981] device vethb0d2857 left promiscuous mode [ 3022.657016] docker0: port 3(vethb0d2857) entered disabled state [ 3357.591305] docker0: port 3(vethb900f81) entered blocking state [ 3357.591306] docker0: port 3(vethb900f81) entered disabled state [ 3357.591425] device vethb900f81 entered promiscuous mode [ 3357.591511] IPv6: ADDRCONF(NETDEV_UP): vethb900f81: link is not ready [ 3357.696529] eth0: renamed from vethf79a1ab [ 3357.716768] IPv6: ADDRCONF(NETDEV_CHANGE): vethb900f81: link becomes ready [ 3357.716854] docker0: port 3(vethb900f81) entered blocking state [ 3357.716855] docker0: port 3(vethb900f81) entered forwarding state [ 3407.848335] vethf79a1ab: renamed from eth0 [ 3407.881076] docker0: port 3(vethb900f81) entered disabled state [ 3407.898598] docker0: port 3(vethb900f81) entered disabled state [ 3407.904083] device vethb900f81 left promiscuous mode [ 3407.904104] docker0: port 3(vethb900f81) entered disabled state [ 3603.292153] unregister_netdevice: waiting for lo to become free. Usage count = 1 |
|
12
SharkIng 2017-06-26 03:35:07 +08:00 1
@fzinfz #11 Kernel Panic, 表现类似于服务器断网，因为我们程序当初跑在 Docker 里面的，造成所有流出 Docker 的流量都无法流出。然后 Docker 的程序基本上就死了（主要是因为无法用了）最后我们就把整个程序做成 Dockerless 的了
Kernel 修复大概是这部分，但是我们没有测试： https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.4.22 |
 |
13
l142857 2017-06-26 09:48:00 +08:00
|
 |
14
ixiaohei 2017-07-04 09:24:15 +08:00 1
貌似我们公司也碰到了，redhat 给我们的建议是升级为 RedHat 发行的 docker 及补充包来解决
|
 |
15
jessynt 2017-07-22 19:09:08 +08:00
踩到
unregister_netdevice: waiting for lo to become free |
 |
16
0312birdzhang 2018-05-14 11:22:12 +08:00
kernel:unregister_netdevice: waiting for lo to become free. Usage count = 1
希望是踩到这个坑了，这样就可以甩锅了  |
 |
17
caimaoy 2018-12-28 16:16:26 +08:00
踩锅 + 1
|
 |
18
AngryPanda 2019-01-04 11:38:48 +08:00
踩锅 + 1
|
 |
19
dozer47528 2019-02-22 17:52:26 +08:00
踩锅+1
|
 |
20
1073850525 2019-04-28 19:11:06 +08:00
踩锅 + 1
|
 |
21
atywz 2019-04-29 21:18:16 +08:00
甩不了锅 怎么办 55
|
 |
22
vincent927 2019-05-07 10:21:48 +08:00
刚踩到，就升级了下 rancher 然后就出现了···有什么影响吗？解决办法是？
|
 |
23
wwek 2022-03-30 17:15:32 +08:00
踩锅 + 1
|
Select Language