求大神解释： inotify 的 max_user_watches 耗尽，导致 No space left on device
SKYNE · SKYNE0 · 2020-04-24 18:50:14 +08:00 · 2139 次点击这是一个创建于 1660 天前的主题，其中的信息可能已经有所发展或是发生改变。
如题，查阅了很多资料，没找到具体为什么这个参数会导致 No space left on device 网上的资料多数是磁盘未满，inode 未满，然后调大此参数，参数生效即可解决该问题。 但对这个深层次的原因没有找到相关的资料，可能自己一根筋，总想知道为什么。。。
第 1 条附言 · 2020-04-25 16:13:33 +08:00
第 2 条附言 · 2020-04-25 16:53:58 +08:00
第 3 条附言 · 2020-04-25 17:45:49 +08:00
使用 strace -v -f 追踪系统调用时发生时的情况：
[pid 29498] inotify_init1(O_CLOEXEC <unfinished ...>
[pid 29499] close(3 <unfinished ...>
[pid 29498] <... inotify_init1 resumed> ) = 4
[pid 29499] <... close resumed> ) = 0
[pid 29498] inotify_add_watch(4, "/run/systemd/ask-password", IN_CLOSE_WRITE|IN_MOVED_TO <unfinished ...>
[pid 29499] open("/lib64/libmount.so.1", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 29498] <... inotify_add_watch resumed> ) = -1 ENOSPC (No space left on device)
[pid 29499] <... open resumed> ) = 3
[pid 29498] close(3 <unfinished ...>
[pid 29499] read(3, <unfinished ...>
[pid 29498] <... close resumed> ) = 0
[pid 29499] <... read resumed> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\237\0\0\0\0\0\0"..., 832) = 832
[pid 29498] close(4 <unfinished ...>
[pid 29499] fstat(3, <unfinished ...>
[pid 29498] <... close resumed> ) = 0
[pid 29499] <... fstat resumed> {st_dev=makedev(253, 1), st_ino=8396, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=544, st_size=277808, st_atime=2019/08/09-11:10:18, st_mtime=2019/08/09-11:10:18, st_ctime=2019/10/09-21:11:09.543361534}) = 0
[pid 29498] writev(2, [{"Error: No space left on device", 30}, {"\n", 1}], 2Error: No space left on device
<unfinished ...>
[pid 29499] mmap(NULL, 2370944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0 <unfinished ...>
看起来是 inotify_add_watch 函数导致的, 还是没看出来具体什么导致了 No space left on device
[pid 29498] inotify_init1(O_CLOEXEC <unfinished ...>
[pid 29499] close(3 <unfinished ...>
[pid 29498] <... inotify_init1 resumed> ) = 4
[pid 29499] <... close resumed> ) = 0
[pid 29498] inotify_add_watch(4, "/run/systemd/ask-password", IN_CLOSE_WRITE|IN_MOVED_TO <unfinished ...>
[pid 29499] open("/lib64/libmount.so.1", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 29498] <... inotify_add_watch resumed> ) = -1 ENOSPC (No space left on device)
[pid 29499] <... open resumed> ) = 3
[pid 29498] close(3 <unfinished ...>
[pid 29499] read(3, <unfinished ...>
[pid 29498] <... close resumed> ) = 0
[pid 29499] <... read resumed> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\237\0\0\0\0\0\0"..., 832) = 832
[pid 29498] close(4 <unfinished ...>
[pid 29499] fstat(3, <unfinished ...>
[pid 29498] <... close resumed> ) = 0
[pid 29499] <... fstat resumed> {st_dev=makedev(253, 1), st_ino=8396, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=544, st_size=277808, st_atime=2019/08/09-11:10:18, st_mtime=2019/08/09-11:10:18, st_ctime=2019/10/09-21:11:09.543361534}) = 0
[pid 29498] writev(2, [{"Error: No space left on device", 30}, {"\n", 1}], 2Error: No space left on device
<unfinished ...>
[pid 29499] mmap(NULL, 2370944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0 <unfinished ...>
看起来是 inotify_add_watch 函数导致的, 还是没看出来具体什么导致了 No space left on device
 |
1
superhan 2020-04-24 21:46:45 +08:00 via Android
研究一下文件系统 应该能提升一波
|
 |
3
fakevam 2020-04-25 17:09:50 +08:00
因为内核资源耗尽，统一返回 NOMEM，这个 errno 的语义不限于 MEM，一般是指某种资源
举个例子，你设置 cgroup，如果没设置 memory 允许的 numa node，直接去设置 cpu 的列表，也会返回这个 errno |
|
5
fakevam 2020-04-25 19:34:07 +08:00
意思就是说，内核很多报错，不是所有场合都和你理解的那个 errno 的字面意思一样
NO space left on device 未必是真的 no space left，可能只是某个抽象资源用完了 no mem 也是一个意思 |
Select Language