一个关于 Linux 进程的奇怪问题

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

现在注册

已注册用户请登录

Distributions

› Ubuntu

› Fedora

› CentOS

中文资源站

› 网易开源镜像站

这是一个创建于 1378 天前的主题，其中的信息可能已经有所发展或是发生改变。

是这样的，今天发现 k8s 有 2 台 master 节点 CPU 使用率有尖峰（ k8s 1.11 Linux 3.10 1.13.1 ），由于没有当时进程状态快照，只能用 top 每秒抓一次进程扔日志里。果然过了一段时间，发现了几个叫 docker-containerd-current 的进程 cpu 飙升导致 CPU 使用率高达 90%，但是我们通过 ps pgrep 等等手段都查不到这个进程了，所以考虑是不是当时突然启动了这些进程然后又死了，后来想从 /proc 里查有没有啥信息，奇怪的事发生了：ll -arth /proc|grep $PID 通过 pid 号对应文件夹查不到这个进程，但是 cd $PID 居然可以进到进程的文件夹里，然后我不死心，退回到 /proc 里又肉眼看了一遍文件夹，还是没找到。进到 /proc/$PID 内查看了 exe 对应的二进制文件，然后通过 ps -ef|grep $EXE_FILE 发现对应的进程启动时间还是去年。

这里的问题就是 /proc 下面的以 pid 命名的文件夹不应该是正在运行着的进程的么？进程确实死了，通过文件名也查不到了，但是为什么可以进到一个根本不存在的文件夹呢？

重点是图里面圈中的部分。

进程

proc

pid

文件夹

24 条回复 • 2021-02-10 15:08:47 +08:00

mikeguan

2021-02-07 18:01:37 +08:00 via Android

ls 显示目录的参数是-d

zhoudaiyu

2021-02-07 18:02:50 +08:00 via iPhone

@mikeguan 不用加这个参数也可以看文件夹啊

lcdtyph

2021-02-07 18:07:49 +08:00 via iPhone

ll 是什么的 alias

zhoudaiyu

2021-02-07 18:12:01 +08:00 via iPhone

@lcdtyph ls -l

mikeguan

2021-02-07 18:15:08 +08:00 via Android

@zhoudaiyu 不加也可以，输出太多更乱

zhoudaiyu

2021-02-07 18:16:54 +08:00 via iPhone

@mikeguan 但是问题是查不到文件夹为啥可以进去，而且进程都死了，不应该在 proc 下了啊

mikeguan

2021-02-07 18:23:11 +08:00 via Android

@zhoudaiyu 具体原因不清楚，可以用 lsof -p 14546 看下有没有结果

zhoudaiyu

2021-02-07 18:23:30 +08:00 via iPhone

@mikeguan 看过没有

Jirajine

2021-02-07 18:26:38 +08:00 via Android

可能是进程被隐藏了。

zhoudaiyu

2021-02-07 18:27:47 +08:00 via iPhone

@Jirajine 还有这种操作？那么有什么方法能看到被隐藏的进程吗？

lcdtyph

2021-02-07 18:29:43 +08:00 via iPhone

@zhoudaiyu
你看一下现在 procfs 的 mount options 是什么

Jirajine

2021-02-07 18:32:23 +08:00 via Android

@zhoudaiyu https://unix.stackexchange.com/questions/534596/hiding-processes-on-unix-linux

lcdtyph

2021-02-07 18:38:12 +08:00 via iPhone

@zhoudaiyu
如果没有什么特殊的 mount options，那说明你这个不是 pid，而是某个 tid，你需要找到它的主进程再看。

/proc/<tid> 是不会出现在 getdents 调用中，所以不会被 ls 看到

barathrum

2021-02-07 18:50:52 +08:00

@lcdtyph 感谢兄弟，学到一个新知识。

zhoudaiyu

2021-02-07 19:03:02 +08:00

@lcdtyph 老哥我看有 mountinfo mounts mountstats 截图里是 mountstats 的输出，tid 是线程 id 吗？

device rootfs mounted on / with fstype rootfsdevice sysfs mounted on /sys with fstype sysfs
device proc mounted on /proc with fstype procdevice devtmpfs mounted on /dev with fstype devtmpfs
device securityfs mounted on /sys/kernel/security with fstype securityfsdevice tmpfs mounted on /dev/shm with fstype tmpfsdevice devpts mounted on /dev/pts with fstype devptsdevice tmpfs mounted on /run with fstype tmpfs
device tmpfs mounted on /sys/fs/cgroup with fstype tmpfsdevice cgroup mounted on /sys/fs/cgroup/systemd with fstype cgroupdevice pstore mounted on /sys/fs/pstore with fstype pstoredevice cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/net_cls,net_prio with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/devices with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/cpu,cpuacct with fstype cgroup
device cgroup mounted on /sys/fs/cgroup/hugetlb with fstype cgroup
device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
device configfs mounted on /sys/kernel/config with fstype configfsdevice /dev/mapper/vg_root-lv_root mounted on / with fstype xfsdevice selinuxfs mounted on /sys/fs/selinux with fstype selinuxfsdevice systemd-1 mounted on /proc/sys/fs/binfmt_misc with fstype autofs
device hugetlbfs mounted on /dev/hugepages with fstype hugetlbfsdevice debugfs mounted on /sys/kernel/debug with fstype debugfs
device mqueue mounted on /dev/mqueue with fstype mqueue
device /dev/sda1 mounted on /boot with fstype xfsdevice /dev/mapper/vg_root-lv_var mounted on /var with fstype xfsdevice /dev/mapper/vg_root-lv_opt mounted on /opt with fstype xfsdevice /dev/mapper/vg_app-lv_DATA mounted on /DATA with fstype xfs
device /dev/mapper/vg_root-lv_tmp mounted on /tmp with fstype xfs
device /dev/mapper/vg_root-lv_home mounted on /home with fstype xfs
device sunrpc mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs
device /dev/mapper/vg_root-lv_var mounted on /var/lib/docker/containers with fstype xfs
device /dev/mapper/vg_root-lv_var mounted on /var/lib/docker/overlay2 with fstype xfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/206f7c4c-52cd-11ea-9f39-0050569a04bf/volumes/kubernetes.io~secret/sdn-token-d66w8 with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/e3695fa7-2dce-11ea-97bb-005056a99505/volumes/kubernetes.io~secret/sdn-token-d66w8 with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/5abfd92e-2dc8-11ea-97bb-005056a99505/volumes/kubernetes.io~secret/sync-token-tr7f8 with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/c47caa96460e233c36b6790b1ca4ea450dee1d297f9e40bf199c6ab83bb1e690/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/edb026a3c373d3ddbd064b9d8e997ccb441d56eb883f19237374bdd839cd876a/merged with fstype overlay
device proc mounted on /run/docker/netns/default with fstype proc
device shm mounted on /var/lib/docker/containers/6d3e50e348e7bc4608b568fd43ad63670c57f0c53de71f446ba17e8ec5e47f4e/shm with fstype tmpfs
device shm mounted on /var/lib/docker/containers/f2c824ce425cfe15dda053af70d4fd02d64afdae9367ae78bcd6b5758338182c/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/60fffaf1c2c9cbe0d12504a67a0b603c9bdbf8b0f6322dc30b2befa9cbb87438/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/0045f1ad6a6556324b91b80892dcc4951ad32ec8dfe153b7c866b942fae74843/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/ec0204696ace63e3c1eb69bd81e029c13d7c1320b16b561ddceb3c56f4ae9172/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/a774eddee040f246b9dea20a33781a1b8cb80a3bc668690d66baec33a0f56a1d/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/cfcfc406430dc094d2b2a7084f38ea44b1eb2200282ce5fa432d46c9fee9d1f6/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/6d7a77c6a7dade24e619f63aa813962ae003e52f15dc146141af3c362564fd27/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/52230e78851c2d66d058e1fda52b3cb8b3493a86804354d234fe64895b3aed56/merged with fstype overlay
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/a684975d-9ef2-11ea-90f1-005056a9a751/volumes/kubernetes.io~secret/node-exporter-token-mttqc with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/c07d51282d5db3eeaa6966bdb277d34fdab0707900b34822b5bb85c6f3dddc84/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/14c0c7e261e57a3eac08de62225b5b33fe8a11d231b13e34ddbdec8231e98d24/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/7c510eedb0ebb50ace710478ceca3037011a99a24ded9b2862d2700064782b22/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/be29c4a1f9bbd28181851f27df38bf277a1984595490599db5d4297de8d0847f/merged with fstype overlay
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/oauth-config with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/serving-cert with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/default-token-mlq5l with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/2a50c3d9829ae039bbe3afb1108e2b131aca83febf40070b0423761aee576354/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/4b04ef977813ab3b24704d433d09d56fd069ee9103a32c5d041a3a072eab4663/shm with fstype tmpfs
device proc mounted on /run/docker/netns/4dd48aec066f with fstype proc
device overlay mounted on /var/lib/docker/overlay2/e1ee93593bab0e24a711c4ce871534fefe651e8be5ee11cf64f544e0b3e2e86f/merged with fstype overlay
device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
device overlay mounted on /var/lib/docker/overlay2/e04f7635399c36249c8e9b90d5619ba593b7633275d1b2c2363d3c93d0e2f55b/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/0be6198515753a8a97b51d078ac64e1153325342221e8d77a956eecabd448d12/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/f54abb6462617d5910b8d41e5fa4c131be18aa59d4d227040e3f3e2ecd69d70a/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/a177d06ddec1fb54a0b04725fabe944293ca72eb4aa0bdbf4e918f828d089466/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/0af1e5fe64c765d3e0a4d574b725829a6b6e3170b929ede21cdd5b5cb97b0fc4/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/284f17f11532b9750a1f15adb3c2361796c42c4e0afc220c49c0cd394c477299/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/ce6e3d2eb2879de645014da2be3650c7f172634131d08718304ef106a3b4ea66/merged with fstype overlay
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/37a179b0-690c-11eb-b1fb-005056a9a751/volumes/kubernetes.io~secret/serving-cert with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/37a179b0-690c-11eb-b1fb-005056a9a751/volumes/kubernetes.io~secret/webconsole-token-spxpx with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/188242c0ee6187645f21d2f0195615af2d450337a461c4c197367bfaf794d5ed/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/5b5fb40679472f61b4b5f37caf20666291fdb38f04648710dbd23a3e952c0845/shm with fstype tmpfs
device proc mounted on /run/docker/netns/feff0fd124b3 with fstype proc
device overlay mounted on /var/lib/docker/overlay2/d6d7732f83a71c8c3f33a257bab98a140b5099e0ba2b1bb789c057520cf75d95/merged with fstype overlay
device tmpfs mounted on /run/user/0 with fstype tmpfs

zhoudaiyu

2021-02-07 19:03:51 +08:00

@lcdtyph 错了，不是截图是是文本 🤣

lcdtyph

2021-02-07 19:13:09 +08:00

@zhoudaiyu #16
这里没有啊，你直接输入 mount 命令就会列出所有 mounted 和 options 了

mount | grep proc

输出括号里面的部分就是 options

lcdtyph

2021-02-07 19:17:37 +08:00

@zhoudaiyu #16
要找 tid 的话给 ps 加上-L 参数就行了
ps -efL | grep 14546
其中的 LWP 列就是 tid

zhoudaiyu

2021-02-07 19:30:26 +08:00

@lcdtyph mount|grep 14546 返回空，ps -efL | grep 14546 这个好使，多谢老哥，但是我还是不太明白为啥 /proc 看不到这个 14546 这个进程号的文件夹，但是可以 cd 到 14546 中来，老哥能解个惑不？

lcdtyph

2021-02-07 19:47:04 +08:00 via iPhone

@zhoudaiyu
mount 那条命令是看 procfs 的 options 的，而且不是 grep 14546，就是 grep proc 。如果设置了 hidepid=2 那么 /proc 下就看不到其他用户的 pid 了

你这里的 14546 不是进程号而是个线程号(tid)，/proc/tid 下的内容和 /proc/pid/task/tid 下的内容是一模一样的，所以实现 procfs 的时候让这些入口不在 getdents 系统调用的结果中出现。而 ls 正是使用 getdents 实现的，所以 ls 自然看不到这些入口。

cd 依赖的是 chdir 系统调用，所以不受影响

howellz

2021-02-07 19:54:11 +08:00

没细看，不过 docker 不就是利用 namespace 来隔离进程和文件系统（包括 proc ）的么？

zhoudaiyu

2021-02-07 19:55:39 +08:00 via iPhone

@lcdtyph 学习到了十分感谢！！

zhoudaiyu

2021-02-08 08:15:50 +08:00 via iPhone

@howellz 但是宿主机上应该都能看到才对

liuxu

2021-02-10 15:08:47 +08:00

@howellz 学到了，多谢大佬，学习了

$ man procfs
/proc/[tid] subdirectories
Each one of these subdirectories contains files and subdirectories exposing information about the thread with the corre‐
sponding thread ID. The contents of these directories are the same as the corresponding /proc/[pid]/task/[tid] directo‐
ries.
The /proc/[tid] subdirectories are not visible when iterating through /proc with getdents(2) (and thus are not visible
when one uses ls(1) to view the contents of /proc).

然后用 sysdig 抓 ls /proc
第一个命令行：$ sudo sysdig proc.name = ls
第二个命令行：$ ls /proc
然后第一个命令行得到：
...
333156 14:54:19.153132412 4 ls (671251) > stat
333159 14:54:19.153141644 4 ls (671251) < stat res=0 path=/proc
333160 14:54:19.153149403 4 ls (671251) > openat
333161 14:54:19.153159898 4 ls (671251) > fstat fd=-82524538700541
333162 14:54:19.153161715 4 ls (671251) < fstat res=0
333164 14:54:19.153169912 4 ls (671251) > getdents64 fd=137455746819
333175 14:54:19.154214139 4 ls (671251) < getdents64 res=11840
333176 14:54:19.154223263 4 ls (671251) > lstat
333177 14:54:19.154232665 4 ls (671251) < lstat res=0 path=/proc/fb
333178 14:54:19.154238761 4 ls (671251) > lstat
333179 14:54:19.154242851 4 ls (671251) < lstat res=0 path=/proc/fs
333180 14:54:19.154247207 4 ls (671251) > lstat
333181 14:54:19.154250957 4 ls (671251) < lstat res=0 path=/proc/bus
333182 14:54:19.154254649 4 ls (671251) > lstat
333183 14:54:19.154257514 4 ls (671251) < lstat res=0 path=/proc/dma
333184 14:54:19.154260911 4 ls (671251) > lstat
333185 14:54:19.154263837 4 ls (671251) < lstat res=0 path=/proc/i8k
...