首页 注册 登录
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
V2EX 提问指南
广告
kosenpai
V2EX  ›  问与答

Digital Ocean 新加坡最近总是 out of memory 崩溃

  •   
  •   kosenpai · 2016-05-17 22:20:20 +08:00 · 2290 次点击
    这是一个创建于 3110 天前的主题，其中的信息可能已经有所发展或是发生改变。

    自用爬梯，一直稳定运行了一年多，最近 VPS 运行半天就完全死掉， ssh 也登不上去。看 syslog 发现崩溃之前都是 out of memory 。但是看内存记录发现崩溃前内存占用完全正常，不正常的部分是 CPU 占用在崩溃前异常高。下面是崩溃的一段，感觉就是这样被搞死的。请问大家这是怎么了？

    May 17 06:16:01 PIMSserver kernel: [18538.680826] sh invoked oom-killer: gfp_mask=0x84d0, order=0, oom_score_adj=0
May 17 06:16:01 PIMSserver kernel: [18538.680852] sh cpuset=/ mems_allowed=0
May 17 06:16:01 PIMSserver kernel: [18538.680875] CPU: 0 PID: 19134 Comm: sh Tainted: P OX 3.13.0-48-generic #80-Ubuntu
May 17 06:16:01 PIMSserver kernel: [18538.680878] Hardware name: DigitalOcean Droplet, BIOS 20160425 04/25/2016
May 17 06:16:01 PIMSserver kernel: [18538.680882] 0000000000000000 ffff880018913978 ffffffff81721506 ffff880019dc8000
May 17 06:16:01 PIMSserver kernel: [18538.680896] ffff880018913a00 ffffffff8171bdc1 ffffffff81067996 ffff8800189139d8
May 17 06:16:01 PIMSserver kernel: [18538.680900] ffffffff810c75dc 0000000000000001 ffff88001fffae28 0000000000000000
May 17 06:16:01 PIMSserver kernel: [18538.680904] Call Trace:
May 17 06:16:01 PIMSserver kernel: [18538.680979] [<ffffffff81721506>] dump_stack+0x45/0x56
May 17 06:16:01 PIMSserver kernel: [18538.680991] [<ffffffff8171bdc1>] dump_header+0x7f/0x1f1
May 17 06:16:01 PIMSserver kernel: [18538.681025] [<ffffffff81067996>] ? put_online_cpus+0x56/0x80
May 17 06:16:01 PIMSserver kernel: [18538.681045] [<ffffffff810c75dc>] ? rcu_oom_notify+0xcc/0xf0
May 17 06:16:01 PIMSserver kernel: [18538.681067] [<ffffffff811529be>] oom_kill_process+0x1ce/0x330
May 17 06:16:01 PIMSserver kernel: [18538.681097] [<ffffffff812d7225>] ? security_capable_noaudit+0x15/0x20
May 17 06:16:01 PIMSserver kernel: [18538.681102] [<ffffffff811530f4>] out_of_memory+0x414/0x450
May 17 06:16:01 PIMSserver kernel: [18538.681112] [<ffffffff81159460>] __alloc_pages_nodemask+0xa60/0xb80
May 17 06:16:01 PIMSserver kernel: [18538.681135] [<ffffffff81197ad3>] alloc_pages_current+0xa3/0x160
May 17 06:16:01 PIMSserver kernel: [18538.681146] [<ffffffff81179064>] __pmd_alloc+0x24/0x110
May 17 06:16:01 PIMSserver kernel: [18538.681152] [<ffffffff8117a825>] handle_mm_fault+0xd75/0xf10
May 17 06:16:01 PIMSserver kernel: [18538.681183] [<ffffffff811787f3>] ? follow_page_mask+0x73/0x5b0
May 17 06:16:01 PIMSserver kernel: [18538.681190] [<ffffffff811787be>] ? follow_page_mask+0x3e/0x5b0
May 17 06:16:01 PIMSserver kernel: [18538.681195] [<ffffffff8117ab26>] __get_user_pages+0x166/0x5e0
May 17 06:16:01 PIMSserver kernel: [18538.681199] [<ffffffff8117aff2>] get_user_pages+0x52/0x60
May 17 06:16:01 PIMSserver kernel: [18538.681214] [<ffffffff811c4246>] copy_strings.isra.17+0x256/0x2e0
May 17 06:16:01 PIMSserver kernel: [18538.681219] [<ffffffff811c4304>] copy_strings_kernel+0x34/0x40
May 17 06:16:01 PIMSserver kernel: [18538.681224] [<ffffffff811c586c>] do_execve_common.isra.22+0x4fc/0x7e0
May 17 06:16:01 PIMSserver kernel: [18538.681229] [<ffffffff811c5de6>] SyS_execve+0x36/0x50
May 17 06:16:01 PIMSserver kernel: [18538.681239] [<ffffffff81732569>] stub_execve+0x69/0xa0
May 17 06:16:01 PIMSserver kernel: [18538.681242] Mem-Info:
May 17 06:16:01 PIMSserver kernel: [18538.681250] Node 0 DMA per-cpu:
May 17 06:16:01 PIMSserver kernel: [18538.681257] CPU 0: hi: 0, btch: 1 usd: 0
May 17 06:16:01 PIMSserver kernel: [18538.681259] Node 0 DMA32 per-cpu:
May 17 06:16:01 PIMSserver kernel: [18538.681262] CPU 0: hi: 186, btch: 31 usd: 160
May 17 06:16:01 PIMSserver kernel: [18538.681268] active_anon:866 inactive_anon:890 isolated_anon:0
May 17 06:16:01 PIMSserver kernel: [18538.681268] active_file:37 inactive_file:46 isolated_file:0
May 17 06:16:01 PIMSserver kernel: [18538.681268] unevictable:0 dirty:1 writeback:0 unstable:0
May 17 06:16:01 PIMSserver kernel: [18538.681268] free:1471 slab_reclaimable:2333 slab_unreclaimable:4106
May 17 06:16:01 PIMSserver kernel: [18538.681268] mapped:24 shmem:5 pagetables:1581 bounce:0
May 17 06:16:01 PIMSserver kernel: [18538.681268] free_cma:0
May 17 06:16:01 PIMSserver kernel: [18538.681274] Node 0 DMA free:1928kB min:88kB low:108kB high:132kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:96kB slab_unreclaimable:404kB kernel_stack:8kB pagetables:144kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6801 all_unreclaimable? yes
May 17 06:16:01 PIMSserver kernel: [18538.681288] lowmem_reserve[]: 0 471 471 471
May 17 06:16:01 PIMSserver kernel: [18538.681296] Node 0 DMA32 free:3956kB min:2732kB low:3412kB high:4096kB active_anon:3464kB inactive_anon:3560kB active_file:132kB inactive_file:172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:507896kB managed:485892kB mlocked:0kB dirty:4kB writeback:0kB mapped:80kB shmem:20kB slab_reclaimable:9236kB slab_unreclaimable:16020kB kernel_stack:992kB pagetables:6180kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:11036 all_unreclaimable? yes
May 17 06:16:01 PIMSserver kernel: [18538.681303] lowmem_reserve[]: 0 0 0 0
May 17 06:16:01 PIMSserver kernel: [18538.681307] Node 0 DMA: 20*4kB (U) 29*8kB (UM) 35*16kB (UM) 15*32kB (UMR) 5*64kB (UMR) 0*128kB 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1928kB
May 17 06:16:01 PIMSserver kernel: [18538.681334] Node 0 DMA32: 537*4kB (UEM) 208*8kB (UM) 1*16kB (M) 0*32kB 0*64kB 1*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3956kB
May 17 06:16:01 PIMSserver kernel: [18538.681353] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May 17 06:16:01 PIMSserver kernel: [18538.681355] 554 total pagecache pages
May 17 06:16:01 PIMSserver kernel: [18538.681357] 453 pages in swap cache
May 17 06:16:01 PIMSserver kernel: [18538.681360] Swap cache stats: add 526366, delete 525913, find 169308/273582
May 17 06:16:01 PIMSserver kernel: [18538.681362] Free swap = 4110736kB
May 17 06:16:01 PIMSserver kernel: [18538.681363] Total swap = 4194300kB
May 17 06:16:01 PIMSserver kernel: [18538.681365] 130972 pages RAM
May 17 06:16:01 PIMSserver kernel: [18538.681367] 0 pages HighMem/MovableOnly
  • pimsserver
  • 06:16:01
  • kernel
  • May
    7 条回复  •  2016-05-20 09:24:05 +08:00
    meiriyitie
        1
    meiriyitie  
       2016-05-17 22:27:51 +08:00
    重装系统大法？ 也许 do 想让你升级到 10 刀哈哈
    tvallday
        2
    tvallday  
       2016-05-17 22:49:46 +08:00 via iPad
    Swap space 太小了？整个大的？其实 10 刀的应该就不会出现 memory 不够的情况了，节省很多无谓的时间。
    ksky
        3
    ksky  
       2016-05-17 22:56:13 +08:00
    装 Mysql 了吗。内存不够不是 apache 就是 mysql 的问题。显然是有人在弄你的网站。
    kosenpai
        4
    kosenpai  
    OP
       2016-05-18 08:57:39 +08:00
    @meiriyitie 现在正在试新建一个，看看是不是还是有问题。
    kosenpai
        5
    kosenpai  
    OP
       2016-05-18 09:04:49 +08:00
    @tvallday 应该不是内存的问题，从图里可以看到崩溃的时候 CPU 爆炸，但是内存还是正常的。写了 tickets 问 DO ，但是他们也不给建议……
    kosenpai
        6
    kosenpai  
    OP
       2016-05-18 09:08:14 +08:00
    @ksky mysql 和 apache 是刚装的，之前只有 PPTP 和 ss 。崩溃在 mysql 安装之前就开始发生了。现在每天都崩溃一次，弄的我也崩溃了。不过谢谢提醒，也许有人通过 PPTP 来攻击我呢？我去查查 log 。
    kosenpai
        7
    kosenpai  
    OP
       2016-05-20 09:24:05 +08:00
    我查了 log 发现每条都和 Supervisor 有关，我都忘了什么时候装了这个。现在卸载掉之后稳定运行一天了。再看一段时间有没有问题。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1118 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 30ms · UTC 18:41 · PVG 02:41 · LAX 10:41 · JFK 13:41
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.