V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
MrMike
V2EX  ›  Python

请教下 supervisor 配置的问题

  •  
  •   MrMike · 2017-04-15 00:45:39 +08:00 · 4286 次点击
    这是一个创建于 2782 天前的主题,其中的信息可能已经有所发展或是发生改变。

    为了监控 elasticsearch 死掉后,能自动重启,网上查了下,supervisor 可以实现这个要求,以下是配置: [program=sticsearch] command=/usr/share/elasticsearch/bin/elasticsearch autostart=true autorestart=true stdout_logfile=/home/demoer/log/elasticsearch/elasticsearch-stdout.log stderr_logfile=/home/demoer/log/elasticsearch/elasticsearch-stderr.log startsecs=5

    我把这个直接写在 supervisor.conf ,在启动 supervisorctl reload, reread, update,都说没有找到配置( No config updates to processes )。 我用 include_file 单独来引入,也是同样的结果。 请问下配置代码是否正确呢?

    29 条回复    2017-04-15 23:34:39 +08:00
    RIcter
        1
    RIcter  
       2017-04-15 01:23:11 +08:00 via iPhone
    得重启 supervisor 服务…
    MrMike
        2
    MrMike  
    OP
       2017-04-15 08:11:24 +08:00
    supervisord -c /etc/supervisord.conf

    [root tmp]# ps aux | grep supervisord
    root 10926 0.0 0.3 220120 11836 ? Ss Apr14 0:11 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
    root 12554 0.0 0.2 217092 10664 ? Ss Apr14 0:10 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
    root 12564 0.0 0.2 217128 10716 ? Ss Apr14 0:10 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
    root 12575 0.0 0.2 217280 10824 ? Ss Apr14 0:11 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
    root 12764 0.0 0.2 219728 11464 ? Ss Apr14 0:11 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
    root 13844 0.0 0.2 217588 11164 ? Ss 00:11 0:05 /usr/bin/python /usr/bin/supervisord
    root 14070 0.0 0.2 216964 9828 ? Ss 01:00 0:04 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
    root 14117 0.0 0.2 217144 10644 ? Ss 01:02 0:04 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
    root 14128 0.0 0.2 217520 11144 ? Ss 01:04 0:04 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
    root 14936 0.0 0.2 216960 9832 ? Ss 08:07 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
    root 14938 0.0 0.0 112648 960 pts/0 S+ 08:07 0:00 grep --color=auto supervisord

    [root tmp]# supervisorctl
    supervisor> reload
    Really restart the remote supervisord process y/N? y
    Restarted supervisord
    supervisor> reread
    No config updates to processes
    supervisor> update
    supervisor> status
    supervisor>

    已经重启过很多次,还是一样的结果。

    下面是 supervisor.conf 和 elasticsearch.conf 的配置内容。

    supervisor.conf:

    [include]
    files = /etc/supervisord/conf.d/elasticsearch.conf

    elasticsearch.conf:

    [program=sticsearch]
    command=systemctl restart elasticsearch.service
    ;command=/usr/share/elasticsearch/bin/elasticsearch
    ;directory=/usr/share/elasticsearch
    autostart=true
    autorestart=true
    startsecs=10
    startretries=3
    redirect_stderr=true
    user=root
    stdout_logfile=/home/public/log/elasticsearch/elasticsearch-stdout.log
    stderr_logfile=/home/public/log/elasticsearch/elasticsearch-stderr.log
    zzlyzq
        3
    zzlyzq  
       2017-04-15 08:25:14 +08:00 via iPhone
    主配置里面默认会包含所有 conf.d 下面的 conf 后缀文件 回头我看看我的配置
    MrMike
        4
    MrMike  
    OP
       2017-04-15 08:29:46 +08:00
    @zzlyzq 我安装完 supervisor 后,在 etc 下面没有 /etc/supervisord 文件夹的,我是自己创建的 /etc/supervisord/con.d/文件夹,然后把 elasticsearch.conf 放在里面。
    zzlyzq
        5
    zzlyzq  
       2017-04-15 09:00:18 +08:00
    恩 看起来没啥错,我的 centos6.8 安装后,主文件是 包括如下两行

    [include]
    files = /etc/supervisor.d/*.conf

    你的配置看起来也没啥问题的,要不试试
    ls /etc/supervisord/conf.d/elasticsearch.conf
    看看这个文件是不是真正的存在呢? 不存在的概率可能比较小,不过工作中经常会遇到这种情况,呵呵。
    MrMike
        6
    MrMike  
    OP
       2017-04-15 09:15:19 +08:00
    @zzlyzq 文件是存在的。郁闷。。一直找不到原因。
    zzlyzq
        7
    zzlyzq  
       2017-04-15 09:24:03 +08:00
    @MrMike 既然文件已经加载了,怀疑还是配置的问题,
    1. 是不是配置的 program 名字有冲突?
    2. 打开界面,可能更一目了然

    在 supervisord.conf 开头加上或者修改如下,打开 http ,一目了然

    [unix_http_server]
    file=/tmp/supervisor.sock ; (the path to the socket file)
    [inet_http_server] ; inet (TCP) server disabled by default
    port=0.0.0.0:1009 ; (ip_address:port specifier, *:port for all iface)
    username=ops ; (default is no username (open server))
    password=123123 ; (default is no password (open server))
    zzlyzq
        8
    zzlyzq  
       2017-04-15 09:27:39 +08:00
    @MrMike 我大概知道了, supervisor 在执行命令的时候要求比较苛刻,比如全路径呀,环境变量报错呀啥的, 刚刚的 http 页面可以让你看到
    1. elasticsearch 进程是否配置到 supervisord
    2. 启动情况如何,是 ok 还是 fail
    3. 瞅瞅 log 里面有啥吧
    MrMike
        9
    MrMike  
    OP
       2017-04-15 09:29:16 +08:00
    @zzlyzq supervisord.conf 用的是默认设置,我没修过过。

    [unix_http_server]
    file=/tmp/supervisor.sock ; (the path to the socket file)
    chmod=0777 ; socket file mode (default 0700)
    ;chown=nobody:nogroup ; socket file uid:gid owner
    ;username=user ; (default is no username (open server))
    ;password=123 ; (default is no password (open server))

    ;[inet_http_server] ; inet (TCP) server disabled by default
    ;port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
    ;username=user ; (default is no username (open server))
    ;password=123 ; (default is no password (open server))

    supervisord.log

    2017-04-15 08:08:21,706 INFO RPC interface 'supervisor' initialized
    2017-04-15 08:08:21,706 CRIT Server 'unix_http_server' running without any HTTP authentication checking
    2017-04-15 08:08:21,707 INFO supervisord started with pid 14936
    2017-04-15 08:43:04,969 CRIT Supervisor running as root (no user in config file)
    2017-04-15 08:43:04,969 INFO Included extra file "/etc/supervisord/conf.d/elasticsearch.conf" during parsing
    2017-04-15 08:43:04,980 INFO RPC interface 'supervisor' initialized
    2017-04-15 08:43:04,980 CRIT Server 'unix_http_server' running without any HTTP authentication checking
    2017-04-15 08:43:04,981 INFO daemonizing the supervisord process
    2017-04-15 08:43:04,981 INFO supervisord started with pid 15044

    貌似没看出有啥问题呢。也没有报错。
    zzlyzq
        10
    zzlyzq  
       2017-04-15 09:42:31 +08:00
    Included extra file "/etc/supervisord/conf.d/elasticsearch.conf" during parsing

    上面的一行说明你的 elasticsearch.conf 已经被加载了,现在我越来越猜测是 elasticsearch.conf 配置问题导致没有启动

    既然没有打开 http ,那就使用如下命令看看 status 吧

    supervisorctl status
    zsz
        11
    zsz  
       2017-04-15 10:06:43 +08:00 via iPhone
    1.supervisor 配置中的 command 一定要处于挂起运行,把标准输出,是他的子进程,运行期间不能返回
    2.你需要用命令行启动 es 的方式,配置到 command
    3.既然用 systemd 和 supervisor 用一个就可以,他们都是守护


    有问题可以加群问,效率高些,面向初学者的 python Linux 学习群, 由一群有经验的工程师组成, python qq 群号: 278529278 , PHP qq 群号: 476648701 ,非商业性质,拒绝广告,只接收真正想学这方面技术的朋友,交流学习,申请请说明来自 v2ex
    yzongyue
        12
    yzongyue  
       2017-04-15 10:16:29 +08:00
    这样写试试 [program:sticsearch]
    ksupertu
        13
    ksupertu  
       2017-04-15 14:52:24 +08:00 via iPhone
    新版本先要启动 supervisord 然后在启动 supervisorctl ,不然会一直报错……
    ksupertu
        14
    ksupertu  
       2017-04-15 14:54:34 +08:00 via iPhone
    而且你 command 那里最好写成 shell 脚本去执行,错误概率小一点
    MrMike
        15
    MrMike  
    OP
       2017-04-15 18:15:47 +08:00
    @ksupertu 已经启动过了。 command 我用 /usr/share/elasticsearch/bin/elasticsearch 也是不能用的。在日志里面也看不到错误提示呢。
    zsz
        16
    zsz  
       2017-04-15 19:11:22 +08:00 via iPhone
    @MrMike ,你看的什么错误日志,是 supervisor 的?还是 supervisor 配置文件中 es 的?这个启动不了应该是环境问题,一般是 java 环境, supervisor 自己的错误日志中应该有信息才对
    MrMike
        17
    MrMike  
    OP
       2017-04-15 21:16:48 +08:00
    @zsz 是 supervisord.log
    a128445
        18
    a128445  
       2017-04-15 21:48:13 +08:00
    [program=sticsearch]

    改为
    [program:sticsearch]
    zsz
        19
    zsz  
       2017-04-15 21:48:56 +08:00
    @MrMike 这两个 log 呢? 有错误信息

    stdout_logfile=/home/public/log/elasticsearch/elasticsearch-stdout.log
    stderr_logfile=/home/public/log/elasticsearch/elasticsearch-stderr.log
    zsz
        20
    zsz  
       2017-04-15 21:51:56 +08:00
    先按照上面说的把配置写对 [program:sticsearch]
    MrMike
        21
    MrMike  
    OP
       2017-04-15 22:22:52 +08:00
    @zsz
    @a128445

    elasticsearch.con :

    [program:elasticsearch]
    command=/usr/share/elasticsearch/bin/elasticsearch
    directory=/usr/share/elasticsearch
    autostart=true
    autorestart=true
    startsecs=10
    startretries=3
    redirect_stderr=true
    user=root
    stdout_logfile=/home/demoer/log/elasticsearch/elasticsearch-stdout.log
    stderr_logfile=/home/demoer/log/elasticsearch/elasticsearch-stderr.log


    elasticsearch-stdout.log :

    Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config];
    Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
    at java.nio.file.Files.walkFileTree(Files.java:2662)
    at org.elasticsearch.common.logging.LogConfigurator.resolveConfig(LogConfigurator.java:150)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:114)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:236)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
    Refer to the log for complete error details.
    Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config];
    Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
    at java.nio.file.Files.walkFileTree(Files.java:2662)
    at org.elasticsearch.common.logging.LogConfigurator.resolveConfig(LogConfigurator.java:150)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:114)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:236)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
    Refer to the log for complete error details.
    Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config];
    Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
    at java.nio.file.Files.walkFileTree(Files.java:2662)
    at org.elasticsearch.common.logging.LogConfigurator.resolveConfig(LogConfigurator.java:150)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:114)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:236)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
    Refer to the log for complete error details.
    Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config];
    Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
    at java.nio.file.Files.walkFileTree(Files.java:2662)
    at org.elasticsearch.common.logging.LogConfigurator.resolveConfig(LogConfigurator.java:150)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:114)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:236)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
    Refer to the log for complete error details.


    elasticsearch-stderr.log 内容为空。

    我在重启 supervisor 后,进入 supervisorctl ,提示“ elasticsearch FATAL Exited too quickly (process log may have details)”
    MrMike
        22
    MrMike  
    OP
       2017-04-15 22:26:56 +08:00
    elasticsearch 安装后,在 /usr/share/elasticsearch/就没有 config 这个文件的。
    zsz
        23
    zsz  
       2017-04-15 22:35:35 +08:00 via iPhone
    1. 你命令行成功启动过 es 么?
    2.如果你只是在 systemd 中成功启动过,你的配置文件应该在系统级的目录下,找到他,先用命令行能启动,再把命令配置进去, es 在命令行可以指定配置文件的
    MrMike
        24
    MrMike  
    OP
       2017-04-15 22:45:04 +08:00
    @zsz ES 是可以正常启动和关闭的。我再看看这个错误提示到底是咋回事。。
    MrMike
        25
    MrMike  
    OP
       2017-04-15 22:45:37 +08:00
    @zsz 我试试用命令行启动下。
    MrMike
        26
    MrMike  
    OP
       2017-04-15 22:59:06 +08:00
    @zsz
    @a128445

    配置文件改成这样的:

    [program:elasticsearch]
    command=/etc/init.d/elasticsearch start
    directory=/usr/share/elasticsearch
    autostart=true
    autorestart=true
    startsecs=10
    startretries=3
    redirect_stderr=true
    user=root
    stdout_logfile=/home/demoer/log/elasticsearch/elasticsearch-stdout.log
    stderr_logfile=/home/demoer/log/elasticsearch/elasticsearch-stderr.log

    我手动关闭 es 后,等了几秒钟 es 没有启动,就使用 supervisorctl restart elasticsearch 启动,
    elasticsearch: ERROR (not running)
    elasticsearch: ERROR (spawn error)

    得到这样的错误提示,但是 es 却重新启动了。不能自动启动 es 呢。。
    MrMike
        27
    MrMike  
    OP
       2017-04-15 23:12:07 +08:00
    @zsz stop es 后,我目前的配置,只能手动执行 supervisorctl reload ,才能再让 es 启动。
    zsz
        28
    zsz  
       2017-04-15 23:16:06 +08:00 via iPhone
    1. 上面我指出过 supervisor 中 command 不能使用 systemctl 或者 init.d 的方式,因为这两种是 daemon 运行,命令执行完就返回了
    2. 你加我上面的发群我用 qq 跟你说,这样解释太慢了,非商业化的群,不用担心
    MrMike
        29
    MrMike  
    OP
       2017-04-15 23:34:39 +08:00
    @zsz 加了,回你信息了
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1173 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 27ms · UTC 18:38 · PVG 02:38 · LAX 10:38 · JFK 13:38
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.