Litao OK Blog

A little blog for my life.

Python Tornado 启动参数解析

Tornado 4.3/Python3.4

Tornado典型server启动方式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import logging as log
import socket
import sys

import os
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web

PROJECT_ROOT = os.path.realpath(os.path.dirname(__file__))
sys.path.insert(0, os.path.join(PROJECT_ROOT, os.pardir))

import config


# 初始化参数
def init_options():
    tornado.options.define('port', default=9000, type=int)
    tornado.options.define('worker', default=2, type=int)
    tornado.options.define('debug', default=config.is_debug, type=bool)
    tornado.options.define('doc', default=False, type=bool)
    tornado.options.define('timeout', default=2, type=int)
    tornado.options.parse_command_line()

    return tornado.options.options


# 处理其他请求
class BaseHandler(tornado.web.RequestHandler):
    def data_received(self, chunk):
        pass

    def get(self):
        self.write_error(404)

    def write_error(self, status_code, **kwargs):
        if status_code == 404:
            self.render('404.html')
        elif status_code == 500:
            self.render('500.html')
        else:
            self.write('error:' + str(status_code))


# 主程序
def main():
    try:
        _options = init_options()

        socket.setdefaulttimeout(_options.timeout)
        # static_path = os.path.join(os.path.dirname(__file__), settings.static_path)

        # 总路由列表
        url_list = [
            (r'/myuri', MyURIHandler),  
            (r'.*', BaseHandler),  # 这个放在最后,处理404和500
        ]

        application = tornado.web.Application(url_list,
                                              template_path=os.path.join(os.path.dirname(__file__), "public"),
                                              debug=_options.debug)

        _http_server = tornado.httpserver.HTTPServer(application, no_keep_alive=True)
        _http_server.bind(_options.port)
        _http_server.start(1 if _options.debug else _options.worker)

        echo_str = "HTTPServer 1 thread(s), {} worker(s), listen on http://{}:{}, debug={}, logging={}"
        log.info(echo_str.format(str(_options.worker),
                                 str(socket.gethostname()),
                                 str(_options.port),
                                 str(_options.debug),
                                 str(_options.logging)))
        tornado.ioloop.IOLoop.instance().start()
    except Exception as e:
        log.exception(e)


if __name__ == "__main__":
    main()

启动时,可以通过命令行参数来确定其他参数和日志处理方式,具体参数可以通过–help命令获得,以上代码放在main.py中,则:

1
python main.py --help

会提示如下帮助信息:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Usage: main.py [OPTIONS]

Options:

  --help                           show this help information

/Users/litao/Documents/LEO/source_code/leo_max_api/.env/lib/python3.4/site-packages/tornado/log.py options:

  --log-file-max-size              max size of log files before rollover
                                   (default 100000000)
  --log-file-num-backups           number of log files to keep (default 10)
  --log-file-prefix=PATH           Path prefix for log files. Note that if you
                                   are running multiple tornado processes,
                                   log_file_prefix must be different for each
                                   of them (e.g. include the port number)
  --log-rotate-interval            The interval value of timed rotating
                                   (default 1)
  --log-rotate-mode                The mode of rotating files(time or size)
                                   (default size)
  --log-rotate-when                specify the type of TimedRotatingFileHandler
                                   interval other options:('S', 'M', 'H', 'D',
                                   'W0'-'W6') (default midnight)
  --log-to-stderr                  Send log output to stderr (colorized if
                                   possible). By default use stderr if
                                   --log_file_prefix is not set and no other
                                   logging is configured.
  --logging=debug|info|warning|error|none 
                                   Set the Python log level. If 'none', tornado
                                   won't touch the logging configuration.
                                   (default info)

main.py options:

  --debug                           (default False)
  --doc                             (default False)
  --port                            (default 9000)
  --timeout                         (default 2)
  --worker                          (default 2)

参数可以用来处理生成人日志,因为官方的按照时间分隔日志的TimedRotatingFileHandler在多进程情况下会丢失日志,所以可以通过Tornado这些参数来给每个进程单独处理日志分隔,示例如下:

1
python main.py --port=9000 --worker=1 --log-file-max-size=0 --log-file-prefix=logs/9000.log --log-rotate-mode=time --log-rotate-when=H --logging=info

以上参数会按照小时来切割9000.log文件,日志写入9000.log时并不检查文件大小(log-file-max-size=0表示无限大),但是在多进程模式下无效。

实际启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#!/bin/bash

APP_DIR=/home/deploy/leo_max_api
APP_EXEC="/home/deploy/leo_max_api/.env/bin/python /home/deploy/leo_max_api/main.py"

LOG_DIR=$APP_DIR/logs
LOG_LEVEL=info # 测试环境用debug,生产环境用error

# 每个实例worker数量
WORKER_NUM=20
# 每个服务器实例监听端口号,配置数量可以测试下每个worker内存占用情况
# 单核1G内存,建议一个实例,10个worker以内
# 4核16G内存,建议4个实例,每个20个worker
PORTS=(9000 9001 9002 9003)

function start_server() {
    su - deploy & cd $APP_DIR
    export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
    source .env/bin/activate
    # chmod +x main.py
    for port in "${PORTS[@]}";
    do
        $APP_EXEC --port=$port --worker=$WORKER_NUM --logging=$LOG_LEVEL --log-file-prefix=$LOG_DIR/$port.log --log-file-max-size=0 &
    done
}

function stop_server() {
    for port in "${PORTS[@]}";
    do
        ps aux | grep "$APP_EXEC" | grep "$port" | awk '{print $2}' | xargs kill -9
    done
}

case "$1" in
start)
    start_server
;;
stop)
    stop_server
;;
restart)
    stop_server
    start_server
;;
*)
    echo 'Usage: bin/tornado.sh [start|stop|restart]'
esac

以上脚本会启动4个实例(进程),每个实例启动20个workers,我使用的服务器4核16G内存的,建议使用4个实例,worker数要看内存大小了。日志方面,会生成4个日志,分别是9000.log、9001.log、9002.log和9003.log,通过size分隔方式,但是size=0表示可以无限大,也就是不切分文件,由crontab定时任务来切割。尝试过–log-rotate-mode=time,完全无效。官方文档说多进程方式时间切分有问题,但是每个实例单独设置也是不行的。

需要用root用户加在crontab中执行:

1
2
# crontab -e
0 0 * * * /home/deploy/leo_max_api/etc/log.sh > /dev/null 2>&1

下面是log.sh内容:

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash
# set path
APP_DIR=/home/deploy/leo_max_api
LOG_DIR=$APP_DIR/logs
PORTS=(9000 9001 9002 9003)
# rename log
for port in "${PORTS[@]}";
do
    mv $LOG_DIR/$port.log $LOG_DIR/$port.log.`date -d yesterday +%Y%m%d`
    touch $LOG_DIR/$port.log
done
# restart tornado
/etc/init.d/your_init_script restart