Linux 的 ReusePort 特性

[firejoke]

最近在看 Django-channels, 因为自带的 daphne asgi 服务器本身没有提供多 worker 的工作方式,
而文档内推荐的部署方案是用 supervisor 做进程管理,
但考虑到本来系统上就有 systemd, 所以尝试用 systmed 来做进程管理,
用 systemd.socket 监听指定端口, 在收到连接后, 启用对应的服务, 并把 socket 传递给服务,
先贴一下单个进程的 dc.socket 和 dc.service 文件

# dc.socket
[Unit]
Description= Django Channels socket

[Socket]
ListenStream=0.0.0.0:9001

[Install]
WantedBy=sockets.target

# dc.service
[Unit]
Description=Django Channels
Requires=veops1.socket

[Service]
Type=simple
WorkingDirectory=/project_path/
NonBlocking=true
ExecStart=/project_path/virtenv/bin/daphne -e systemd:domain=INET:index=0 \
-s DC --access-log /var/log/dc/daphne_access1.log dc.asgi:application
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGQUIT
KillMode = control-group
Restart=on-failure
RestartSec=10s
User=root
Group=root


[Install]
WantedBy=multi-user.target

这样只能启用一个进程, 如果要像 supervisor 那样多个进程监听一个端口, 需要用到从 Linux kernel 3.9 开始增加的 SO_REUSEPORT 特性, 把之前的实例配置变成模板配置 [email protected] 和 [email protected] 文件转变成模板, 像这样

# [email protected]
[Unit]
Description= Django Channels %i socket

[Socket]
ListenStream=0.0.0.0:9001
ReusePort=true
Service=dc@%i.service

[Install]
WantedBy=sockets.target

# [email protected]
[Unit]
Description= Django Channels %i
After=dc@%i.socket

[Service]
Type=simple
WorkingDirectory=/project_path/
ExecStart=/project_path/virtenv/bin/daphne -e systemd:domain=INET:index=0 \
-s DC --access-log /var/log/dc/daphne%i.log dc.asgi:application
NonBlocking=yes
ExecReload=/bin/kill -HUP $MAINPID
KillMode = control-group
Restart=on-failure
RestartSec=10s
User=root

[Install]
WantedBy=multi-user.target

用
systemctl start dc@0
systemctl start dc@1
systemctl start dc@2
...
可以生成多个 service 实例并监听同一个端口
所有访问这个端口的连接都会被分发给各个进程

测试环境:
system-release: CentOS Linux release 7.9.2009 (Core)
uname: 3.10.0-1160.6.1.el7.x86_64
启用第二个 socket 的时候报错了

failed to listen on sockets: Address already in use

看起来 ReusePort 并没有生效
但放在 Centos8, kernel 4.19 上却可以生效,
怀疑是系统有这个特性, 但 systemd 没有启用, 搜了一番也没搜到
有在 3.10 上用 systemd 使用过这个特性的吗?
希望能告诉我一下是不是要改什么配置

[warcraft1236]

升级吧，现在内核都 5 开头了

[50infivedays]

3.10 可以用的 应该是配置错了 要设置 socket option 的

[boboliu]

systemd 对 reuseport 引入还挺早了，v206 就有，centos7 的包还没这么老

是不是没关干净不 reuse 的

[firejoke]

@boboliu #3 没有, 这是个测试环境, 也检查了端口占用

[firejoke]

@50infivedays #2 你说的 socket option 是指 service 文件内的吗,
我单独启 dc@0. socket [email protected] 也不行

[julyclyde]

systemd 传递的 socket，是 listen socket 还是 accepted FD 呢？

[firejoke]

@julyclyde #6 传递的应该是 fd
https://www.freedesktop.org/software/systemd/man/systemd.socket.html#
```
Note that the daemon software configured for socket activation with socket units needs to be able to accept sockets from systemd, either via systemd's native socket passing interface (see sd_listen_fds(3) for details about the precise protocol used and the order in which the file descriptors are passed) or via traditional inetd(8)-style socket passing (i.e. sockets passed in via standard input and output, using StandardInput=socket in the service file).
```

[julyclyde]

原来两种都行啊？
前者似乎挺高级的，看起来是需要特地编程才能支持的功能
见过 docker.service 和 docker.socket 就是这种关系

后者 inetd-style 的话其实就是 stdio 模式了，应用程序自己并不负责多进程管理的，每个 accept 就给一个单独的进程处理

[firejoke]

@julyclyde #8 但我的问题还是没找到原因, 同样的 sokcet 配置, 在 centos8 kernel 4.x 上可以, centos7 kernel 3.10 就报错了...
目前唯一发现的区别就是
/usr/include/asm-generic/socket.h 里面和 REUSEPORT 有关的, 在 centos8 多了几个

[tomychen]

SO_REUSEPORT (since Linux 3.9)
Permits multiple AF_INET or AF_INET6 sockets to be bound to an identical socket address. This option must be set on each socket (including the first socket) prior to calling bind(2) on the
socket. To prevent port hijacking, all of the processes binding to the same address must have the same effective UID. This option can be employed with both TCP and UDP sockets.

For TCP sockets, this option allows accept(2) load distribution in a multi-threaded server to be improved by using a distinct listener socket for each thread. This provides improved load dis‐
tribution as compared to traditional techniques such using a single accept(2)ing thread that distributes connections, or having multiple threads that compete to accept(2) from the same socket.

For UDP sockets, the use of this option can provide better distribution of incoming datagrams to multiple processes (or threads) as compared to the traditional technique of having multiple
processes compete to receive datagrams on the same socket.


```c
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <string.h>
#include <netinet/in.h>
#include <unistd.h>

int main(void)
{
int sock = -1;
int flags = 1;
int ret;
struct sockaddr_in sa;

sock = socket(AF_INET, SOCK_STREAM, 0);
if (sock == -1) {
perror("socket");
return 1;
}
if (setsockopt(sock, SOL_SOCKET, SO_REUSEPORT, &flags, sizeof(int)) <
0) {
perror("setsockopt");
return 1;
}

memset(&sa, 0, sizeof(sa));

sa.sin_addr.s_addr = htonl(INADDR_ANY);
sa.sin_family = AF_INET;
sa.sin_port = htons(10086);

ret = bind(sock, (struct sockaddr *)&sa, sizeof(sa));
if (ret == -1) {
perror("bind()");
return 1;
}

ret = listen(sock, 10);
if (ret == -1) {
perror("listen()");
return 1;
}
while (1) {
printf("listen\n");
sleep(2);
}
}
```
uname -a
Linux localhost.localdomain 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

代码工作正常，所以和系统无关

[firejoke]

@tomychen #10 嗯, 我也用程序试了的, 系统支持, 现在怀疑是 systemd 版本太老了