求问一个 python3 的 gevent。 monkey.patch_ssl() 之后运行代码会出错的问题

import requests
import gevent
from gevent.pool import Pool
from gevent import monkey
# monkey.patch_all()
monkey.patch_socket()
monkey.patch_ssl()


contents = []
pool = Pool(20)

def req(url):
    res = requests.get(url)
    contents.append(res.status_code)
urls = [
    "http://www.baidu.com",
    "http://www.iplaysoft.com",
    "http://www.cupfox.com",
    "http://www.opbear.com",
    "http://www.bing.ren"
]

gtasks = [pool.spawn(req, url) for url in urls]
gevent.joinall(gtasks)
print(contents)

就这段代码用的是最简单的 gevent 协程请求 requests 如果只用 monkey.patch_socket()是完全没有问题的但凡加上了 monkey.patch_ssl()就会报错或者加上 monkey.patch_all()也会报错

Traceback (most recent call last):
File "/usr/local/python3.6/lib/python3.6/site-packages/gevent/greenlet.py", line 536, in run
result = self._run(*self.args, **self.kwargs)
File "test.py", line 14, in req
res = requests.get(url)
File "/usr/local/python3.6/lib/python3.6/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/python3.6/lib/python3.6/ssl.py", line 459, in options
super(SSLContext, SSLContext).options.set(self, value)
[Previous line repeated 316 more times]

 RecursionError: maximum recursion depth exceeded while calling a Python object`
 During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/python3.6/lib/python3.6/site-packages/gevent/hub.py", line 866, in switch
switch(value)
File "/usr/local/python3.6/lib/python3.6/site-packages/gevent/_tblib.py", line 425, in dump_traceback
return dumps(tb)

Tue Sep 19 10:28:11 2017 <built-in method switch of Greenlet object at 0x10c6c4f20> failed with RecursionError<br/>

Traceback (most recent call last):
File "test.py", line 25, in <module>
gevent.joinall(gtasks)
File "/usr/local/python3.6/lib/python3.6/site-packages/gevent/greenlet.py", line 649, in joinall
return wait(greenlets, timeout=timeout, count=count)
File "/usr/local/python3.6/lib/python3.6/site-packages/gevent/hub.py", line 1037, in wait
return list(iwait(objects, timeout, count))

gevent.hub.LoopExit: ('This operation would block forever',<Hub at 0x10c6c4af8 select default pending=0 ref=0 resolver=<gevent.resolver_thread.Resolver at 0x10c7905f8 pool=<ThreadPool at 0x10c790668 0/2/10>> threadpool=<ThreadPool at 0x10c790668 0/2/10>>)

今天的错误代码长这样。因为错误代码很长很乱我就删掉了中间的一些过程把头尾都留下来了。 gevent 版本 1.2.2 requests 版本 2.18.4

有没有大大帮忙解释下出现错误的原因以及 monkey.patch_ssl()所做的事情

10 条回复 • 2017-09-21 13:55:58 +08:00

nightstart

2017-09-19 10:49:13 +08:00

求不沉.... 这个问题困扰我好久了

NoAnyLove

2017-09-19 11:46:48 +08:00

这种问题，翻一下官方 repo 的 issue 就知道了。Python 3 下不知道什么问题造成的，解决方法是，把下面几行内容移动到最开头：

from gevent.pool import Pool
from gevent import monkey
# monkey.patch_all()
monkey.patch_socket()
monkey.patch_ssl()

nightstart

2017-09-19 14:25:17 +08:00

@NoAnyLove 可以问一下是在哪里看的吗。

2017-09-19 14:34:55 +08:00

同样的问题我本地跑会报这个错误测试服务器跑和线上服务器跑没这个错误
官方 repo 的 issue 里说是 gevent 和 requests 的问题 gevent 版本升级到 1.21 也都是在 3.6 上出现的可以尝试升级下 gevent

2017-09-19 15:01:25 +08:00

老哥 python3 还用 gevent ？ asyncio 不好吗

janxin

2017-09-19 15:04:19 +08:00

https://github.com/gevent/gevent/issues/1016

calease

2017-09-19 16:37:37 +08:00

gevent 文档里写的很清楚了一定要先 from gevent import monkey; monkey.patch_all()之后才能 import 别的 library。
http://www.gevent.org/intro.html#monkey-patching

When monkey patching, it is recommended to do so as early as possible in the lifetime of the process. If possible, monkey patching should be the first lines executed.

NoAnyLove

2017-09-19 22:15:10 +08:00

我之前看的是 https://github.com/gevent/gevent/issues/941

nightstart

2017-09-21 13:54:04 +08:00

@qs 我这边也发生了这个情况我的自己上跑有问题我小伙伴上的就没问题

nightstart

2017-09-21 13:55:58 +08:00