V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
Verydiao
V2EX  ›  Python

愣逼 OS X 安装 scrapy 成功记!

  •  
  •   Verydiao · 2016-08-28 21:59:27 +08:00 · 5247 次点击
    这是一个创建于 2787 天前的主题,其中的信息可能已经有所发展或是发生改变。
    OS X 安装了从 python 官网下的 3.5.2 的文件,然后准备安装 scrapy ,用的 Homebrew ,碰上了系统自带版本的梗,准备卸载系统自带 python ,网上搜了教程下小手颤抖,听说卸载系统自带 2.7 版本是作死行为,于是乖乖模仿各种网上的其它教程进行尝试。。。无数次失败之后。忽然发现自己的 pip 命令失效了(到现在还不知道为毛失效。有请大神解答。。。。)
    jomato-MacBook-Pro:~ xxx$ sudo pip install Scrapy
    sudo: pip: command not found


    于是安静地重新 get 了一遍 pip


    jomato-MacBook-Pro:~ xxx$ sudo python get-pip.py
    The directory '/Users/xxx/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
    The directory '/Users/xxx/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
    Requirement already up-to-date: pip in /Library/Python/2.7/site-packages


    就在这个地方我忽然想起了之前安装好 pip ,直接执行 scrapy 安装命令,是调用 2.7 的版本,我如果 get python3 里面的 pip ,然后直接安装 scrapy ,系统是不是会直接调用 python3 的版本对 scrapy 进行安装呢?于是尝试了一下。


    jomato-MacBook-Pro:~ xxx$ sudo python3 get-pip.py
    The directory '/Users/xxx/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
    The directory '/Users/xxx/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
    Collecting pip
    Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 1.1MB/s
    Collecting wheel
    Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 5.4MB/s
    Installing collected packages: pip, wheel
    Found existing installation: pip 8.1.1
    Uninstalling pip-8.1.1:
    Successfully uninstalled pip-8.1.1
    Successfully installed pip-8.1.2 wheel-0.29.0
    jomato-MacBook-Pro:~ xxx$ sudo pip install Scrapy
    The directory '/Users/xxx/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
    The directory '/Users/xxx/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
    Collecting Scrapy
    Downloading Scrapy-1.1.2-py2.py3-none-any.whl (295kB)
    100% |████████████████████████████████| 296kB 1.6MB/s
    Collecting cssselect>=0.9 (from Scrapy)
    Downloading cssselect-0.9.2-py2.py3-none-any.whl
    Collecting six>=1.5.2 (from Scrapy)
    Downloading six-1.10.0-py2.py3-none-any.whl
    Collecting PyDispatcher>=2.0.5 (from Scrapy)
    Downloading PyDispatcher-2.0.5.tar.gz
    Collecting service-identity (from Scrapy)
    Downloading service_identity-16.0.0-py2.py3-none-any.whl
    Collecting parsel>=0.9.3 (from Scrapy)
    Downloading parsel-1.0.3-py2.py3-none-any.whl
    Collecting Twisted>=10.0.0 (from Scrapy)
    Downloading Twisted-16.4.0.tar.bz2 (3.0MB)
    100% |████████████████████████████████| 3.0MB 392kB/s
    Collecting queuelib (from Scrapy)
    Downloading queuelib-1.4.2-py2.py3-none-any.whl
    Collecting w3lib>=1.14.2 (from Scrapy)
    Downloading w3lib-1.15.0-py2.py3-none-any.whl
    Collecting lxml (from Scrapy)
    Downloading lxml-3.6.4.tar.gz (3.7MB)
    100% |████████████████████████████████| 3.7MB 268kB/s
    Collecting pyOpenSSL (from Scrapy)
    Downloading pyOpenSSL-16.1.0-py2.py3-none-any.whl (43kB)
    100% |████████████████████████████████| 51kB 5.6MB/s
    Collecting attrs (from service-identity->Scrapy)
    Downloading attrs-16.0.0-py2.py3-none-any.whl
    Collecting pyasn1 (from service-identity->Scrapy)
    Downloading pyasn1-0.1.9-py2.py3-none-any.whl
    Collecting pyasn1-modules (from service-identity->Scrapy)
    Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl
    Collecting zope.interface>=4.0.2 (from Twisted>=10.0.0->Scrapy)
    Downloading zope.interface-4.2.0-cp35-cp35m-macosx_10_9_x86_64.whl (130kB)
    100% |████████████████████████████████| 133kB 838kB/s
    Collecting cryptography>=1.3.4 (from pyOpenSSL->Scrapy)
    Downloading cryptography-1.5-cp35-cp35m-macosx_10_10_x86_64.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 539kB/s
    Requirement already satisfied (use --upgrade to upgrade): setuptools in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (from zope.interface>=4.0.2->Twisted>=10.0.0->Scrapy)
    Collecting idna>=2.0 (from cryptography>=1.3.4->pyOpenSSL->Scrapy)
    Downloading idna-2.1-py2.py3-none-any.whl (54kB)
    100% |████████████████████████████████| 61kB 5.6MB/s
    Collecting cffi>=1.4.1 (from cryptography>=1.3.4->pyOpenSSL->Scrapy)
    Downloading cffi-1.7.0-cp35-cp35m-macosx_10_10_x86_64.whl (154kB)
    100% |████████████████████████████████| 163kB 1.2MB/s
    Collecting pycparser (from cffi>=1.4.1->cryptography>=1.3.4->pyOpenSSL->Scrapy)
    Downloading pycparser-2.14.tar.gz (223kB)
    100% |████████████████████████████████| 225kB 1.2MB/s
    Installing collected packages: cssselect, six, PyDispatcher, attrs, pyasn1, idna, pycparser, cffi, cryptography, pyOpenSSL, pyasn1-modules, service-identity, lxml, w3lib, parsel, zope.interface, Twisted, queuelib, Scrapy
    Running setup.py install for PyDispatcher ... done
    Running setup.py install for pycparser ... done
    Running setup.py install for lxml ... done
    Running setup.py install for Twisted ... done
    Successfully installed PyDispatcher-2.0.5 Scrapy-1.1.2 Twisted-16.4.0 attrs-16.0.0 cffi-1.7.0 cryptography-1.5 cssselect-0.9.2 idna-2.1 lxml-3.6.4 parsel-1.0.3 pyOpenSSL-16.1.0 pyasn1-0.1.9 pyasn1-modules-0.0.8 pycparser-2.14 queuelib-1.4.2 service-identity-16.0.0 six-1.10.0 w3lib-1.15.0 zope.interface-4.2.0

    除去前面 python3 的安装和 Homebrew 的安装以外,就用以下两条命令即可完成,亲测可用。
    sudo python3 get-pip.py
    sudo pip install Scrapy


    虽然还没回过神来,但确实看到了 successfull 的字样,逗比的 scrapy 之旅就此开启,系统版本 OS X EI Capitan.希望对入坑的朋友有所帮助。
    17 条回复    2018-05-03 17:02:25 +08:00
    TimePPT
        1
    TimePPT  
       2016-08-28 22:04:54 +08:00 via iPhone   ❤️ 2
    Mac 下玩 python 的正确姿势难道不是 pyenv + virtualenv + pip 么?
    Yc1992
        2
    Yc1992  
       2016-08-28 22:06:46 +08:00
    @TimePPT 最近 conda 比较 fashion
    TimePPT
        3
    TimePPT  
       2016-08-28 22:10:17 +08:00 via iPhone
    @Yc1992 嗯,看到有人推过介绍,回头瞅瞅
    Mirage09
        4
    Mirage09  
       2016-08-28 22:25:30 +08:00 via iPhone
    看来我当时死也装不上 scrapy 的原因就是没用 Python3 ……
    187j3x1
        5
    187j3x1  
       2016-08-28 22:36:39 +08:00
    这暴力 sudo ,果然入坑,小心后遗症。
    Verydiao
        6
    Verydiao  
    OP
       2016-08-28 22:43:10 +08:00
    @TimePPT 求大神赐链接,包全套么,我以为直接这样就可以用了哈哈。小白不懂啊。
    mind3x
        7
    mind3x  
       2016-08-29 09:16:37 +08:00 via Android
    用 Docker 啊

    https://hub.docker.com/r/rickl/docker-scrapy/

    docker pull rickl/docker-scrapy

    只要 34MB 。无论 OS X, win, Linux 一行搞定。
    TimePPT
        8
    TimePPT  
       2016-08-29 10:03:56 +08:00
    @Verydiao 非大神,只是平时爱折腾点小脚本啥的 233 。

    话说回来,这坑我也趟过😂,最简单粗暴的办法是把重要资料备份了抹盘重装,因为可能连你自己都不记得中间到底操作过啥。

    系统自带的程序环境有很多是给系统的依赖,所以自己的编程环境最好是和系统环境做隔离,最好办法就是用虚拟环境工具模拟一个独立的操作环境出来。

    至于怎么搭建,其实楼里已经给了好几个关键词了,或者搜「 python 编程环境搭建」应该有不少文章。

    小白的话善用 Google ,八成的问题都能解决。

    另外,没事瞎 sudo 是个坏习惯,别问我怎么知道的,说多了都是泪
    dotpig
        9
    dotpig  
       2016-08-29 10:05:13 +08:00
    系统自带 python 2.7 和 homebrew 安装 python 3.5 一点不矛盾。特意去试了一下 pip3 install scrapy, 一个 error 都找不到。你这样到处 sudo 还是蛮厉害的, homebrew 不就是为了不要处处 sudo 嘛。
    TimePPT
        10
    TimePPT  
       2016-08-29 10:07:38 +08:00
    以及,如果确实小白随便玩玩编程不当饭吃,用 virtualbox 啊 PD 啊装个 ubuntu 随便造吧,造之前只用存个快照,玩坏了回滚起来也方便。
    brucedone
        11
    brucedone  
       2016-08-29 10:12:49 +08:00
    why not anaconda ?
    xsstomy
        12
    xsstomy  
       2016-08-29 11:49:33 +08:00
    跟楼主差不多,当时装了好久才装上去,然后就玩了一下,然后就没有然后了。
    quxw
        13
    quxw  
       2016-08-29 17:03:11 +08:00
    很麻烦吗?!点在哪?
    woostundy
        14
    woostundy  
       2016-08-29 19:24:44 +08:00
    @mind3x docker 能在 windows 和 os x 上跑吗?
    jyf007
        15
    jyf007  
       2016-08-29 20:02:40 +08:00 via Android
    gentoo 路过,可以共存,十分可怕。
    mind3x
        16
    mind3x  
       2016-08-29 21:21:53 +08:00
    @woostundy 从 Docker 1.12 开始。之前需要 VirtualBox ,现在不用了。
    Verydiao
        17
    Verydiao  
    OP
       2018-05-03 17:02:25 +08:00
    @187j3x1 系统已崩溃,应该是后遗症所致。。。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   我们的愿景   ·   实用小工具   ·   3871 人在线   最高记录 6543   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 29ms · UTC 10:39 · PVG 18:39 · LAX 03:39 · JFK 06:39
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.