requests-html 0.10.0

Requests-HTML：为人类设计的HTML解析™

这个库旨在使解析HTML（例如，网络抓取）尽可能简单直观。

如果您有兴趣从经济上支持Kenneth Reitz的开源项目，请考虑 访问此链接。您的支持对维持动力至关重要，因为开源已不再是我的日常工作的一部分。

使用此库时，您将自动获得

• 完整的JavaScript支持!

• CSS选择器（即jQuery风格的，多亏了PyQuery）。

• XPath选择器，适合那些胆小的人。

• 模拟用户代理（就像真正的网络浏览器一样）。

• 自动跟随重定向。

• 连接池和Cookie持久性。

• 您所熟悉和喜爱的请求体验，拥有魔法般的解析能力。

• 异步支持

教程 & 使用

使用Requests向‘python.org’发送GET请求

>>> from requests_html import HTMLSession
>>> session = HTMLSession()

>>> r = session.get('https://pythonlang.cn/')

尝试异步并同时获取一些网站

>>> from requests_html import AsyncHTMLSession
>>> asession = AsyncHTMLSession()

>>> async def get_pythonorg():
...    r = await asession.get('https://pythonlang.cn/')

>>> async def get_reddit():
...    r = await asession.get('https://reddit.com/')

>>> async def get_google():
...    r = await asession.get('https://google.com/')

>>> result = session.run(get_pythonorg, get_reddit, get_google)

获取页面中所有链接的列表，原样（排除锚点）

>>> r.html.links
{'//docs.pythonlang.cn/3/tutorial/', '/about/apps/', 'https://github.com/python/pythondotorg/issues', '/accounts/login/', '/dev/peps/', '/about/legal/', '//docs.pythonlang.cn/3/tutorial/introduction.html#lists', '/download/alternatives', 'http://feedproxy.google.com/~r/PythonInsider/~3/kihd2DW98YY/python-370a4-is-available-for-testing.html', '/download/other/', '/downloads/windows/', 'https://mail.python.org/mailman/listinfo/python-dev', '/doc/av', 'https://devguide.pythonlang.cn/', '/about/success/#engineering', 'https://wiki.python.org/moin/PythonEventsCalendar#Submitting_an_Event', 'https://www.openstack.org', '/about/gettingstarted/', 'http://feedproxy.google.com/~r/PythonInsider/~3/AMoBel8b8Mc/python-3.html', '/success-stories/industrial-light-magic-runs-python/', 'https://docs.pythonlang.cn/3/tutorial/introduction.html#using-python-as-a-calculator', '/', 'http://pyfound.blogspot.com/', '/events/python-events/past/', '/downloads/release/python-2714/', 'https://wiki.python.org/moin/PythonBooks', 'http://plus.google.com/+Python', 'https://wiki.python.org/moin/', 'https://status.python.org/', '/community/workshops/', '/community/lists/', 'http://buildbot.net/', '/community/awards', 'http://twitter.com/ThePSF', 'https://docs.pythonlang.cn/3/license.html', '/psf/donations/', 'http://wiki.python.org/moin/Languages', '/dev/', '/events/python-user-group/', 'https://wiki.qt.io/PySide', '/community/sigs/', 'https://wiki.gnome.org/Projects/PyGObject', 'https://ansible.org.cn', 'http://www.saltstack.com', 'http://planetpython.org/', '/events/python-events', '/about/help/', '/events/python-user-group/past/', '/about/success/', '/psf-landing/', '/about/apps', '/about/', 'http://www.wxpython.org/', '/events/python-user-group/665/', 'https://pythonlang.cn/psf/codeofconduct/', '/dev/peps/peps.rss', '/downloads/source/', '/psf/sponsorship/sponsors/', 'http://bottlepy.org', 'http://roundup.sourceforge.net/', 'https://pandas.ac.cn/', 'http://brochure.getpython.info/', 'https://bugs.python.org/', '/community/merchandise/', 'http://tornadoweb.org', '/events/python-user-group/650/', 'https://flask.org.cn/', '/downloads/release/python-364/', '/events/python-user-group/660/', '/events/python-user-group/638/', '/psf/', '/doc/', 'http://blog.python.org', '/events/python-events/604/', '/about/success/#government', 'https://pythonlang.cn/dev/peps/', 'https://docs.pythonlang.cn', 'http://feedproxy.google.com/~r/PythonInsider/~3/zVC80sq9s00/python-364-is-now-available.html', '/users/membership/', '/about/success/#arts', 'https://wiki.python.org/moin/Python2orPython3', '/downloads/', '/jobs/', 'http://trac.edgewall.org/', 'http://feedproxy.google.com/~r/PythonInsider/~3/wh73_1A-N7Q/python-355rc1-and-python-348rc1-are-now.html', '/privacy/', 'https://pypi.python.org/', 'http://www.riverbankcomputing.co.uk/software/pyqt/intro', 'https://scipy.org.cn', '/community/forums/', '/about/success/#scientific', '/about/success/#software-development', '/shell/', '/accounts/signup/', 'https://#/pythonlang?fref=ts', '/community/', 'https://kivy.org/', '/about/quotes/', 'http://www.web2py.com/', '/community/logos/', '/community/diversity/', '/events/calendars/', 'https://wiki.python.org/moin/BeginnersGuide', '/success-stories/', '/doc/essays/', '/dev/core-mentorship/', 'https://ipython.pythonlang.cn', '/events/', '//docs.pythonlang.cn/3/tutorial/controlflow.html', '/about/success/#education', '/blogs/', '/community/irc/', 'http://pycon.blogspot.com/', '//jobs.python.org', 'http://www.pylonsproject.org/', 'https://django.ac.cn/', '/downloads/mac-osx/', '/about/success/#business', 'http://feedproxy.google.com/~r/PythonInsider/~3/x_c9D0S-4C4/python-370b1-is-now-available-for.html', 'http://wiki.python.org/moin/TkInter', 'https://docs.pythonlang.cn/faq/', '//docs.pythonlang.cn/3/tutorial/controlflow.html#defining-functions'}

获取页面中所有链接的列表，绝对形式（排除锚点）

>>> r.html.absolute_links
{'https://github.com/python/pythondotorg/issues', 'https://docs.pythonlang.cn/3/tutorial/', 'https://pythonlang.cn/about/success/', 'http://feedproxy.google.com/~r/PythonInsider/~3/kihd2DW98YY/python-370a4-is-available-for-testing.html', 'https://pythonlang.cn/dev/peps/', 'https://mail.python.org/mailman/listinfo/python-dev', 'https://pythonlang.cn/doc/', 'https://pythonlang.cn/', 'https://pythonlang.cn/about/', 'https://pythonlang.cn/events/python-events/past/', 'https://devguide.pythonlang.cn/', 'https://wiki.python.org/moin/PythonEventsCalendar#Submitting_an_Event', 'https://www.openstack.org', 'http://feedproxy.google.com/~r/PythonInsider/~3/AMoBel8b8Mc/python-3.html', 'https://docs.pythonlang.cn/3/tutorial/introduction.html#lists', 'https://docs.pythonlang.cn/3/tutorial/introduction.html#using-python-as-a-calculator', 'http://pyfound.blogspot.com/', 'https://wiki.python.org/moin/PythonBooks', 'http://plus.google.com/+Python', 'https://wiki.python.org/moin/', 'https://pythonlang.cn/events/python-events', 'https://status.python.org/', 'https://pythonlang.cn/about/apps', 'https://pythonlang.cn/downloads/release/python-2714/', 'https://pythonlang.cn/psf/donations/', 'http://buildbot.net/', 'http://twitter.com/ThePSF', 'https://docs.pythonlang.cn/3/license.html', 'http://wiki.python.org/moin/Languages', 'https://docs.pythonlang.cn/faq/', 'https://jobs.python.org', 'https://pythonlang.cn/about/success/#software-development', 'https://pythonlang.cn/about/success/#education', 'https://pythonlang.cn/community/logos/', 'https://pythonlang.cn/doc/av', 'https://wiki.qt.io/PySide', 'https://pythonlang.cn/events/python-user-group/660/', 'https://wiki.gnome.org/Projects/PyGObject', 'https://ansible.org.cn', 'http://www.saltstack.com', 'https://pythonlang.cn/dev/peps/peps.rss', 'http://planetpython.org/', 'https://pythonlang.cn/events/python-user-group/past/', 'https://docs.pythonlang.cn/3/tutorial/controlflow.html#defining-functions', 'https://pythonlang.cn/community/diversity/', 'https://docs.pythonlang.cn/3/tutorial/controlflow.html', 'https://pythonlang.cn/community/awards', 'https://pythonlang.cn/events/python-user-group/638/', 'https://pythonlang.cn/about/legal/', 'https://pythonlang.cn/dev/', 'https://pythonlang.cn/download/alternatives', 'https://pythonlang.cn/downloads/', 'https://pythonlang.cn/community/lists/', 'http://www.wxpython.org/', 'https://pythonlang.cn/about/success/#government', 'https://pythonlang.cn/psf/', 'https://pythonlang.cn/psf/codeofconduct/', 'http://bottlepy.org', 'http://roundup.sourceforge.net/', 'https://pandas.ac.cn/', 'http://brochure.getpython.info/', 'https://pythonlang.cn/downloads/source/', 'https://bugs.python.org/', 'https://pythonlang.cn/downloads/mac-osx/', 'https://pythonlang.cn/about/help/', 'http://tornadoweb.org', 'https://flask.org.cn/', 'https://pythonlang.cn/users/membership/', 'http://blog.python.org', 'https://pythonlang.cn/privacy/', 'https://pythonlang.cn/about/gettingstarted/', 'https://pythonlang.cn/dev/peps/', 'https://pythonlang.cn/about/apps/', 'https://docs.pythonlang.cn', 'https://pythonlang.cn/success-stories/', 'https://pythonlang.cn/community/forums/', 'http://feedproxy.google.com/~r/PythonInsider/~3/zVC80sq9s00/python-364-is-now-available.html', 'https://pythonlang.cn/community/merchandise/', 'https://pythonlang.cn/about/success/#arts', 'https://wiki.python.org/moin/Python2orPython3', 'http://trac.edgewall.org/', 'http://feedproxy.google.com/~r/PythonInsider/~3/wh73_1A-N7Q/python-355rc1-and-python-348rc1-are-now.html', 'https://pypi.python.org/', 'https://pythonlang.cn/events/python-user-group/650/', 'http://www.riverbankcomputing.co.uk/software/pyqt/intro', 'https://pythonlang.cn/about/quotes/', 'https://pythonlang.cn/downloads/windows/', 'https://pythonlang.cn/events/calendars/', 'https://scipy.org.cn', 'https://pythonlang.cn/community/workshops/', 'https://pythonlang.cn/blogs/', 'https://pythonlang.cn/accounts/signup/', 'https://pythonlang.cn/events/', 'https://kivy.org/', 'https://#/pythonlang?fref=ts', 'http://www.web2py.com/', 'https://pythonlang.cn/psf/sponsorship/sponsors/', 'https://pythonlang.cn/community/', 'https://pythonlang.cn/download/other/', 'https://pythonlang.cn/psf-landing/', 'https://pythonlang.cn/events/python-user-group/665/', 'https://wiki.python.org/moin/BeginnersGuide', 'https://pythonlang.cn/accounts/login/', 'https://pythonlang.cn/downloads/release/python-364/', 'https://pythonlang.cn/dev/core-mentorship/', 'https://pythonlang.cn/about/success/#business', 'https://pythonlang.cn/community/sigs/', 'https://pythonlang.cn/events/python-user-group/', 'https://ipython.pythonlang.cn', 'https://pythonlang.cn/shell/', 'https://pythonlang.cn/community/irc/', 'https://pythonlang.cn/about/success/#engineering', 'http://www.pylonsproject.org/', 'http://pycon.blogspot.com/', 'https://pythonlang.cn/about/success/#scientific', 'https://pythonlang.cn/doc/essays/', 'https://django.ac.cn/', 'https://pythonlang.cn/success-stories/industrial-light-magic-runs-python/', 'http://feedproxy.google.com/~r/PythonInsider/~3/x_c9D0S-4C4/python-370b1-is-now-available-for.html', 'http://wiki.python.org/moin/TkInter', 'https://pythonlang.cn/jobs/', 'https://pythonlang.cn/events/python-events/604/'}

使用CSS选择器选择元素

>>> about = r.html.find('#about', first=True)

获取元素的文本内容

>>> print(about.text)
About
Applications
Quotes
Getting Started
Help
Python Brochure

检查元素属性

>>> about.attrs
{'id': 'about', 'class': ('tier-1', 'element-1'), 'aria-haspopup': 'true'}

渲染元素的HTML

>>> about.html
'<li aria-haspopup="true" class="tier-1 element-1 " id="about">\n<a class="" href="/about/" title="">About</a>\n<ul aria-hidden="true" class="subnav menu" role="menu">\n<li class="tier-2 element-1" role="treeitem"><a href="/about/apps/" title="">Applications</a></li>\n<li class="tier-2 element-2" role="treeitem"><a href="/about/quotes/" title="">Quotes</a></li>\n<li class="tier-2 element-3" role="treeitem"><a href="/about/gettingstarted/" title="">Getting Started</a></li>\n<li class="tier-2 element-4" role="treeitem"><a href="/about/help/" title="">Help</a></li>\n<li class="tier-2 element-5" role="treeitem"><a href="http://brochure.getpython.info/" title="">Python Brochure</a></li>\n</ul>\n</li>'

在元素内部选择元素

>>> about.find('a')
[<Element 'a' href='/about/' title='' class=''>, <Element 'a' href='/about/apps/' title=''>, <Element 'a' href='/about/quotes/' title=''>, <Element 'a' href='/about/gettingstarted/' title=''>, <Element 'a' href='/about/help/' title=''>, <Element 'a' href='http://brochure.getpython.info/' title=''>]

在元素内搜索链接

>>> about.absolute_links
{'http://brochure.getpython.info/', 'https://pythonlang.cn/about/gettingstarted/', 'https://pythonlang.cn/about/', 'https://pythonlang.cn/about/quotes/', 'https://pythonlang.cn/about/help/', 'https://pythonlang.cn/about/apps/'}

在页面中搜索文本

>>> r.html.search('Python is a {} language')[0]
programming

更复杂的CSS选择器示例（从Chrome开发者工具复制）

>>> r = session.get('https://github.com/')
>>> sel = 'body > div.application-main > div.jumbotron.jumbotron-codelines > div > div > div.col-md-7.text-center.text-md-left > p'

>>> print(r.html.find(sel, first=True).text)
GitHub is a development platform inspired by the way you work. From open source to business, you can host and review code, manage projects, and build software alongside millions of other developers.

也支持XPath

>>> r.html.xpath('/html/body/div[1]/a')
[<Element 'a' class=('px-2', 'py-4', 'show-on-focus', 'js-skip-to-content') href='#start-of-content' tabindex='1'>]

JavaScript支持

让我们获取由JavaScript渲染的文本

>>> r = session.get('http://python-requests.org')

>>> r.html.render()

>>> r.html.search('Python 2 will retire in only {months} months!')['months']
'<time>25</time>'

或者你也可以这样做异步

>>> r = asession.get('http://python-requests.org/')

>>> await r.html.arender()

>>> r.html.search('Python 2 will retire in only {months} months!')['months']
'<time>25</time>'

注意，您第一次运行 render() 方法时，它将在您的家目录中下载Chromium（例如 ~/.pyppeteer/）。这只会发生一次。

不使用Requests使用

您也可以不使用Requests使用这个库

>>> from requests_html import HTML
>>> doc = """<a href='https://httpbin.org'>"""

>>> html = HTML(html=doc)
>>> html.links
{'https://httpbin.org'}

安装

$ pipenv install requests-html
✨🍰✨

仅支持 Python 3.6。