pymcurl 8.6.0.1

使用 Curl & CurlMulti 管理出站 HTTP 连接

描述

mcurl 是对 libcurl 的高层 API 封装，简化了与 libcurl easy 和 multi 接口的交互。最初是为 Px 代理服务器创建的，该服务器使用 libcurl 来处理上游代理身份验证。

用法

可以使用 pip 安装 mcurl

pip install pymcurl

以下平台提供二进制 包：

• aarch64-linux-gnu
• aarch64-linux-musl
• i686-linux-gnu
• x86_64-linux-gnu
• x86_64-linux-musl
• x86_64-macos
• x86_64-windows

mcurl 利用 cffi 与 libcurl 交互，所有二进制依赖项均来自 binarybuilder.org。在 Linux 上使用 auditwheel，在 MacOS 上使用 delocate，在 Windows 上使用 delvewheel 将共享库打包到 wheel 中。

感谢 cffi 和 Py_LIMITED_API，这些 mcurl 二进制文件应适用于从 v3.2 开始的任何 Python 版本。

简单接口

from mcurl import Curl

c = Curl('http://httpbin.org/get')
c.buffer()
ret = c.perform()
if ret == 0:
    ret, resp = c.get_response()
    headers = c.get_headers()
    data = c.get_data()
    print(f"Response: {resp}\n\n{headers}{data}")

多接口

from mcurl import Curl, MCurl

m = MCurl()

c1 = Curl('http://httpbin.org/get')
c1.buffer()
m.add(c1)

data = "test8192".encode("utf-8")
c2 = Curl('https://httpbin.org/post', 'POST')
c2.buffer(data=data)
c2.set_headers({"Content-Length": len(data)})
m.add(c2)

ret1 = m.do(c1)
ret2 = m.do(c2)

if ret1:
    c1.get_response()
    c1.get_headers()
    c1.get_data()
    print(f"Response: {c1.get_response()}\n\n{c1.get_headers()}{c1.get_data()}")
else:
    print(f"Failed with error: {c1.errstr}")

if ret2:
    c2.get_response()
    c2.get_headers()
    c2.get_data()
    print(f"Response: {c2.get_response()}\n\n{c2.get_headers()}{c2.get_data()}")
else:
    print(f"Failed with error: {c2.errstr}")

m.close()

libcurl API

如果需要，可以直接访问 libcurl API，就像在 mcurl 中做的那样。

from _libcurl_cffi import lib as libcurl
from _libcurl_cffi import ffi

url = "http://httpbin.org/get"
curl = ffi.new("char []", url.encode("utf-8"))

easy = libcurl.curl_easy_init()
libcurl.curl_easy_setopt(easy, libcurl.CURLOPT_URL, curl)
cerr = libcurl.curl_easy_perform(easy)

API 参考

NAME
    mcurl - Manage outbound HTTP connections using Curl & CurlMulti

CLASSES
    builtins.object
        Curl
        MCurl

    class Curl(builtins.object)
     |  Curl(url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
     |
     |  Helper class to manage a curl easy instance
     |
     |  Methods defined here:
     |
     |  __del__(self)
     |      Destructor - clean up resources
     |
     |  __init__(self, url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
     |      Initialize curl instance
     |
     |      method = GET, POST, PUT, CONNECT, etc.
     |      request_version = HTTP/1.0, HTTP/1.1, etc.
     |
     |  bridge(self, client_rfile=None, client_wfile=None, client_hfile=None)
     |      Bridge curl reads/writes to sockets specified
     |
     |      Reads POST/PATCH data from client_rfile
     |      Writes data back to client_wfile
     |      Writes headers back to client_hfile
     |
     |  buffer(self, data=None)
     |      Setup buffers to bridge curl perform
     |
     |  get_activesocket(self)
     |      Return active socket for this easy instance
     |
     |  get_data(self, encoding='utf-8')
     |      Return data written by curl perform to buffer()
     |
     |      encoding = "utf-8" by default, change or set to None if bytes preferred
     |
     |  get_headers(self, encoding='utf-8')
     |      Return headers written by curl perform to buffer()
     |
     |      encoding = "utf-8" by default, change or set to None if bytes preferred
     |
     |  get_primary_ip(self)
     |      Return primary IP address of this easy instance
     |
     |  get_response(self)
     |      Return response code of completed request
     |
     |  perform(self)
     |      Perform the easy handle
     |
     |  reset(self, url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
     |      Reuse existing curl instance for another request
     |
     |  set_auth(self, user, password=None, auth='ANY')
     |      Set proxy authentication info - call after set_proxy() to enable auth caching
     |
     |  set_debug(self, enable=True)
     |      Enable debug output
     |        Call after set_proxy() and set_auth() to enable discovery and caching of proxy
     |        auth mechanism - libcurl does not provide an API to get this today - need to
     |        find it in sent header debug output
     |
     |  set_follow(self, enable=True)
     |      Set curl to follow 3xx responses
     |
     |  set_headers(self, xheaders)
     |      Set headers to send
     |
     |  set_insecure(self, enable=True)
     |      Set curl to ignore SSL errors
     |
     |  set_proxy(self, proxy, port=0, noproxy=None)
     |      Set proxy options - returns False if this proxy server has auth failures
     |
     |  set_transfer_decoding(self, enable=False)
     |      Set curl to turn off transfer decoding - let client do it
     |
     |  set_tunnel(self, tunnel=True)
     |      Set to tunnel through proxy if no proxy or proxy + auth
     |
     |  set_useragent(self, useragent)
     |      Set user agent to send
     |
     |  set_verbose(self, enable=True)
     |      Set verbose mode
     |

    class MCurl(builtins.object)
     |  MCurl(debug_print=None)
     |
     |  Helper class to manage a curl multi instance
     |
     |  Methods defined here:
     |
     |  __init__(self, debug_print=None)
     |      Initialize multi interface
     |
     |  add(self, curl: mcurl.Curl)
     |      Add a Curl handle to perform
     |
     |  close(self)
     |      Stop any running transfers and close this multi handle
     |
     |  do(self, curl: mcurl.Curl)
     |      Add a Curl handle and peform until completion
     |
     |  remove(self, curl: mcurl.Curl)
     |      Remove a Curl handle once done
     |
     |  select(self, curl: mcurl.Curl, client_sock, idle=30)
     |      Run select loop between client and curl
     |
     |  setopt(self, option, value)
     |      Configure multi options
     |
     |  stop(self, curl: mcurl.Curl)
     |      Stop a running curl handle and remove

FUNCTIONS
    curl_version()

    cvp2pystr(cvoidp)
        Convert void * to Python string

    debug_callback(easy, infotype, data, size, userp)
        Prints out curl debug info and headers sent/received

    dprint lambda x
        # Debug shortcut

    getauth(auth)
        Return auth value for specified authentication string

        Supported values can be found here: https://curl.se/libcurl/c/CURLOPT_HTTPAUTH.html

        Skip the CURLAUTH_ portion in input - e.g. getauth("ANY")

        To control which methods are available during proxy detection:
          Prefix NO to avoid method - e.g. NONTLM => ANY - NTLM
          Prefix SAFENO to avoid method - e.g. SAFENONTLM => ANYSAFE - NTLM
          Prefix ONLY to support only that method - e.g ONLYNTLM => ONLY + NTLM

    gethash(easy)
        Return hash value for easy to allow usage as a dict key

    header_callback(buffer, size, nitems, userdata)

    multi_timer_callback(multi, timeout_ms, userp)

    print_curl_version()
        Display curl version information

    py2cbool(pbool)
        Convert Python bool to long

    py2clong(plong)
        Convert Python int to long

    py2cstr(pstr)
        Convert Python string to char *

    py2custr(pstr)
        Convert Python string to char *

    read_callback(buffer, size, nitems, userdata)

    sanitized(msg)
        Hide user sensitive data from debug output

    save_auth(curl, msg)
        Find and cache proxy auth mechanism from headers sent by libcurl

    save_upstream(curl, msg)
        Find which server libcurl connected to - upstream proxy or target server

    socket_callback(easy, sock_fd, ev_bitmask, userp, socketp)

    sockopt_callback(clientp, sock_fd, purpose)

    wa_callback(easy, infotype, data, size, userp)
        curl debug callback to get info not provided by libcurl today
        - proxy auth mechanism from sent headers
        - upstream server connected to from curl info

    write_callback(buffer, size, nitems, userdata)

    yield_msgs(data, size)
        Generator for curl debug messages

构建 mcurl

mcurl 使用 Linux 上的 gcc、MacOS 上的 clang 和 Windows 上的 mingw-x64 构建。共享库通过 binarybuilder.org 使用 jbb 下载，适用于 Linux 和 Windows。MacOS 通过 brew 打包 libcurl 二进制文件。

build.sh 可以用于构建支持的所有平台上的 mcurl，包括 Windows。