conda-mirror

将上游conda通道镜像到本地目录。

安装

conda-mirror可在PyPI和conda-forge上使用。

使用以下方式安装

pip install conda-mirror

或

conda install conda-mirror -c conda-forge

兼容性

conda-mirror是有意设计的仅适用于py3的包

命令行界面

conda-mirror.py的命令行界面

usage: conda-mirror [-h] [--upstream-channel UPSTREAM_CHANNEL]
                    [--target-directory TARGET_DIRECTORY]
                    [--temp-directory TEMP_DIRECTORY] [--platform PLATFORM]
                    [-D] [-v] [--config CONFIG] [--pdb]
                    [--num-threads NUM_THREADS] [--version] [--dry-run]
                    [--no-validate-target]
                    [--minimum-free-space MINIMUM_FREE_SPACE] [--proxy PROXY]
                    [--ssl-verify SSL_VERIFY] [-k]
                    [--max-retries MAX_RETRIES] [--no-progress]

CLI interface for conda-mirror.py

optional arguments:
  -h, --help            show this help message and exit
  --upstream-channel UPSTREAM_CHANNEL
                        The target channel to mirror. Can be a channel on
                        anaconda.org like "conda-forge" or a full qualified
                        channel like "https://repo.continuum.io/pkgs/free/"
  --target-directory TARGET_DIRECTORY
                        The place where packages should be mirrored to
  --temp-directory TEMP_DIRECTORY
                        Temporary download location for the packages.
                        Defaults to a randomly selected temporary directory.
                        Note that you might need to specify a different
                        location if your default temp directory has less
                        available space than your mirroring target
  --platform PLATFORM   The OS platform(s) to mirror. one of: {'linux-64',
                        'linux-32','osx-64', 'win-32', 'win-64'}
  -D, --include-depends
                        Include packages matching any dependencies of
                        packages in whitelist.
  -v, --verbose         logging defaults to error/exception only. Takes up to
                        three '-v' flags. '-v': warning. '-vv': info. '-vvv':
                        debug.
  --config CONFIG       Path to the yaml config file
  --pdb                 Enable PDB debugging on exception
  --num-threads NUM_THREADS
                        Num of threads for validation. 1: Serial mode. 0: All
                        available.
  --version             Print version and quit
  --dry-run             Show what will be downloaded and what will be
                        removed. Will not validate existing packages
  --no-validate-target  Skip validation of files already present in target-
                        directory
  --minimum-free-space MINIMUM_FREE_SPACE
                        Threshold for free diskspace. Given in megabytes.
  --proxy PROXY         Proxy URL to access internet if needed
  --ssl-verify SSL_VERIFY, --ssl_verify SSL_VERIFY
                        Path to a CA_BUNDLE file with certificates of trusted
                        CAs, this may be "False" to disable verification as
                        per the requests API.
  -k, --insecure        Allow conda to perform "insecure" SSL connections and
                        transfers. Equivalent to setting 'ssl_verify' to
                        'false'.
  --max-retries MAX_RETRIES
                        Maximum number of retries before a download error is
                        reraised, defaults to 100
  --no-progress         Do not display progress bars.

示例用法

警告：执行此命令将拉取约10GB，并至少需要一小时

conda-mirror --upstream-channel conda-forge --target-directory local_mirror --platform linux-64

更多信息

黑名单/白名单配置

example-conf.yaml

blacklist:
  - license: "*agpl*"
  - license: None
  - license: ""

whitelist:
  - name: system

blacklist从上游repodata中删除匹配条件（列表）的软件包。

白名单重新包含匹配白名单条件的任何黑名单中的软件包。

黑名单和白名单都采用字典列表。字典中的键必须是 repodata.json 元数据中的值。值是用于匹配的 (Unix) glob，但在 version 属性的情况下，也可以使用 conda 软件包匹配版本规范。

上游 "defaults" 频道的完整 repodata 请访问此处：http://conda.anaconda.org/anaconda/linux-64/repodata.json

以下是 repodata['packages'] 中条目之一的内容

{'botocore-1.4.10-py34_0.tar.bz2': {'arch': 'x86_64',
  'binstar': {'channel': 'main',
   'owner_id': '55fc8527d3234d09d4951c71',
   'package_id': '56b88ea1be1cc95a362b218e'},
  'build': 'py34_0',
  'build_number': 0,
  'date': '2016-04-11',
  'depends': ['docutils >=0.10',
   'jmespath >=0.7.1,<1.0.0',
   'python 3.4*',
   'python-dateutil >=2.1,<3.0.0'],
  'license': 'Apache',
  'md5': 'b35a5c1240ba672e0d9d1296141e383c',
  'name': 'botocore',
  'platform': 'linux',
  'requires': [],
  'size': 1831799,
  'version': '1.4.10'}}

有关更多信息，请参阅 conda_mirror:match 函数的实现细节。

常见用法模式

仅镜像一个特定软件包

如果您想与您的配置匹配上面列出的 botocore 软件包，则可以使用以下配置首先黑名单所有软件包，然后只包含 botocore 软件包

blacklist:
  - name: "*"
whitelist:
  - name: botocore
    version: 1.4.10
    build: py34_0

您可以使用标准的 conda 软件包版本指定符来过滤一系列版本

blacklist:
  - name: "*"
whitelist:
  - name: botocore
    version: ">=1.4.10,<1.5"

镜像所有软件包，但不包括 agpl 许可证

blacklist:
  - license: "*agpl*"

仅镜像 Python 3 软件包

blacklist:
  - name: "*"
whitelist:
  - build: "*py3*"

镜像指定的软件包及其依赖项

这将包括至少版本 1.4.10 的所有 botocore 实例，以及与其依赖项匹配的任何软件包（以及这些软件包的依赖项）。

blacklist:
  - name: "*"
whitelist:
  - name: botocore
    version: ">=1.4.10"
include_depends: True

如果包含太多的软件包版本，您可以在白名单中添加更多条目以限制要包含的内容。

测试

安装测试需求

注意：将安装 pip 软件包

$ pip install -r test-requirements.txt
Requirement already satisfied: pytest in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 1))
Requirement already satisfied: coverage in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 2))
Requirement already satisfied: pytest-ordering in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 3))
Requirement already satisfied: py>=1.4.29 in /home/edill/miniconda/lib/python3.5/site-packages (from pytest->-r test-requirements.txt (line 1))

运行测试，使用 coverage 工具调用。

$ coverage run run_tests.py
sys.argv=['run_tests.py']
========================================= test session starts ==========================================
platform linux -- Python 3.5.3, pytest-3.0.6, py-1.4.31, pluggy-0.4.0 -- /home/edill/miniconda/bin/python
cachedir: .cache
rootdir: /home/edill/dev/maxpoint/github/conda-mirror, inifile:
plugins: xonsh-0.5.2, ordering-0.4
collected 4 items

test/test_conda_mirror.py::test_match PASSED
test/test_conda_mirror.py::test_cli[https://repo.continuum.io/pkgs/free-linux-64] PASSED
test/test_conda_mirror.py::test_cli[conda-forge-linux-64] PASSED
test/test_conda_mirror.py::test_handling_bad_package PASSED

======================================= 4 passed in 4.41 seconds =======================================

显示覆盖率统计信息

$ coverage report -m
Name                           Stmts   Miss  Cover   Missing
------------------------------------------------------------
conda_mirror/__init__.py           3      0   100%
conda_mirror/conda_mirror.py     236     20    92%   203-205, 209-210, 214, 240, 249-254, 262-264, 303, 366, 497, 542-543, 629
------------------------------------------------------------
TOTAL                            239     20    92%

其他

当新贡献者提交的拉取请求被批准后，我们将联系您，邀请您成为项目的维护者。

发布

要发布，您需要三样东西

1. 对 conda-mirror 的提交权限
2. GitHub 令牌
3. 您想用于新标签的版本号

在您有了这三样东西之后，运行 release.sh 脚本（在 Unix 机器上），并传递您想要使用的标签和您的 GitHub 令牌

GITHUB_TOKEN=<github_token> ./release.sh <tag>