构建分析库和工具，用于处理和加工构建日志。
项目描述
Thoth: 构建分析器
这是一个分析器集合，用于从构建日志中检索信息。
以示例为导
考虑由pip包安装程序生成的日志
日志文件内容
Processing /home/macermak/RedHat/aicoe/lab
Collecting networkx (from thoth-lab==0.0.3)
Collecting requests (from thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting pandas (from thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/e6/de/a0d3defd8f338eaf53ef716e40ef6d6c277c35d50e09b586e170169cdf0d/pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting plotly (from thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/fd/db/003b5cfbc710f4d4982440451185b952269e4080a57ae7e760a2ceb8ce0c/plotly-3.6.1-py2.py3-none-any.whl
Collecting decorator>=4.3.0 (from networkx->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/f1/cd/7c8240007e9716b14679bc217a1baefa4432aa30394f7e2ec40a52b1a708/decorator-4.3.2-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting python-dateutil>=2.5.0 (from pandas->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting pytz>=2011k (from pandas->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl
Collecting numpy>=1.12.0 (from pandas->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/f5/bf/4981bcbee43934f0adb8f764a1e70ab0ee5a448f6505bd04a87a2fda2a8b/numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting six (from plotly->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting retrying>=1.3.3 (from plotly->thoth-lab==0.0.3)
Collecting nbformat>=4.2 (from plotly->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/da/27/9a654d2b6cc1eaa517d1c5a4405166c7f6d72f04f6e7eea41855fe808a46/nbformat-4.4.0-py2.py3-none-any.whl
Collecting traitlets>=4.1 (from nbformat>=4.2->plotly->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl
Collecting jupyter-core (from nbformat>=4.2->plotly->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/1d/44/065d2d7bae7bebc06f1dd70d23c36da8c50c0f08b4236716743d706762a8/jupyter_core-4.4.0-py2.py3-none-any.whl
Collecting jsonschema!=2.5.0,>=2.4 (from nbformat>=4.2->plotly->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting ipython-genutils (from nbformat>=4.2->plotly->thoth-lab==0.0.3)
Using cached https://files.pythonhosted.org/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl
Building wheels for collected packages: thoth-lab
Running setup.py bdist_wheel for thoth-lab: started
Running setup.py bdist_wheel for thoth-lab: finished with status 'done'
Stored in directory: /tmp/pip-ephem-wheel-cache-_4menpyv/wheels/c6/6e/7d/ef751ae03b7d549816c6658c7f8065cb889a8de3f0c2b3ca6c
Successfully built thoth-lab
selinon 0.1.0rc9 requires celery>=4, which is not installed.
thoth-analyzer 0.1.0 requires distro, which is not installed.
thoth-storages 0.9.5 requires aiogremlin==3.2.6rc1, which is not installed.
tensorflow-tensorboard 1.5.1 has requirement bleach==1.5.0, but you'll have bleach 2.1.4 which is incompatible.
tensorflow-tensorboard 1.5.1 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible.
tensorboard 1.6.0 has requirement bleach==1.5.0, but you'll have bleach 2.1.4 which is incompatible.
tensorboard 1.6.0 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible.
notebook 5.7.0 has requirement jupyter-client>=5.2.0, but you'll have jupyter-client 5.1.0 which is incompatible.
notebook 5.7.0 has requirement pyzmq>=17, but you'll have pyzmq 16.0.3 which is incompatible.
kaggle 1.4.7.1 has requirement urllib3<1.23.0,>=1.15, but you'll have urllib3 1.24.1 which is incompatible.
botocore 1.11.3 has requirement urllib3<1.24,>=1.20, but you'll have urllib3 1.24.1 which is incompatible.
awscli 1.14.25 has requirement botocore==1.8.29, but you'll have botocore 1.11.3 which is incompatible.
awscli 1.14.25 has requirement colorama<=0.3.7,>=0.2.5, but you'll have colorama 0.3.9 which is incompatible.
awscli 1.14.25 has requirement PyYAML<=3.12,>=3.10, but you'll have pyyaml 3.13 which is incompatible.
thoth-storages 0.9.5 has requirement yarl<1.2, but you'll have yarl 1.3.0 which is incompatible.
spyder-kernels 0.2.6 has requirement jupyter-client>=5.2.3, but you'll have jupyter-client 5.1.0 which is incompatible.
spyder-kernels 0.2.6 has requirement pyzmq>=17, but you'll have pyzmq 16.0.3 which is incompatible.
rstcheck 2.2 has requirement sphinx<1.5,>=1.3, but you'll have sphinx 1.8.1 which is incompatible.
gremlinpython 3.2.6 has requirement six==1.10.0, but you'll have six 1.12.0 which is incompatible.
gremlinpython 3.2.6 has requirement tornado==4.4.1, but you'll have tornado 4.5.2 which is incompatible.
Installing collected packages: decorator, networkx, idna, urllib3, certifi, chardet, requests, six, python-dateutil, pytz, numpy, pandas, retrying, ipython-genutils, traitlets, jupyter-core, jsonschema, nbformat, plotly, thoth-lab
Found existing installation: decorator 4.3.2
Uninstalling decorator-4.3.2:
Successfully uninstalled decorator-4.3.2
Found existing installation: networkx 2.2
Uninstalling networkx-2.2:
Successfully uninstalled networkx-2.2
Found existing installation: idna 2.8
Uninstalling idna-2.8:
Successfully uninstalled idna-2.8
Found existing installation: urllib3 1.24.1
Uninstalling urllib3-1.24.1:
Successfully uninstalled urllib3-1.24.1
Found existing installation: certifi 2018.11.29
Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
You are using pip version 18.1, however version 19.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
在日志文件中，有一行76包含关于打破安装的包的信息：certifi。我们想要提取这个信息。在大多数情况下，我们希望获取一个包含关于这个特定构建日志观察结果的 报告。
from thoth.build_analysers.analysis import build_breaker_report
log: str = ...
report: dict = build_breaker_report(log)
结果是可以通过方便的 build_breaker_format_report 函数进一步格式化的字典。
from thoth.build_analysers.analysis import build_breaker_format_report
print(build_breaker_format_report(report))
您应该会看到如下内容
Build breaker:
{
"already_satisfied": null,
"source": "thoth-lab",
"target": "certifi",
"version_installed": null,
"version_specified": ">=2017.4.17"
}
Probable reason:
76: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Candidates:
[
"76: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall."
]
如何工作
逻辑非常简单（也很天真）。实际上，没有火箭科学，只是大量的字符串匹配。
直觉
当前的方法可以处理pip和pipenv包安装程序产生的日志。对于它们两者，都有一组模式，如果在日志中找到这些模式，可能表示存在错误。这些模式是按照PEP3101字符串格式化语法进行格式化的字符串，并且以pandas DataFrames的形式存储在resources文件夹中。
我们通过相关的模式集评估日志文件中每行的匹配“良好”程度，并根据给定的阈值对它们进行标记。目前，ERROR阈值设置为0到1.0的评分中的>=0.6。最后，我们尝试在日志文件中搜索包名以构建依赖关系表。我们过滤包含包名的ERROR，从而得到一个构建破坏者列表。
安装
该软件包可在PyPI上找到。
pip install thoth-build-analysers
一切就绪 :thumbsup
作者：Marek Cermak macermak@redhat.com，@AICoE - 项目Thoth
下载文件
下载适合您平台的文件。如果您不确定选择哪个，请了解更多关于安装包的信息。
源代码分发
thoth-build-analysers-0.1.0.tar.gz (745.0 kB 查看哈希值)
构建分发
关闭
thoth-build-analysers-0.1.0.tar.gz的哈希值
| 算法 | 哈希摘要 | |
|---|---|---|
| SHA256 | 375faf23f0faca723710b78abaacbb8bc62e2b903792ded1a362d6d7ad3b4303 |
|
| MD5 | 268b1a381eebe904e71564b119b35140 |
|
| BLAKE2b-256 | 7345d36b403f07a89f20c9441cfbf67e178330f4c6291e46792b0efcc5058658 |
关闭
thoth_build_analysers-0.1.0-py3-none-any.whl的哈希值
| 算法 | 哈希摘要 | |
|---|---|---|
| SHA256 | 7c9766828c6132e6dad8ae56db1db0af668ab07aa7370b6ea5ff0a65f64303c3 |
|
| MD5 | eb8783f59060fd92d1c4e64bbf705cd9 |
|
| BLAKE2b-256 | d95e67fd6b6a1a0392083b306764c0f18c65e415c26d75c61173628f01409115 |
