2025 Python Packaging Survey is now live!  Take the survey now

PFN IO library

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for belldandyxtq from gravatar.com belldandyxtq Avatar for ksugihara from gravatar.com ksugihara Avatar for kuenishi from gravatar.com kuenishi Avatar for y1r from gravatar.com y1r

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License
  • Author: Tianqi Xu, Kota Uenishi
  • Tags filesystem , hdfs , chainer , development
  • Requires: Python >=3.6
Classifiers
Report project as malware

Project description

PFIO

PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports

  • Filesystem API abstraction with unified error semantics,
  • Explicit user-land caching system,
  • IO performance tracing and metrics stats, and
  • Fileset container utilities to save metadata.

Dependency

  • HDFS client and libhdfs for HDFS access
  • CPython >= 3.6

Installation and Document build

Installation

$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .

Documentation

$ cd pfio/docs
$ make html
$ open build/html/index.html

Test

$ cd pfio
$ pip install .[test]
$ pytest tests/

How to use

Please refer to the official document for more information about the usage.

Release

Check the official document for latest release procedure.

Run tests locally:

$ pip install --user -e .[test]
$ pytest

Bump version numbers in pfio/version.py and in docs/source/conf.py .

Push and open a pull request to invoke CI. Once CI passed and the pull request merged, tag a release:

$ git tag -s X.Y.Z
$ git push --tags

Build:

$ rm -rf dist
$ pip3 install --user build
$ python3 -m build

Release to PyPI:

$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page