DVCx

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for dmpetrov from gravatar.com dmpetrov Avatar for dtulga from gravatar.com dtulga Avatar for shcheklein from gravatar.com shcheklein

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache-2.0
  • Author: Dmitry Petrov
  • Requires: Python >=3.9
Classifiers
Report project as malware

Project description

PyPI Status Python Version License

Tests Codecov pre-commit Black

What is DVCx?

DVCx is a Python data manipulation library designed to work with unstructured AI datasets. It provides a dataframe-like interface which can automatically reference data stored as files (text, images, video) locally or in the cloud.

Why use DVCx?

  1. Storage as a single source of truth. DVCx can organize unstructured data from storages and datalakes (local files, S3, GCS, Azure ADLS) into overlapping datasets without unnecessary file copies.

  2. Compute. DVCx supports local parallelization and external compute workers for efficient data processing and AI metadata creation.

  3. Large scale. In contrast to in-memory frameworks (like Pandas data frame), DVCx can work with datasets of millions and billions of records by using out-of-memory algorithms.

  4. Persistence and versioning. Your datasets, your computed metadata, and paid API call results remain versioned and reusable.

Installation

You can install DVCx via pip from PyPI:

$ pip install dvcx

Usage

DVCx can be used as a CLI (from system terminal), or as a Python library.

TODO: CLI usage

To use it from Python code, import class dvcx.catalog.Catalog, which provides methods for all the same commands above, like ls(), get(), find(), du() and index().

from dvcx.catalog import Catalog
catalog = Catalog()
catalog.ls(["gcs://dvcx-datalakes/dogs-and-cats/"], update=True)

Contributing

Contributions are very welcome. To learn more, see the Contributor Guide.

License

Distributed under the terms of the Apache 2.0 license, DVCx is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page