dask-deltalake 0.0.1

Dask Deltalake

使用 delta-rs 从 Dask 读取和写入 deltalake

Dask Deltalake Reader

使用 Dask 从 Deltalake 读取数据

尝试此软件包

pip install dask_deltalake

功能

1. 使用 dask 引擎并行读取基于 delta 日志的 parquet 文件
2. 支持所有三种文件系统，如 s3、azurefs、gcsfs
3. 支持一些 delta 功能，如
   • 时间旅行
   • 架构演变
   • parquet 过滤器
     • 行过滤器
     • 分区过滤器
4. 查询 Delta 提交信息 - 历史
5. 清理旧的/未使用的 parquet 文件
6. 使用日期时间加载数据的不同版本

使用方法

import dask_deltalake as ddl

# read delta table
ddl.read_delta("delta_path")

# read delta table for specific version
ddl.read_delta("delta_path",version=3)

# read delta table for specific datetime
ddl.read_delta("delta_path",datetime="2018-12-19T16:39:57-08:00")


# read delta complete history
ddl.read_delta_history("delta_path")

# read delta history upto given limit
ddl.read_delta_history("delta_path",limit=5)

# read delta history to delete the files
ddl.vacuum("delta_path",dry_run=False)

# Can read from S3,azure,gcfs etc.
ddl.read_delta("s3://bucket_name/delta_path",version=3)
# please ensure the credentials are properly configured as environment variable or
# configured as in ~/.aws/credential

# can connect with AWS Glue catalog and read the complete delta table (currently only AWS catalog available)
# will take expilicit AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from environment
# variables if available otherwise fallback to ~/.aws/credential
ddl.read_delta(catalog=glue,database_name="science",table_name="physics")