shardedstore

提供分片Zarr存储。

功能

• 对于大型Zarr存储，避免过多对象或极其大的对象，从而绕过文件系统inode使用和对象存储限制。
• 性能敏感的实现。
• 使用现有的Zarr v2存储。
• 混合和匹配分片存储类型。
• 使用JSON序列化和反序列化ShardedStore。
• 分片组或数组块。
• 轻松对存储分片执行转换。

安装

pip install shardedstore

示例

from shardedstore import ShardedStore, array_shard_directory_store, to_zip_store_with_prefix

from zarr.storage import DirectoryStore

# xarray example, but works with zarr in general
import xarray as xr
from datatree import DataTree, open_datatree
import json
import numpy as np
import os

创建组件分片存储

base_store = DirectoryStore("base.zarr")
shard1 = DirectoryStore("shard1.zarr")
shard2 = DirectoryStore("shard2.zarr")
array_shards1 = array_shard_directory_store("array_shards1")
array_shards2 = array_shard_directory_store("array_shards2")

为示例生成数据

# xarray-datatree Quick Overview
data = xr.DataArray(np.random.randn(2, 3), dims=("x", "y"), coords={"x": [10, 20]})
# Sharded array dimensions must have a chunk shape of 1.
data = data.chunk([1,2])
ds = xr.Dataset(dict(foo=data, bar=("x", [1, 2]), baz=np.pi))
ds2 = ds.interp(coords={"x": [10, 12, 14, 16, 18, 20]})
ds2 = ds2.chunk({'x':1, 'y':2})
ds3 = xr.Dataset(
    dict(people=["alice", "bob"], heights=("people", [1.57, 1.82])),
    coords={"species": "human"},
    )
dt = DataTree.from_dict({"simulation/coarse": ds, "simulation/fine": ds2, "/": ds3})

单体存储

single_store = DirectoryStore("single.zarr")
dt.to_zarr(single_store)

展示在组和数组上分片的分片存储。

数组在1维上分片。

sharded_store = ShardedStore(base_store,
    {'people': shard1, 'species': shard2},
    {'simulation/coarse/foo': (1, array_shards1), 'simulation/fine/foo': (1, array_shards2)})
dt.to_zarr(sharded_store)

序列化/反序列化

config = sharded_store.get_config()
config_str = json.dumps(config)
config = json.loads(config_str)
sharded_store = ShardedStore.from_config(config)

验证

from_single = open_datatree(single_store, engine='zarr').compute()
from_sharded = open_datatree(sharded_store, engine='zarr').compute()
assert from_single.identical(from_sharded)

使用map_shards在组件分片上运行转换

to_zip_stores = to_zip_store_with_prefix("zip_stores")
zip_sharded_stores = sharded_store.map_shards(to_zip_stores)

开发

欢迎并感谢贡献。

git clone https://github.com/thewtex/shardedstore
cd shardedstore
pip install -e ".[test]"
pytest