欢迎使用arrowdantic

Arrowdantic是一个小的Python库，它背后有一个成熟的Rust实现，Apache Arrow，它可以与

• Parquet
• Apache Arrow 和
• ODBC（数据库）交互。

对于简单（但数据密集型）的数据工程任务，这个包实际上取代了pyarrow：它支持从Parquet和Arrow读取和写入，性能更高，安全性更高（例如，没有段错误）。

此外，它支持以与turbodbc相同的性能或更高的性能从ODBC兼容的数据库（例如，postgres，mongoDB）读取和写入。

这个包特别适合AWS Lambda等环境 - 它仅占用8M的磁盘空间，而pyarrow则占用82M。

功能

• 声明和访问由Arrow支持的数组（整数、浮点数、布尔值、字符串、二进制）
• 从Apache Arrow IPC文件读取和写入
• 从Apache Parquet读取和写入
• 从ODBC兼容的数据库（例如postgres，mongoDB）读取和写入

示例

使用Parquet

import io
import arrowdantic as ad

original_arrays = [ad.UInt32Array([1, None])]

schema = ad.Schema(
    [ad.Field(f"c{i}", array.type, True) for i, array in enumerate(original_arrays)]
)

data = io.BytesIO()
with ad.ParquetFileWriter(data, schema) as writer:
    writer.write(ad.Chunk(original_arrays))
data.seek(0)

reader = ad.ParquetFileReader(data)
chunk = next(reader)
assert chunk.arrays() == original_arrays

使用Arrow文件

import arrowdantic as ad

original_arrays = [ad.UInt32Array([1, None])]

schema = ad.Schema(
    [ad.Field(f"c{i}", array.type, True) for i, array in enumerate(original_arrays)]
)

import io

data = io.BytesIO()
with ad.ArrowFileWriter(data, schema) as writer:
    writer.write(ad.Chunk(original_arrays))
data.seek(0)

reader = ad.ArrowFileReader(data)
chunk = next(reader)
assert chunk.arrays() == original_arrays

使用ODBC

import arrowdantic as ad


arrays = [ad.Int32Array([1, None]), ad.StringArray(["aa", None])]

with ad.ODBCConnector(r"Driver={SQLite3};Database=sqlite-test.db") as con:
    # create an empty table with a schema
    con.execute("DROP TABLE IF EXISTS example;")
    con.execute("CREATE TABLE example (c1 INT, c2 TEXT);")

    # insert the arrays
    con.write("INSERT INTO example (c1, c2) VALUES (?, ?)", ad.Chunk(arrays))

    # read the arrays
    with con.execute("SELECT c1, c2 FROM example", 1024) as chunks:
        assert chunks.fields() == [
            ad.Field("c1", ad.DataType.int32(), True),
            ad.Field("c2", ad.DataType.string(), True),
        ]
        chunk = next(chunks)
assert chunk.arrays() == arrays

使用时区

此包完全支持datetime及其与arrow之间的转换

import arrowdantic as ad


dt = datetime.datetime(
    year=2021,
    month=1,
    day=1,
    hour=1,
    minute=1,
    second=1,
    microsecond=1,
    tzinfo=datetime.timezone.utc,
)
a = ad.TimestampArray([dt, None])
assert (
    str(a)
    == 'Timestamp(Microsecond, Some("+00:00"))[2021-01-01 01:01:01.000001 +00:00, None]'
)
assert list(a) == [dt, None]
assert a.type == ad.DataType.timestamp(datetime.timezone.utc)