Ceph使用boto3进行选择对象操作的外壳，可确认读取的完整性并修复间歇性连接。
项目描述
以下boto3 s3客户端操作的外壳，具有连接重试和校验和验证
get_object
head_object
upload_fileobj
copy_object
delete_object
list_objects_v2
特性
重试连接失败
确认上传和检索对象的校验和
移动对象
受限制的接口，支持现有桶中对象的简单CRUD操作
配置
以下环境变量配置object clerk。
变量 |
描述 |
类型 |
默认值 |
|---|---|---|---|
MULTIPART_THRESHOLD |
在哪个字节阈值上传被拆分为多个部分进行上传。影响存储在eTag中的校验和 |
STR |
524288000 |
S3_CLIENT_CONFIG |
Boto Core客户端配置 https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html 默认值反映了在更高层面捕获失败操作的重试。 |
JSON |
`json {"connect_timeout": 60, "read_timeout": 60, "retries": {"max_attempts": 0}} ` |
S3_UPLOAD_CONFIG |
上传时应用的传输配置 https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig multipart_threshold和multipart_chunksize由MULTIPART_THRESHOLD环境变量设置。S3_UPLOAD_CONFIG中的这些键的值将被丢弃。 |
JSON |
`json {} ` |
S3_DOWNLOAD_CONFIG |
JSON |
`json {} ` |
安装
pip install object-clerk示例
初始化
clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'获取对象
# with checksum verified
bytes_response = clerk.get_object("bucket", "object_key")
# without checksum verified
bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)删除对象
clerk.delete_object("bucket", "object_key")获取对象信息
dict_response = clerk.get_object_info("bucket", "object_key")复制对象
# with checksum verified
clerk.copy_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
)
# without checksum verified
clerk.copy_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
verify_checksum=False
)上传对象
# with checksum verified
with open("file", mode='rb') as f:
clerk.upload_object(f, "bucket", "object_key")
# without checksum verified
with open("file", mode='rb') as f:
clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)移动对象
# with checksum verified
clerk.move_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
)
# without checksum verified
clerk.move_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
verify_checksum=False
)列表对象
clerk.list_objects(
"bucket_name",
1000
)测试
git clone git@bitbucket.org:swiant/object_store_wrapper.git
pip install -e .
export HOST=<host>
export PORT=<port>
export ACCESS_KEY=<access_key>
export SECRET_KEY=<secret_key>
pytest -v object_clerk
object-clerk-0.1.1.tar.gz的哈希值
| 算法 | 哈希摘要 | |
|---|---|---|
| SHA256 | c2e4a54fc9d9c1f0656d67e2a95f4c9285a380eba4ac6a9d4d63f2a5afccf6c0 |
|
| MD5 | 800de8aac05faf474fa065a8c0d71063 |
|
| BLAKE2b-256 | 5b554ac73522dd8647619be2c790eabb2bb91c84d5cbca621ebc14c0d1669f7f |