big-map-archive-api-client

这是一个用于与BIG-MAP存档数据仓库交互的命令行客户端。

目录

• 快速开始
  • 安装
  • 配置文件
  • 元数据文件
  • 数据文件
• 使用方法
  • 概述
  • 获取记录
  • 创建记录
  • 更新记录
  • 备份FINALES数据库
• FINALES数据库的备份策略

快速开始

安装

big-map-archive-api-client 是一个托管在 PyPI 上的Python包。

我们建议您按照以下步骤在Linux机器上安装Python包及其依赖项

1. 创建虚拟环境

python -m venv /home/<username>/.virtualenvs/<virtual_env_name>

2. 激活虚拟环境

source /home/<username>/.virtualenvs/<virtual_env_name>/bin/activate

3. 如果需要，安装 wheel 包（见 此处）

pip install wheel

4. 在虚拟环境中安装 big-map-archive-api-client 包及其依赖项。为最新的BIG-MAP存档安装版本1.2.0

pip install big-map-archive-api-client==1.2.0

5. 【可选】安装完成后，请检查与 bma 相关的可执行文件是否确实位于虚拟环境中

which bma

6. 【可选】创建一个项目目录（用于存储配置、输入和输出文件）。

配置文件

每个命令都需要一个YAML配置文件，该文件指定目标数据存储库的域名和API令牌。这通过命令选项 --config-file 和 --bma-config-file 来表示。我们还建议将该文件放在您的项目目录中，并将其命名为 bma_config.yaml。其内容应类似于

domain_name: "big-map-archive-demo.materialscloud.org" # Options: archive.big-map.eu, big-map-archive-demo.materialscloud.org, big-map-archive-demo-public.materialscloud.org
token: "123456789"

请注意，为了获取目标 BIG-MAP 存档的API令牌，您需要在数据存储库中拥有账户。要请求账户，请发送电子邮件至 big-map-archive@materialscloud.org。登录后，导航至 https://<archive_domain_name>/account/settings/applications 并创建令牌。

命令 bma finales-db back-up 需要另一个YAML配置文件，该文件指定目标 FINALES 服务器的主机地址和端口以及服务器上用户账户的凭证。这对应于命令选项 --finales-config-file。我们建议将该文件放在您的项目目录中，并将其命名为 finales_config.yaml。其内容应类似于

ip_address: "0.0.0.0" # Replace by IP address for a FINALES server
port: "1234" # Replace by port for a FINALES server
username: "test" # Replace by valid username
password: "test" # Replace by valid password

元数据文件

一些命令（例如，bma record create）使用命令选项 --metadata-file 指定的YAML文件来创建/更新记录的元数据。我们建议将该文件放在您的项目目录中，并将其命名为 metadata.yaml。其内容应类似于

resource_type: "Dataset" # Choose one of these options: Dataset, Software, Other
title: "Mobilities in Two-Dimensional Materials" # Record's title
authors: # Record's list of authors with their affiliations
  - family_name: "Doe"
    given_name: "Jane"
    affiliations:
      - "Physics Institute, École Polytechnique Fédérale de Lausanne, CH-1015 Lausanne, Switzerland"
  - family_name: "Dell"
    given_name: "John"
    affiliations:
      - "Department of Physics, Technical University of Denmark (DTU), 2800 Kongens Lyngby, Denmark"
      - "Physics Institute, École Polytechnique Fédérale de Lausanne, CH-1015 Lausanne, Switzerland"
description: "Two-dimensional materials are emerging as a promising platform for ultrathin channels..." # Abstract-like description of the record
license: "BIG-MAP Archive License" # Choose one of these options: BIG-MAP Archive License, Creative Commons Attribution Share Alike 4.0 International, MIT License
keywords: # Any keyword is accepted
  - "2D materials"
  - "transport"
references: # Choose among these options for the reference scheme: arxiv, doi, isbn, url
  - scheme: "arxiv"
    identifier: "2308.08462"
  - scheme: "doi"
    identifier: "10.1093/ajae/aaq063"
  - scheme: "isbn"
    identifier: "978-3-16-148410-0"
  - scheme: "url"
    identifier: "https://arxiv.org/abs/2308.08462"

数据文件

两个命令提供了将您选择的数据文件附加到记录的可能性：bma record create 和 bma record update。

请遵守以下规则

• 每个记录最多100个文件，
• 每个记录的总文件大小小于100 GB。

命令选项 --data-files 应指向要上传并附加到新记录的文件的目录。我们通常将此类文件夹放在我们的项目目录中，并命名为 upload。

社区

要将记录发布到社区，您需要指定社区 slug。登录至 https://<archive_domain_name>。您可以访问的社区列表位于链接 https://<archive_domain_name>/api/communities。查找 slug 的值，您在发布记录到社区时需要添加此值。例如：对于 BIG-MAP 社区，slug 是 bigmap，对于 BATTERY2030，slug 是 battery2030。

使用方法

在针对 演示实例 执行命令之前，您可能想先对其进行测试，然后再针对 主数据存储库 执行。

概述

bma --help

Usage: bma [OPTIONS] COMMAND [ARGS]...

  Command line client to interact with a BIG-MAP Archive.

Options:
  --help  Show this message and exit.

Commands:
  finales-db  Copy data from the database of a FINALES server to a...
  record      Manage records on a BIG-MAP Archive.

bma record --help

Usage: bma record [OPTIONS] COMMAND [ARGS]...

  Manage records on a BIG-MAP Archive.

Options:
  --help  Show this message and exit.

Commands:
  create   Create a record on a BIG-MAP Archive and optionally publish it.
  get      Get the metadata of a published version of an entry on a...
  get-all  Get the metadata of the latest published version for each...
  update   Update a published version of an archive entry, or create a...

bma finales-db --help

Usage: bma finales-db [OPTIONS] COMMAND [ARGS]...

  Copy data from the database of a FINALES server to a BIG-MAP Archive.

Options:
  --help  Show this message and exit.

Commands:
  back-up  Perform a partial back-up from the database of a FINALES...

获取记录

bma record get --help

Usage: bma record get [OPTIONS]

  Get the metadata of a published version of an entry on a BIG-MAP Archive and
  save it to a file.

Options:
  --config-file FILE  Path to the YAML file that specifies the domain name and
                      a personal access token for the targeted BIG-MAP
                      Archive. See bma_config.yaml in the GitHub repository.
                      [required]
  --record-id TEXT    Id of the published version of an archive entry (e.g.,
                      "pxrf9-zfh45").  [required]
  --output-file FILE  Path to the JSON file where the obtained record's
                      metadata will be exported to.  [required]
  --help              Show this message and exit.

bma record get-all --help

Usage: bma record get-all [OPTIONS]

  Get the metadata of the latest published version for each entry on a BIG-MAP
  Archive and save them to a file.

Options:
  --config-file FILE  Path to the YAML file that specifies the domain name and
                      a personal access token for the targeted BIG-MAP
                      Archive. See bma_config.yaml in the GitHub repository.
                      [required]
  --all-versions      Get all published versions for each entry. By default,
                      only the latest published version for each entry is
                      retrieved.
  --output-file FILE  Path to the JSON file where the obtained record's
                      metadata will be exported to.  [required]
  --help              Show this message and exit.

创建记录

bma record create --help

Usage: bma record create [OPTIONS]

  Create a record on a BIG-MAP Archive and optionally publish it.

Options:
  --config-file FILE      Path to the YAML file that specifies the domain name
                          and a personal access token for the targeted BIG-MAP
                          Archive. See bma_config.yaml in the GitHub
                          repository.  [required]
  --metadata-file FILE    Path to the YAML file for the record's metadata
                          (title, list of authors, etc). See
                          data/input/example/create_record/metadata.yaml in
                          the GitHub repository.  [required]
  --data-files DIRECTORY  Path to the directory that contains the data files
                          to be uploaded and linked to the record. See
                          data/input/example/create_record/upload in the
                          GitHub repository.  [required]
  --slug TEXT             Community slug of the record. Example: for the BIG-
                          MAP community the slug is bigmap.  [required]
  --publish               Publish the created record.
  --help                  Show this message and exit.

更新记录

bma record update --help

Usage: bma record update [OPTIONS]

  Update a published version of an archive entry, or create a new version and
  optionally publish it. When updating a published version, only the metadata
  (title, list of authors, etc) can be modified.

Options:
  --config-file FILE              Path to the YAML file that specifies the
                                  domain name and a personal access token for
                                  the targeted BIG-MAP Archive. See
                                  bma_config.yaml in the GitHub repository.
                                  [required]
  --record-id TEXT                Id of the published version (e.g.,
                                  "pxrf9-zfh45").  [required]
  --update-only                   Update the metadata of the published
                                  version, without creating a new version. By
                                  default, a new version is created.
  --metadata-file FILE            Path to the YAML file that contains the
                                  metadata (title, list of authors, etc) for
                                  updating the published version or creating a
                                  new version. See data/input/example/update_r
                                  ecord/metadata.yaml in the GitHub
                                  repository.  [required]
  --data-files DIRECTORY          Path to the directory that contains the data
                                  files to be linked to the newly created
                                  version. See
                                  data/input/example/update_record/upload in
                                  the GitHub repository.  [required]
  --link-all-files-from-previous  Link all files that are already linked to
                                  the previous version to the new version,
                                  with the exception of files whose content
                                  changed.
  --publish                       Publish the newly created version.
  --help                          Show this message and exit.

备份FINALES数据库

bma finales-db back-up --help

Usage: bma finales-db back-up [OPTIONS]

  Back up the SQLite database of a FINALES server to a BIG-MAP Archive. This creates and publishes a new entry version, which provides links to data extracted from the database (capabilities, requests, and results for requests) and a copy of the whole database.
Options:
  --bma-config-file FILE          Path to the YAML file that specifies the
                                  domain name and a personal access token for
                                  the targeted BIG-MAP Archive. See
                                  bma_config.yaml in the GitHub repository.
                                  [required]
  --finales-config-file FILE      Path to the YAML file that specifies the IP
                                  address, the port, and the credentials of a
                                  user account for the targeted FINALES
                                  server. See finales_config.yaml in the
                                  GitHub repository.  [required]
  --record-id TEXT                Id of the published version for the previous
                                  back-up (e.g., "pxrf9-zfh45"). For the first
                                  back-up, leave to the default.
  --metadata-file FILE            Path to the YAML file that contains the
                                  metadata (title, list of authors, etc) for
                                  creating a new version. See data/input/examp
                                  le/create_record/metadata.yaml in the GitHub
                                  repository.  [required]
  --link-all-files-from-previous  Link all files that are already linked to
                                  the previous version to the new version,
                                  with the exception of files whose content
                                  changed.
  --no-publish                    Do not publish the newly created version.
                                  This is discouraged in production. If you
                                  select this option, either publish or delete
                                  the newly created draft, e.g., via the GUI.
  --slug TEXT                     Community slug of the record. Example: for
                                  the BIG-MAP community the slug is bigmap.
                                  [required]
  --help                          Show this message and exit.

在执行命令时，如果用户

• 尝试创建条目（即未提供记录ID），但他/她已经拥有具有相同标题的已发布记录，这将要求用户进行确认。这是为了防止用户无意中创建新的条目。
• 尝试更新现有条目（提供记录ID），但新版本将具有不同的标题。这是为了执行我们的“每个“活动”一个标题”政策（请参阅 FINALES数据库备份策略）

在生产数据库备份时，将相应的 metadata.yaml 文件放在 big-map-archive-api-client-finales GitHub 仓库中，并进行版本控制。

FINALES数据库的备份策略

以下备份策略适用于生产中的 FINALES 服务器数据库

• 在 FINALES 服务器上的每个“活动”中，主 BIG-MAP 存档应只有一个条目。
• 条目可能具有多个版本，每次进行数据库备份时都会创建和发布一个版本。请注意，如果数据文件从一个版本到下一个版本保持不变，则文件仅上传一次。然而，相应的文件链接出现在两个条目版本中。这节省了存储空间并减少了备份时间。
• 每个条目的版本都有一个标题。它可以更改，但由于它作为活动的标识符，因此最好在同一个条目的所有版本中保持不变。为了强制执行“每个“活动”一个标题”的政策，当用户在创建新版本时尝试更改标题时，命令bma finales-db back-up会请求确认。
• 单个服务账户用于备份特定的“活动”。
• 同一服务账户可用于多个“活动”。

支持

如果您有任何评论或问题，请通过电子邮件发送给我们：big-map-archive@materialscloud.org。

问题

如果您发现了一个错误，请直接在GitHub中创建一个问题。如果可能，提供足够的详细信息，以便BIG-MAP存档团队能够重现遇到的问题。谢谢！

致谢

该项目已获得欧盟地平线2020研究与创新能力提升计划的资助，合同编号No 957189。该项目是BATTERY 2030+的一部分，这是一个旨在发明未来可持续电池的大规模欧洲研究倡议。