fireflow

一个用于与FirecREST交互的简单异步工作流引擎。

工作流引擎使用pyfirecrest来与REST API交互。

还可以查看https://github.com/eth-cscs/firecrest的演示服务器。

用法

安装软件包后，将提供fireflow命令。

然后您可以从YAML文件创建工作流数据库，例如example_setup.yml

clients:
  - label: test-client
    client_url: "https://:8000/"
    client_id: "firecrest-sample"
    client_secret: "b391e177-fa50-4987-beaf-e6d33ca93571"
    token_uri: "https://:8080/auth/realms/kcrealm/protocol/openid-connect/token"
    machine_name: "cluster"
    work_dir: "/home/service-account-firecrest-sample"
    small_file_size_mb: 5
codes:
  - label: test-code1
    client_label: test-client
    script: |
      #!/bin/bash
      #SBATCH --job-name={{ calc.uuid }}
      mkdir -p output
      echo '{{ calc.parameters.echo_string }}' > output.txt
calcjobs:
  - label: test-calcjob1
    code_label: test-code1
    parameters:
      echo_string: "Hello world!"
    download_globs:
    - output.txt
  - label: test-calcjob2
    code_label: test-code1
    parameters:
      echo_string: "Hello world 2!"
    download_globs:
    - output.txt

然后运行fireflow init -a example_setup.yaml，这将创建一个位于.fireflow_project文件夹中的数据库。

您可以使用以下命令列出所有calcjobs

$ fireflow calcjob tree
Calcjobs 1-2 of 2
└── 1 - test-client
    └── 1 - test-code1
        ├── 1 - test-calcjob1 ▶
        └── 2 - test-calcjob2 ▶

然后您可以运行所有calcjobs

$ fireflow run
2023-02-07 20:10:58:fireflow.process:REPORT: PK-1: Uploading files to remote
2023-02-07 20:11:00:fireflow.process:REPORT: PK-2: Uploading files to remote
2023-02-07 20:11:01:fireflow.process:REPORT: PK-1: submitting on remote
2023-02-07 20:11:03:fireflow.process:REPORT: PK-2: submitting on remote
2023-02-07 20:11:05:fireflow.process:REPORT: PK-1: polling job until finished
2023-02-07 20:11:06:fireflow.process:REPORT: PK-2: polling job until finished
2023-02-07 20:11:08:fireflow.process:REPORT: PK-1: copying from remote folder
2023-02-07 20:11:09:fireflow.process:REPORT: PK-2: copying from remote folder
2023-02-07 20:11:13:fireflow.process:REPORT: PK-1: parsing output files
2023-02-07 20:11:13:fireflow.process:REPORT: PK-1: paths: ['job.sh', 'output.txt', 'slurm-147.out']
2023-02-07 20:11:13:fireflow.process:REPORT: PK-2: parsing output files
2023-02-07 20:11:13:fireflow.process:REPORT: PK-2: paths: ['slurm-148.out', 'job.sh', 'output.txt']

calcjobs以异步方式运行，步骤如下

• 准备提交：创建作业脚本
• 复制到远程文件夹：将作业脚本和输入文件复制到远程文件夹
• 在远程提交：将作业脚本提交到调度器
• 轮询作业直到完成：轮询作业状态直到完成
• 从远程文件夹复制：从远程文件夹复制输出文件
• 解析输出文件：解析输出文件并将结果存储在数据库中

目标

• 最小化对服务器的请求数量

上游 o AiiDA

• 打印作业脚本，以进行调试
• 添加对sqlalchemy.engine日志的自定义https://docs.sqlalchemy.org.cn/en/20/core/engines.html#configuring-logging

https://chat.openai.com/chat/e453e10b-19fd-46a8-9cfb-0c8d31d1a60d