Auror Core

为Azkaban创建简单的流程

安装

pip install auror_core

支持的作业类型

V1

• 命令
• 流程

V2

• 命令

用法

• 使用一个命令创建简单的Azkaban流程
• 使用一个命令创建简单的V2 Azkaban流程
• 创建具有依赖关系的流程
• 共享作业属性
• 具有额外定制和配置的作业
• 与Flow集成（仅V1）
• 使用Flow参数
• 使用流环境变量
• 使用流环境变量和参数
• 在一个地方合并多个变量
• 从YAML文件中加载作业（仅适用于V2）
• 将内存流导出到Python文件（仅适用于V2）

使用一个命令创建简单的Azkaban流程

只需导入作业类型和项目

from auror_core.v1.job import Job, Command
from auror_core import Project

com1 = Job()\
.as_type(Command)\
.with_name("commands job")\
.with_command("bash echo 1")\
.with_another_command("bash echo 2")

Project("folder_to_generate_files", com1).write()

使用一个命令创建简单的V2 Azkaban流程

V2流实现在v2子文件夹中，具有相同的作业类型

from auror_core.v2.job import Job, Command
from auror_core import Project

com1 = Job()\
.as_type(Command)\
.with_name("commands job")\
.with_command("bash echo 1")\
.with_another_command("bash echo 2")

Project("folder_to_generate_files", com1).is_v2().write()

创建具有依赖关系的流程

from auror_core.v2.job import Job, Command
from auror_core import Project

com1 = Job()\
.as_type(Command)\
.with_name("commands job")\
.with_command("bash echo 1")\
.with_another_command("bash echo 2")

com2 = Command()\
.with_name("sub command job")\
.with_command("bash echo 1")\
.with_dependencies(com1)

Project("folder_to_generate_files", com1, com2).is_v2().write()

共享作业属性

组织配置相同的作业

from auror_core.v2.job import Command
from auror_core import Project

com = Command()\
.with_command("bash echo 1")

com1 = com.with_name("commands job")\
.with_another_command("bash echo 2")

com2 = com.with_name("sub command job")\
.with_dependencies(com1)

Project("folder_to_generate_files", com1, com2).is_v2().write()

具有额外定制和配置的作业

使用基本作业模拟命令（不推荐）

from auror_core.v1.job import Job
from auror_core import Project

com1 = Job()\
.with_name("commands job")\
.with_(command="bash echo 1")

com1._type = "command"

Project("folder_to_generate_files", com1).write()

与Flow集成（仅V1）

V2已包含流

from auror_core.v1.job import Command, Flow, Job
from auror_core import Project

com1 = Command()\
.with_name("commands job")\
.with_command("bash echo 1")

flow = Job()\
.as_type(Flow)\
.with_name("flow")\
.with_dependencies(com1)

Project("folder_to_generate_files", com1, flow).write()

使用Flow参数

from auror_core.v2.job import Command
from auror_core.v2.params import Params
from auror_core import Project

params = Params(
    teste1="my test",
    variable="my variable"
)

com = Command()\
.with_command("bash echo ${variable}")

Project("folder_to_generate_files", com)\
.is_v2()\
.with_params(params)\
.write()

使用流环境变量

from auror_core.v2.job import Command
from auror_core.v2.params import Env
from auror_core import Project

env = Env(
    TESTE="my test",
    VARIABLE="my variable"
)

com = Command()\
.with_command("bash echo $VARIABLE")

Project("folder_to_generate_files", com)\
.is_v2()\
.with_params(env)\
.write()

使用流环境变量和参数

from auror_core.v2.job import Command
from auror_core.v2.params import Env, Params
from auror_core import Project

env = Env(
    TESTE="my test",
    VARIABLE="my variable"
)

params = Params(
    teste1="my test",
    variable="my variable"
)

com = Command()\
.with_command("bash echo $VARIABLE ${variable}")

Project("folder_to_generate_files", com)\
.is_v2()\
.with_params(params, env)\
.write()

在一个地方合并多个变量

from auror_core.v2.job import Command
from auror_core.v2.params import Env
from auror_core import Project

env = Env(
    TESTE="env test",
    VARIABLE="env variable"
)

params = Params(
    teste1="my test",
    variable="my variable"
)

one_param = ParamsJoin("params_strange_name", ",") ## param name and separator

com = Command()\
.with_command("bash echo ${params_strange_name}") 
## it will print: my test,my variable,env test,env variable
## THERE IS NO ORDER GUARANTEE, JUST Python 3.6 >

Project("folder_to_generate_files", com)\
.is_v2()\
.with_params(one_param(params, env))\
.write()

从YAML文件中加载作业（仅适用于V2）

您可以在Azkaban 流文档中找到一些YAML文件示例

from auror_core.v2.loader import Loader

loader = Loader('/path/to/file/flow.yaml')
jobs = loader.as_job_objects()

或者您可以将这些作业导出为Python文件

from auror_core.v2.loader import Loader

loader = Loader('/path/to/file/flow.yaml')
jobs = loader.as_python_file('/path/to/desired/directory') # will be dumped with 'flow.py' name

将内存流导出到Python文件（仅适用于V2）

from auror_core.v2.dumper import Dumper

com1 = Job() \
.with_name("commands job 1") \
.with_(command="bash echo 1")

com2 = Job()\
.with_name("commands job 2")\
.with_(command="bash echo 2")

dumper = Dumper('/path/to/desired/directory') # will be dumped with 'flow.py' name
dumper.dump_jobs(com1, com2)

插件

插件只是从auror_core扩展而来的

还有一个用于新azkaban作业类型的cookiecutter，其中包含Auror模板： https://github.com/globocom/azkaban-jobtype-cookiecutter

我们已经有了一个电子邮件插件： https://github.com/globocom/azkaban-jobtype-email

贡献

有关开发和贡献，请遵循贡献指南并始终遵守行为准则