2025 Python Packaging Survey is now live!  Take the survey now

Apache Airflow Providers containing 3rd party integrations supported natively in Airflow

Navigation

Verified details

These details have been verified by PyPI
Owner

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache License 2.0)
  • Author: Astronomer
  • Requires: Python >=3.7
Classifiers
Report project as malware

Reason this release was yanked:

Pre-alpha test release

Project description

A framework for generating Apache Airflow DAGs from other workflows.

Installation

Install and update using pip:

pip install astronomer-cosmos

This only installs dependencies for core provider. To install all dependencies, run:

pip install 'astronomer-cosmos[all]'

To only install the dependencies for a specific integration, specify the integration name as extra argument, example to install dbt integration dependencies, run:

pip install 'astronomer-cosmos[dbt]'

Extras

Extra Name

Installation Command

Dependencies

all

pip install 'astronomer-cosmos[all]'

All

dbt

pip install 'astronomer-cosmos[dbt]'

dbt core

Example Usage

Imagine we have a dbt project located at ./dbt/my_project.

from cosmos.providers.dbt import DbtDag, DbtTaskGroup, DbtTask

# render as a DAG
dag = DbtDag(
    project_dir="./dbt/my_project",
    dag_id="my_dag",
    schedule_interval="@daily",
    default_args={"owner": "airflow"},
)

# render as a task group
with DAG("my_dag", default_args={"owner": "airflow"}) as dag:
    task_group = DbtTaskGroup(
        project_dir="./dbt/my_project",
        task_group_id="my_task_group",
    )

# render as an individual task
with DAG("my_dag", default_args={"owner": "airflow"}) as dag:
    task = DbtTask(
        project_dir="./dbt/my_project",
        task_id="my_task",
    )

Principles

Astronomer Cosmos provides a framework for generating Apache Airflow DAGs from other workflows. Every provider comes with two main components:

  • extractors: These are responsible for extracting the workflow from the provider and converting it into Task and Group objects.

  • operators: These are used when the workflow is converted into a DAG. They are responsible for executing the tasks in the workflow.

Astronomer Cosmos is not opinionated in the sense that it does not enforce any rendering method. Rather, it comes with the tools to render workflows as Airflow DAGs, task groups, or individual tasks.

Changelog

We follow Semantic Versioning for releases. Check CHANGELOG.rst for the latest changes.

Contributing Guide

All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.

A detailed overview an how to contribute can be found in the Contributing Guide.

As contributors and maintainers to this project, you are expected to abide by the Contributor Code of Conduct.

Goals for the project

  • Goal 1

  • Goal 2

  • Goal 3

Limitations

  • List any limitations

License

Apache License 2.0

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page