prefect

https://github.com/PrefectHQ/prefect

https://docs.prefect.io/

Prefect

We‘ve rebuilt data engineering for the data science era.

Prefect is a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize Tasks into Flows, and Prefect takes care of the rest.

Demo code

Prefect‘s Pythonic API should feel familiar for newcomers. Mark functions as tasks and call them on each other to build up a flow.

from prefect import task, Flow, Parameter


@task(log_stdout=True)
def say_hello(name):
    print("Hello, {}!".format(name))


with Flow("My First Flow") as flow:
    name = Parameter(‘name‘)
    say_hello(name)


flow.run(name=‘world‘) # "Hello, world!"
flow.run(name=‘Marvin‘) # "Hello, Marvin!"

安装

https://docs.prefect.io/core/getting_started/installation.html

https://docs.docker.com/compose/install/

发布

https://docs.prefect.io/orchestration/concepts/flows.html#registration

https://docs.prefect.io/api/latest/cli/create.html

prefect create project  "my project"

 prefect register --project "my project" --path test.py

prefect - The easiest way to automate your data