Prefect

Prefect is a workflow orchestration platform for Python data and ML engineers, with an Apache 2.0 open-source core and a hosted Prefect Cloud priced per seat.

Prefect turns ordinary Python functions into workflows that run on a schedule or on an event, retry when a step fails and report the state of every run. Prefect Technologies, Inc. develops it from its headquarters in Washington, D.C., and has no parent company. The framework is open source under the Apache 2.0 license. Prefect Cloud adds a hosted control plane for scheduling and visibility, while the code runs where the team chooses. A flow is a Python function marked with the @flow decorator, and its tasks declare their own retries and the delays between attempts. A deployment is the server-side definition of a flow. Cron, Interval or RRule schedules start it, and so do event triggers in Automations, which also accept webhooks from outside systems. Work pools decide where runs execute: Prefect Managed pools use Prefect Serverless compute, and hybrid pools hand runs to a worker in the team's own infrastructure. The .map method fans a task out across inputs found during the run, and the @materialize decorator records the tables and files it produces as assets with their upstream links. Logs from get_run_logger stay attached to each task run, and prefect server start brings up the same server and UI on a laptop. Prefect is built for data engineers and ML engineers who want orchestration written in Python and kept in the same repository as the pipeline. Platform teams required to run workloads on their own cloud accounts or Kubernetes clusters choose hybrid work pools. Smaller teams with no workers to operate run on Prefect Serverless instead. Developers building AI agents use the same flows and tasks for durable execution, and teams that want no hosted service can self-host the open-source server. A common pattern is a flow tested on a laptop, then deployed to Prefect Cloud by a GitHub Actions workflow on each push to main. Its runs then land on Kubernetes, AWS ECS, Google Cloud Run or Azure Container Instances, or on Modal and Coiled, depending on the work pool. A data team can run its dbt project as one step among the others in that flow. Prefect fits teams that want the schedule, the retries and the run history to live in the same Python code as the pipeline itself.