The Prefect Blog

Command Line Interfaces and YAML are loved by DevOps engineers — we use the same for DataflowOps

Introducing Blocks: open source, modular, reusable, and sharable configuration plus code

It just got easier to run serverless Python flows on Azure — without thinking about servers, clusters, or pods.

Part 1: Up and running

Coordinate dataflow across multiple domains and teams, and orchestrate your data platform with a parent flow in Prefect

Coordinate ML, analytics & BI reporting data platform workflows with Prefect, dbt, Snowflake, and modular building blocks

Spend more time building data pipelines and less time maintaining infrastructure!

Failure handling with alert notifications and retries for data ingestion and dbt transformations in your Modern Data Stack

Scheduling, data ingestion, and backfilling implemented with modular building blocks and simple deployment patterns

Data ingestion & transformation flows implemented with modular building blocks and simple deployment patterns

DAGs don’t describe what you do. Your teams, projects & systems do. Use blocks to draw a map of your stack and orchestrate it with Prefect.

And many more enhancements to the UI, CLI, and docs

This is a conceptual post about building a data platform with Prefect, dbt, and Snowflake — hands-on demos follow later

New serverless infrastructure capabilities, batch deletion from the UI, and utilities to orchestrate flow runs from deployments

Scheduling, with or without cron, is only a small part of the data coordination puzzle

The easiest way to get started with scheduled serverless workflows built in Python

…and many enhancements based on user feedback!

And how to define state and data dependencies in your Prefect data pipelines

Prefect is like air traffic control for your data. Docker Compose is like air traffic control for you containers. They make a great team!

CI/CD, Scheduling, Parametrization and Serverless AWS ECS Deployments are now even easier

When and when not to schedule your workflows and why sensors and daemon processes are a terrible waste of resources

How to quickly make a Prefect Python deployment file

And several important fixes and enhancements!

Forget about sensors and DAGs. Run your flows based on events.

From Task Library to Databricks Collection

Introducing the prefect-airbyte collection

A guide to fully automated serverless real-time data pipelines