In fact, “Scheduler Performance” was the most asked for improvement in the Community Survey. As Airflow matured and the number of users running hundreds of thousands of tasks grew, however, we at Astronomer came to see great opportunity in driving a dedicated effort to improve on Scheduler functionality and push Airflow to a new level of scalability. The Airflow Scheduler as a core component has been key to the growth and success of the project since its creation in 2014. A New Scheduler: Low-Latency + High-Availability Many of the significant improvements were influenced and inspired by feedback from Airflow’s 2019 Community Survey, which garnered over 300 responses. Major Features in Airflow 2.0Īirflow 2.0 includes hundreds of features and bug fixes both large and small. In celebration of the highly anticipated release, we’ve put together an overview of major Airflow 2.0 features below. Throughout 2020, various organizations and leaders within the Airflow Community collaborated closely to refine the scope of Airflow 2.0, focusing on enhancing existing functionality and introducing changes to make Airflow faster, more reliable, and more performant at scale. As committed members of the community, we at Astronomer were delighted to announce the release of Airflow 2.0 by the end of 2020. In 2022, Airflow reached 15K+ commits and 25K+ GitHub stars.Īs Apache Airflow grew in adoption, a major release to expand on the project’s core strengths came to be long overdue. Airflow boasts tens of thousands of users and more than 2,000 contributors who regularly submit features, plugins, content, and bug fixes to ensure continuous momentum and improvement. Its ability to meet the needs of simple and complex use cases alike make it both easy to adopt and scale. Today it supports more than 70 providers, including AWS, GCP, Microsoft Azure, Salesforce, Slack, and Snowflake. Airflow was designed to make data integration between systems easy. Airflow competitively delivers in scheduling, scalable task execution, and UI-based task management and monitoring. Proven core functionality for data pipelining.It was brought into the Apache Software Foundation’s Incubator Program in March 2016, and saw growing success in the wake of Maxime’s well-known blog post on “The Rise of the Data Engineer.” By January of 2019, Airflow was announced as a Top-Level Apache Project by the Foundation, and it is now widely recognized as the industry’s leading workflow orchestration solution.Īirflow’s strength as a tool for dataflow automation has grown for a few reasons: If you’d like to learn more about the latest features, head over to the articles about Airflow 2.2 and Airflow 2.3.Īpache Airflow was created by Airbnb’s Maxime Beauchemin as an open-source project in late 2014. Note: This article focuses mainly on Airflow 2.0. If your team is running Airflow 1 and would like help establishing a migration path, reach out to us. We strongly encourage your team to upgrade to Airflow 2.x. Note: With the release of Airflow 2.0 in late 2020, and with subsequent releases, the open-source project addressed a significant number of pain points commonly reported by users running previous versions.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |