{"id":304,"date":"2021-08-16T00:00:00","date_gmt":"2021-08-16T00:00:00","guid":{"rendered":"https:\/\/tac.debuzzify.com\/?p=304"},"modified":"2023-06-23T00:15:05","modified_gmt":"2023-06-23T00:15:05","slug":"the-prefect-way-to-automate-orchestrate-data-pipelines","status":"publish","type":"post","link":"https:\/\/www.the-analytics.club\/the-prefect-way-to-automate-orchestrate-data-pipelines\/","title":{"rendered":"Python Workflow Automation With Prefect (A Better Airflow)"},"content":{"rendered":"\n\n\n<p>I was a big fan of Apache Airflow. Even today, I don&#8217;t have many complaints about it. But the new technology, Prefect, amazed me in many ways, and I can&#8217;t help but migrate everything to it.<\/p>\n\n\n\n<p>Prefect (and Airflow) is a workflow automation tool. You can orchestrate individual tasks to do more complex work. You could manage task dependencies, retry tasks when they fail, schedule them, etc.<\/p>\n\n\n\n<p>Workflow management is the\u00a0backbone of every data science project. Even small projects can have remarkable benefits with a tool like Prefect. It eliminates a significant part of repetitive tasks. With Prefect you can start automating tasks inside your Jupyter notebook. You can&#8217;t do airflow tasks in Jupyter Notebooks.<\/p>\n\n\n\n<p><b>Related: <\/b><a href=\"\/advantages-of-cloud-databases-over-on-premise-databases\"><b><i>11 Advantages of Cloud Databases Over On-Premise Databases.<\/i><\/b><\/a><\/p>\n\n\n\n<p>This article covers some of the frequent questions about Prefect. It includes<\/p>\n\n\n\n<ul>\n<li>a short intro to Prefect&#8217;s core concepts;<\/li>\n\n\n\n<li>why I decided to migrate from Airflow;<\/li>\n\n\n\n<li>Prefect&#8217;s incredible features and integration with other technologies and;<\/li>\n\n\n\n<li>how to decide between its cloud vs. on-premise deployment options.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Quickstart Prefect.<\/h2>\n\n\n\n<p>Prefect is both a minimal and complete workflow management tool. Just a bunch of Python functions to organize, and it&#8217;s unbelievably simple to set up. Yet it can do everything tools such as Airflow can and more.<\/p>\n\n\n\n<p>You can use PyPI, Conda, or Pipenv to install it, and it&#8217;s ready to rock. More on this in comparison with the Airflow section.<\/p>\n\n\n\n<p><script><br \/>\n            \/* This is ugly, but to ensure we only create this function once, and only call<br \/>\n            each JS library once, we need to check here in case there are multiple Wagtail<br \/>\n            blocks on this page. This will ensure we only load the minimum payload. *\/<br \/>\n            if(typeof loadPrismLanguage != 'function') {<br \/>\n                window.loadPrismLanguage = function(language) {<br \/>\n                    var libraries = [<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs\",<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/prism.min.js\"<br \/>\n                        },<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs-\" + language,<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/components\/prism-\" + language + \".min.js\"<br \/>\n                        },<br \/>\n        {<br \/>\n            \"id\": \"code-block-line-numbers\",<br \/>\n            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/plugins\/line-numbers\/prism-line-numbers.min.js\"<br \/>\n        }<\/p>\n<p>                    ];<\/p>\n<p>                    for(const library of libraries) {<br \/>\n                        if(document.getElementById(library[\"id\"]) == null) {<br \/>\n                            var s = document.createElement(\"script\");<br \/>\n                            s.id = library[\"id\"];<br \/>\n                            s.type = \"text\/javascript\";<br \/>\n                            s.src = library[\"url\"];<br \/>\n                            s.async = false;<br \/>\n                            document.body.appendChild(s);<br \/>\n                        }<br \/>\n                    }<br \/>\n                };<br \/>\n            }<\/p>\n<p>            loadPrismLanguage('bash');<\/p>\n<p>            language_class_name = 'language-bash';<br \/>\n            <\/script><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"pip install prefect\n# conda install -c conda-forge prefect\n# pipenv install --pre prefect\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #88C0D0\">pip<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">install<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">prefect<\/span><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># conda install -c conda-forge prefect<\/span><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># pipenv install --pre prefect<\/span><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Bash<\/span><\/div>\n\n\n\n<p><script><br \/>\n                var block_num = (typeof block_num === 'undefined') ? 0 : block_num;<br \/>\n                block_num++;<br \/>\n                document.getElementById('target-element-current').className = language_class_name;<br \/>\n                document.getElementById('target-element-current').id = 'target-element-' + block_num;<br \/>\n            <\/script><\/p>\n\n\n\n<p>Before we dive into using Prefect, let&#8217;s first see an unmanaged workflow. It makes understanding the role of Prefect in workflow management easy.<\/p>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<p>The below script queries an API (Extract\u2014E), picks the relevant fields from it (Transform\u2014T), and appends them to a file (Load\u2014L). It contains three Python functions that perform each of the tasks mentioned. It&#8217;s a straightforward yet everyday use case of workflow management tools\u2014ETL.<\/p>\n\n\n\n<p><script><br \/>\n            \/* This is ugly, but to ensure we only create this function once, and only call<br \/>\n            each JS library once, we need to check here in case there are multiple Wagtail<br \/>\n            blocks on this page. This will ensure we only load the minimum payload. *\/<br \/>\n            if(typeof loadPrismLanguage != 'function') {<br \/>\n                window.loadPrismLanguage = function(language) {<br \/>\n                    var libraries = [<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs\",<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/prism.min.js\"<br \/>\n                        },<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs-\" + language,<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/components\/prism-\" + language + \".min.js\"<br \/>\n                        },<br \/>\n        {<br \/>\n            \"id\": \"code-block-line-numbers\",<br \/>\n            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/plugins\/line-numbers\/prism-line-numbers.min.js\"<br \/>\n        }<\/p>\n<p>                    ];<\/p>\n<p>                    for(const library of libraries) {<br \/>\n                        if(document.getElementById(library[\"id\"]) == null) {<br \/>\n                            var s = document.createElement(\"script\");<br \/>\n                            s.id = library[\"id\"];<br \/>\n                            s.type = \"text\/javascript\";<br \/>\n                            s.src = library[\"url\"];<br \/>\n                            s.async = false;<br \/>\n                            document.body.appendChild(s);<br \/>\n                        }<br \/>\n                    }<br \/>\n                };<br \/>\n            }<\/p>\n<p>            loadPrismLanguage('python');<\/p>\n<p>            language_class_name = 'language-python';<br \/>\n            <\/script><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"import requests\n\ndef extract() -&gt; dict:\n    &quot;&quot;&quot;Use the Open Weather Map API to fetch Boston weather data.\n    Returns:\n        dict: a JSON response of many wheather measurements.\n    &quot;&quot;&quot;\n\n    url = &quot;https:\/\/api.openweathermap.org\/data\/2.5\/weather&quot;\n\n    # TODO: Use a real API key. You can get a free one from https:\/\/openweathermap.org\/\n    response = requests.request(\n        &quot;GET&quot;, url, params={&quot;q&quot;: &quot;Boston&quot;, &quot;appid&quot;: &quot;e5ecbcc49e3debeead24d0fe012fb46e&quot;}\n    )\n\n    return response.json()\n\n\ndef transform(response: dict) -&gt; float:\n    &quot;&quot;&quot;Process the response and extract windspeed information\n    Args:\n        response (dict): Response JSON from extract task\n    Returns:\n        float: Current wind speed\n    &quot;&quot;&quot;\n    return response.get(&quot;wind&quot;, {}).get(&quot;speed&quot;, 0.0)\n\n\ndef load(speed: float):\n    &quot;&quot;&quot;Append data to file\n    Args:\n        speed (float): Windspeed from transform task\n    &quot;&quot;&quot;\n\n    with open(&quot;windspeed.txt&quot;, &quot;a&quot;) as f:\n        f.write(str(speed) + &quot;\\n&quot;)\n\n\nif __name__ == &quot;__main__&quot;:\n    # Execute tasks in the right order.\n    response = extract()\n    windspeed = transform(response)\n    load(windspeed)\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> requests<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">extract<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">-&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">dict<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&quot;&quot;&quot;<\/span><span style=\"color: #A3BE8C\">Use the Open Weather Map API to fetch Boston weather data.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">    Returns:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">        dict: a JSON response of many wheather measurements.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">    <\/span><span style=\"color: #ECEFF4\">&quot;&quot;&quot;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    url <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">https:\/\/api.openweathermap.org\/data\/2.5\/weather<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #616E88\"># <\/span><span style=\"color: #81A1C1\">TODO<\/span><span style=\"color: #616E88\">: Use a real API key. You can get a free one from https:\/\/openweathermap.org\/<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    response <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> requests<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">request<\/span><span style=\"color: #ECEFF4\">(<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">GET<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> url<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">params<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #ECEFF4\">{<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">q<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">Boston<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">appid<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">e5ecbcc49e3debeead24d0fe012fb46e<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">}<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> response<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">json<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">transform<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">response<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">dict<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">-&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">float<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&quot;&quot;&quot;<\/span><span style=\"color: #A3BE8C\">Process the response and extract windspeed information<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">    Args:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">        response (dict): Response JSON from extract task<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">    Returns:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">        float: Current wind speed<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">    <\/span><span style=\"color: #ECEFF4\">&quot;&quot;&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> response<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">get<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">wind<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">{}).<\/span><span style=\"color: #88C0D0\">get<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">speed<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #B48EAD\">0.0<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">load<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">speed<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">float<\/span><span style=\"color: #ECEFF4\">):<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&quot;&quot;&quot;<\/span><span style=\"color: #A3BE8C\">Append data to file<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">    Args:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">        speed (float): Windspeed from transform task<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A3BE8C\">    <\/span><span style=\"color: #ECEFF4\">&quot;&quot;&quot;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">with<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">open<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">windspeed.txt<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">a<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">as<\/span><span style=\"color: #D8DEE9FF\"> f<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        f<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">write<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #88C0D0\">str<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">speed<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">+<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #EBCB8B\">\\n<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> __name__ <\/span><span style=\"color: #81A1C1\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">__main__<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #616E88\"># Execute tasks in the right order.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    response <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">extract<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    windspeed <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">transform<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">response<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">load<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">windspeed<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Python<\/span><\/div>\n\n\n\n<p><script><br \/>\n                var block_num = (typeof block_num === 'undefined') ? 0 : block_num;<br \/>\n                block_num++;<br \/>\n                document.getElementById('target-element-current').className = language_class_name;<br \/>\n                document.getElementById('target-element-current').id = 'target-element-' + block_num;<br \/>\n            <\/script><\/p>\n\n\n\n<p>This script downloads weather data from the OpenWeatherMap API and stores the windspeed value in a file. ETL applications in real life could be complex. But this example application covers the fundamental aspects very well.<\/p>\n\n\n\n<p>Note: Please replace the API key with a real one. You can get one from&nbsp;<a href=\"https:\/\/openweathermap.org\/api\" target=\"_blank\" rel=\"noopener\">https:\/\/openweathermap.org\/api<\/a>.<\/p>\n\n\n\n<p>You can run this script with the command&nbsp;<code>python app.py<\/code>where app.py is the name of your script file. This will create a new file called windspeed.txt in the current directory with one value. It&#8217;s the windspeed at Boston, MA, at the time you reach the API. If you rerun the script, it&#8217;ll append another value to the same file.<\/p>\n\n\n\n<p><strong>Related: <a href=\"\/how-to-make-a-pdf-reader-in-python\"><em>How to Make a PDF Text-to-Speech Reader in Python<\/em><\/a><\/strong><br><strong>Related: <a href=\"\/python-auto-run-tests\"><em>How to Run Python Tests on Every Commit Using GitHub Actions<\/em><\/a><\/strong><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Your first Prefect ETL workflow.<\/h3>\n\n\n\n<p>The above script works well. Yet, it lacks some critical features of a complete ETL, such as retrying and scheduling. Also, as mentioned earlier, a real-life ETL may have hundreds of tasks in a single workflow. Some of them can be run in parallel, whereas some depend on one or more other tasks.<\/p>\n\n\n\n<p>Imagine if there is a temporary network issue that prevents you from calling the API. The script would fail immediately with no further attempt. In live applications, such downtimes aren&#8217;t a miracle. They happen for several reasons\u2014server downtime, network downtime, and server query limit exceeds.<\/p>\n\n\n\n<p>Also, you have to manually execute the above script every time to update your windspeed.txt file. Yet, scheduling the workflow to run at a specific time in a predefined interval is common in ETL workflows.<\/p>\n\n\n\n<p>This is where tools such as Prefect and Airflow come to the rescue. Here&#8217;s how you could tweak the above Python code to make it a Prefect workflow.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"import requests\n\n# Importing Prefect task, Flow and Python timdelta\nfrom prefect import task, Flow\nfrom datetime import timedelta\n\n# decorater specifying how many times to retry and it's iterval.\n@task(max_retries=3, retry_delay=timedelta(minutes=3))\ndef extract() -&gt; dict:\n    ...\n\n\n@task\ndef transform(response: dict) -&gt; float:\n    ...\n\n@task(max_retries=3, retry_delay=timedelta(5))\ndef load(speed: float):\n    ...\n\n\n# Create a Prefect flow\nwith Flow(&quot;Windspeed Tracker&quot;) as flow:\n    # Execute tasks in the right order.\n    response = extract()\n    windspeed = transform(response)\n    load(windspeed)\n\nif __name__ == &quot;__main__&quot;:\n    # Execute the flow\n    flow.run()\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> requests<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Importing Prefect task, Flow and Python timdelta<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">from<\/span><span style=\"color: #D8DEE9FF\"> prefect <\/span><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> task<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> Flow<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">from<\/span><span style=\"color: #D8DEE9FF\"> datetime <\/span><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> timedelta<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># decorater specifying how many times to retry and it&#39;s iterval.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #ECEFF4\">@<\/span><span style=\"color: #D08770\">task<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">max_retries<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #B48EAD\">3<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">retry_delay<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #88C0D0\">timedelta<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">minutes<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #B48EAD\">3<\/span><span style=\"color: #ECEFF4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">extract<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">-&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">dict<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    ...<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #ECEFF4\">@<\/span><span style=\"color: #D08770\">task<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">transform<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">response<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">dict<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">-&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">float<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    ...<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #ECEFF4\">@<\/span><span style=\"color: #D08770\">task<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">max_retries<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #B48EAD\">3<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">retry_delay<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #88C0D0\">timedelta<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #B48EAD\">5<\/span><span style=\"color: #ECEFF4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">load<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">speed<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">float<\/span><span style=\"color: #ECEFF4\">):<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    ...<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Create a Prefect flow<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">with<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">Flow<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">Windspeed Tracker<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">as<\/span><span style=\"color: #D8DEE9FF\"> flow<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #616E88\"># Execute tasks in the right order.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    response <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">extract<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    windspeed <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">transform<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">response<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">load<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">windspeed<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> __name__ <\/span><span style=\"color: #81A1C1\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">__main__<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #616E88\"># Execute the flow<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    flow<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">run<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Python<\/span><\/div>\n\n\n\n<p>The\u00a0<code>@task<\/code>\u00a0decorator converts a regular Python function into a Prefect task. The optional arguments allow you to specify its retry behavior. We&#8217;ve configured the function to attempt three times before it fails in the above example. We&#8217;ve also configured it to delay each retry by three minutes.<\/p>\n\n\n\n<p>With this new setup, our ETL is resilient to the network issues we discussed earlier.<\/p>\n\n\n\n<p>To test its functioning, disconnect your computer from the network and run the script with\u00a0<code>python app.py<\/code>. You&#8217;ll see a message that the first attempt failed, and the next one will begin in the next 3 minutes. Within three minutes, connect your computer back to the internet. The already-running script will now finish without any errors.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Scheduling workflows with Prefect.<\/h3>\n\n\n\n<p>Retrying is only part of the ETL story. Another challenge for many workflow applications is to run them in scheduled intervals. Prefect&#8217;s scheduling API is straightforward for any Python programmer. Here&#8217;s how it works.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"...\n# Imports to facilitate Scheduling.\nfrom datetime import timedelta, datetime\nfrom prefect.schedules import IntervalSchedule\n\n...\n\n# Create a schedule object\nschedule = IntervalSchedule(\n    start_date=datetime.utcnow() + timedelta(seconds=5),\n    interval=timedelta(minutes=1),\n)\n\n# Attach the schedule object to the windspeed trakcer flow.\nwith Flow(&quot;Windspeed Tracker&quot;, schedule=schedule) as flow:\n    response = extract()\n    windspeed = transform(response)\n    load(windspeed)\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #D8DEE9FF\">...<\/span><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Imports to facilitate Scheduling.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">from<\/span><span style=\"color: #D8DEE9FF\"> datetime <\/span><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> timedelta<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> datetime<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">from<\/span><span style=\"color: #D8DEE9FF\"> prefect<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9FF\">schedules <\/span><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> IntervalSchedule<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">...<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Create a schedule object<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">schedule <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">IntervalSchedule<\/span><span style=\"color: #ECEFF4\">(<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #D8DEE9\">start_date<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\">datetime<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">utcnow<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">+<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">timedelta<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">seconds<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #B48EAD\">5<\/span><span style=\"color: #ECEFF4\">),<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #D8DEE9\">interval<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #88C0D0\">timedelta<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">minutes<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #B48EAD\">1<\/span><span style=\"color: #ECEFF4\">),<\/span><\/span>\n<span class=\"line\"><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Attach the schedule object to the windspeed trakcer flow.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">with<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">Flow<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">Windspeed Tracker<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">schedule<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\">schedule<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">as<\/span><span style=\"color: #D8DEE9FF\"> flow<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    response <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">extract<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    windspeed <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">transform<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">response<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">load<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">windspeed<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Python<\/span><\/div>\n\n\n\n<p>We&#8217;ve created an IntervalSchedule object that starts five seconds from the execution of the script. We&#8217;ve also configured it to run in a one-minute interval.<\/p>\n\n\n\n<p>If you run the script with&nbsp;<code>python app.py<\/code>&nbsp;and monitor the windspeed.txt file, you will see new values in it every minute.<\/p>\n\n\n\n<p>In addition to this simple scheduling, Prefect&#8217;s schedule API offers more control over it. You can schedule workflows in a cron-like method, use clock time with timezones, or do more fun stuff like executing workflow only on weekends. I haven&#8217;t covered them all here, but Prefect&#8217;s official&nbsp;<a href=\"https:\/\/docs.prefect.io\/core\/concepts\/schedules.html#clocks\" target=\"_blank\" rel=\"noopener\">docs<\/a>&nbsp;about this are perfect.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The Prefect UI.<\/h3>\n\n\n\n<p>Like Airflow (and many others,) Prefect too ships with a server with a beautiful UI. It allows you to control and visualize your workflow executions.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"489\" src=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-84-1024x489.png\" alt=\"The prefect UI\" class=\"wp-image-931\" title=\"\" srcset=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-84-1024x489.png 1024w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-84-300x143.png 300w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-84-768x366.png 768w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-84.png 1400w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u00a0You need to have docker and docker-compose installed on your computer to run this. But starting it is surprisingly a single command.<\/p>\n\n\n\n<p><script><br \/>\n            \/* This is ugly, but to ensure we only create this function once, and only call<br \/>\n            each JS library once, we need to check here in case there are multiple Wagtail<br \/>\n            blocks on this page. This will ensure we only load the minimum payload. *\/<br \/>\n            if(typeof loadPrismLanguage != 'function') {<br \/>\n                window.loadPrismLanguage = function(language) {<br \/>\n                    var libraries = [<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs\",<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/prism.min.js\"<br \/>\n                        },<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs-\" + language,<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/components\/prism-\" + language + \".min.js\"<br \/>\n                        },<br \/>\n        {<br \/>\n            \"id\": \"code-block-line-numbers\",<br \/>\n            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/plugins\/line-numbers\/prism-line-numbers.min.js\"<br \/>\n        }<\/p>\n<p>                    ];<\/p>\n<p>                    for(const library of libraries) {<br \/>\n                        if(document.getElementById(library[\"id\"]) == null) {<br \/>\n                            var s = document.createElement(\"script\");<br \/>\n                            s.id = library[\"id\"];<br \/>\n                            s.type = \"text\/javascript\";<br \/>\n                            s.src = library[\"url\"];<br \/>\n                            s.async = false;<br \/>\n                            document.body.appendChild(s);<br \/>\n                        }<br \/>\n                    }<br \/>\n                };<br \/>\n            }<\/p>\n<p>            loadPrismLanguage('bash');<\/p>\n<p>            language_class_name = 'language-bash';<br \/>\n            <\/script><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"prefect server start\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #88C0D0\">prefect<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">server<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">start<\/span><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Bash<\/span><\/div>\n\n\n\n<p><script><br \/>\n                var block_num = (typeof block_num === 'undefined') ? 0 : block_num;<br \/>\n                block_num++;<br \/>\n                document.getElementById('target-element-current').className = language_class_name;<br \/>\n                document.getElementById('target-element-current').id = 'target-element-' + block_num;<br \/>\n            <\/script><\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"190\" src=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_hqErfnvjCAOwkU0WcSebcA.gif\" alt=\"Starting the Prefect Server\" class=\"wp-image-932\" title=\"\"><\/figure><\/div>\n\n\n<p>This command will start the prefect server, and you can access it through your web browser:&nbsp;<code>&lt;http:\/\/localhost:8080\/&gt;<\/code><a href=\"http:\/\/localhost:8080\/\" target=\"_blank\" rel=\"noopener\">.<\/a><\/p>\n\n\n\n<p>However, the Prefect server alone could not execute your workflows. Its role is only enabling a control panel for all your Prefect activities. Because this dashboard is decoupled from the rest of the application, you can use Prefect Cloud to do the same. We&#8217;ll discuss this in detail later.<\/p>\n\n\n\n<p><b>Related:<\/b> <a href=\"\/plotly-dashboards-in-python\"><b><i>How I Create Dazzling Dashboards Purely in Python.<\/i><\/b><\/a><\/p>\n\n\n\n<p>To execute tasks, we need a few more things. The good news is they, too, aren&#8217;t complicated.<\/p>\n\n\n\n<p>Because servers are merely control panels, we need an agent to execute the workflow. The below command will start a local agent. Instead of a local agent; you can choose a docker agent or a Kubernetes one if your project needs them.<\/p>\n\n\n\n<p><script><br \/>\n            \/* This is ugly, but to ensure we only create this function once, and only call<br \/>\n            each JS library once, we need to check here in case there are multiple Wagtail<br \/>\n            blocks on this page. This will ensure we only load the minimum payload. *\/<br \/>\n            if(typeof loadPrismLanguage != 'function') {<br \/>\n                window.loadPrismLanguage = function(language) {<br \/>\n                    var libraries = [<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs\",<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/prism.min.js\"<br \/>\n                        },<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs-\" + language,<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/components\/prism-\" + language + \".min.js\"<br \/>\n                        },<br \/>\n        {<br \/>\n            \"id\": \"code-block-line-numbers\",<br \/>\n            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/plugins\/line-numbers\/prism-line-numbers.min.js\"<br \/>\n        }<\/p>\n<p>                    ];<\/p>\n<p>                    for(const library of libraries) {<br \/>\n                        if(document.getElementById(library[\"id\"]) == null) {<br \/>\n                            var s = document.createElement(\"script\");<br \/>\n                            s.id = library[\"id\"];<br \/>\n                            s.type = \"text\/javascript\";<br \/>\n                            s.src = library[\"url\"];<br \/>\n                            s.async = false;<br \/>\n                            document.body.appendChild(s);<br \/>\n                        }<br \/>\n                    }<br \/>\n                };<br \/>\n            }<\/p>\n<p>            loadPrismLanguage('bash');<\/p>\n<p>            language_class_name = 'language-bash';<br \/>\n            <\/script><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"prefect agent local start\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #88C0D0\">prefect<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">agent<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">local<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">start<\/span><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Bash<\/span><\/div>\n\n\n\n<p><script><br \/>\n                var block_num = (typeof block_num === 'undefined') ? 0 : block_num;<br \/>\n                block_num++;<br \/>\n                document.getElementById('target-element-current').className = language_class_name;<br \/>\n                document.getElementById('target-element-current').id = 'target-element-' + block_num;<br \/>\n            <\/script><\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"256\" src=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_LN65oSw6ekNSE7LXFVrkYA-1024x256.gif\" alt=\"Starting Prefect local agent\" class=\"wp-image-933\" title=\"\" srcset=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_LN65oSw6ekNSE7LXFVrkYA-1024x256.gif 1024w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_LN65oSw6ekNSE7LXFVrkYA-300x75.gif 300w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_LN65oSw6ekNSE7LXFVrkYA-768x192.gif 768w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>Once the server and the agent are running, you&#8217;ll have to create a project and register your workflow with that project. To do this, change the line that executes the flow to the following.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"if __name__ == &quot;__main__&quot;:\n    flow.register('Tutorial')\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> __name__ <\/span><span style=\"color: #81A1C1\">==<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">__main__<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    flow<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">register<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">Tutorial<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Python<\/span><\/div>\n\n\n\n<p>Now in the terminal, you can create a project with the&nbsp;<code>prefect create project &lt;project name&gt;<\/code>&nbsp;command. Then rerunning the script will register it to the project instead of running it immediately.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"prefect create project 'Tutorial'\npython app.py\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #88C0D0\">prefect<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">create<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">project<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">Tutorial<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #88C0D0\">python<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #A3BE8C\">app.py<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Bash<\/span><\/div>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"256\" src=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_0YKYt89OzjROFQCxqwkD4A-1024x256.gif\" alt=\"Running the Prefect project to register it with the server. \" class=\"wp-image-934\" title=\"\" srcset=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_0YKYt89OzjROFQCxqwkD4A-1024x256.gif 1024w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_0YKYt89OzjROFQCxqwkD4A-300x75.gif 300w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_0YKYt89OzjROFQCxqwkD4A-768x192.gif 768w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>In the web UI, you can see the new Project &#8216;Tutorial&#8217; is in the dropdown, and our windspeed tracker is in the list of flows. The flow is already scheduled and running. If you prefer, you can run them manually as well.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"720\" height=\"343\" src=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-85.png\" alt=\"The registered project on Prefect server\" class=\"wp-image-935\" title=\"\" srcset=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-85.png 720w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-85-300x143.png 300w\" sizes=\"(max-width: 720px) 100vw, 720px\" \/><\/figure><\/div>\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Running workflows with parameters.<\/h3>\n\n\n\n<p>The workflow we created in the previous exercise is rigid. It queries only for Boston, MA, and we can not change it. This is where we can use parameters. Here&#8217;s how we tweak our Python code to accept a parameter at the run time.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"# Import parameters\nfrom prefect import task, Flow, Parameter\n\n# Tweak the function to accept city argument.\n@task(max_retries=3, retry_delay=timedelta(minutes=3))\ndef extract(city: str) -&gt; dict:\n    url = &quot;https:\/\/api.openweathermap.org\/data\/2.5\/weather&quot;\n\n    # TODO: Use a real API key. You can get a free one from https:\/\/openweathermap.org\/\n    response = requests.request(\n        &quot;GET&quot;, url, params={&quot;q&quot;: city, &quot;appid&quot;: &quot;e5ecbcc49e3debeead24d0fe012fb46e&quot;}\n    )\n\n    return response.json()\n\n...\n\nwith Flow(&quot;Windspeed Tracker&quot;, schedule=schedule) as flow:\n    # create a city parameter with the default value Boston and pass it to the extract task.\n    city = Parameter(&quot;city&quot;, default=&quot;Boston&quot;)\n    response = extract(city)\n    ...\n\n...\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #616E88\"># Import parameters<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">from<\/span><span style=\"color: #D8DEE9FF\"> prefect <\/span><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> task<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> Flow<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> Parameter<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Tweak the function to accept city argument.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #ECEFF4\">@<\/span><span style=\"color: #D08770\">task<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">max_retries<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #B48EAD\">3<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">retry_delay<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #88C0D0\">timedelta<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">minutes<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #B48EAD\">3<\/span><span style=\"color: #ECEFF4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">def<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">extract<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9\">city<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">str<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">-&gt;<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">dict<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    url <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">https:\/\/api.openweathermap.org\/data\/2.5\/weather<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #616E88\"># <\/span><span style=\"color: #81A1C1\">TODO<\/span><span style=\"color: #616E88\">: Use a real API key. You can get a free one from https:\/\/openweathermap.org\/<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    response <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> requests<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">request<\/span><span style=\"color: #ECEFF4\">(<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">GET<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> url<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">params<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #ECEFF4\">{<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">q<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> city<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">appid<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">e5ecbcc49e3debeead24d0fe012fb46e<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">}<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> response<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">json<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">...<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">with<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">Flow<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">Windspeed Tracker<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">schedule<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\">schedule<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">as<\/span><span style=\"color: #D8DEE9FF\"> flow<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #616E88\"># create a city parameter with the default value Boston and pass it to the extract task.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    city <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">Parameter<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">city<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">default<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">Boston<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    response <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">extract<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">city<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    ...<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">...<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Python<\/span><\/div>\n\n\n\n<p>We&#8217;ve changed the function to accept the city argument and set it dynamically in the API query. Inside the Flow, we create a parameter object with the default value &#8216;Boston&#8217; and pass it to the Extract task.<\/p>\n\n\n\n<p>If you run the windspeed tracker workflow manually in the UI, you&#8217;ll see a section called input. Here you can set the value of the city for every execution.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"800\" height=\"450\" src=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/1_FkFidj7Co6utyMJ-duejhQ.gif\" alt=\"Running the project from Prefect server\" class=\"wp-image-936\" title=\"\"><\/figure><\/div>\n\n\n<p>This is a convenient way to run workflows. In many cases, ETLs and any other workflow come with run-time parameters.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why did I decide to migrate from Airflow to Prefect?<\/h2>\n\n\n\n<p>Airflow is a fantastic platform for workflow management. It saved me a ton of time on many projects. Yet, we need to appreciate new technologies taking over the old ones. That&#8217;s the case with Airflow and Prefect.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>Airflow got many things right, but its core assumptions never anticipated the rich variety of data applications that have emerged.<\/p>\n\n\n\n<p>\u2014 Prefect Documentation.<\/p>\n<\/blockquote>\n\n\n\n<p>What I describe here aren&#8217;t dead-ends if you prefer Airflow. We have workarounds for most problems. Yet, it&#8217;s convenient in Prefect because the tool natively supports them.<\/p>\n\n\n\n<p>Prefect&#8217;s&nbsp;installation&nbsp;is exceptionally straightforward compared to Airflow. For trained eyes, it may not be a problem. Yet, for whoever wants to start on workflow orchestration and automation, it&#8217;s a hassle.<\/p>\n\n\n\n<p>Airflow needs a&nbsp;server&nbsp;running in the backend to perform any task. Yet, in Prefect, a server is optional. This is a massive benefit of using Prefect. I have many pet projects running on my computer as services. Earlier, I had to have an Airflow server commencing at the startup. Because Prefect could run standalone, I don&#8217;t have to turn on this additional server anymore.<\/p>\n\n\n\n<p>Airflow doesn&#8217;t have the flexibility to run workflows (or DAGs) with\u00a0parameters. The workaround I used to have is to let the application read them from a database. This isn&#8217;t an excellent programming technique for such a simple task. Prefect&#8217;s parameter concept is exceptional on this front.<\/p>\n\n\n\n<p>Prefect allows having different&nbsp;versions&nbsp;of the same workflow. Every time you register a workflow to the project, it creates a new version. If you need to run a previous version, you can easily select it in a dropdown. This isn&#8217;t possible with Airflow.<\/p>\n\n\n\n<p>Prefect also allows us to create teams and role-based access controls. Each team could manage its configuration.&nbsp;Authorization&nbsp;is a critical part of every modern application, and Prefect handles it in the best way possible.<\/p>\n\n\n\n<p>Lastly, I find Prefect&#8217;s&nbsp;UI&nbsp;more intuitive and appealing. Airflow&#8217;s UI, especially its task execution visualization, was difficult at first to understand.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prefect&#8217;s ecosystem and integration with other technologies.<\/h2>\n\n\n\n<p>Prefect has inbuilt integration with many other technologies. It eliminates a ton of overhead and makes working with them super easy.<\/p>\n\n\n\n<p>Live projects often have to deal with several technologies. For example, when your ETL fails, you may want to send an email or a Slack notification to the maintainer.<\/p>\n\n\n\n<p>In Prefect, sending such notifications is effortless. You can use the EmailTask from the Prefect&#8217;s task library, set the credentials, and start sending emails.<\/p>\n\n\n\n<p>You can learn more about Prefect&#8217;s rich ecosystem in their&nbsp;<a href=\"https:\/\/docs.prefect.io\/core\/task_library\/overview.html#task-library-in-action\" target=\"_blank\" rel=\"noopener\">official documentation<\/a>. In this article, we&#8217;ll see how to send email notifications.<\/p>\n\n\n\n<p>To send emails, we need to make the credentials accessible to the Prefect agent. You can do that by creating the below file in&nbsp;<code>$HOME\/.prefect\/config.toml<\/code>.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"[context.secrets]\nEMAIL_USERNAME = &quot;<your email id&gt;&quot;\nEMAIL_PASSWORD = &quot;<your email password&gt;&quot;\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #ECEFF4\">[<\/span><span style=\"color: #D8DEE9FF\">context.secrets<\/span><span style=\"color: #ECEFF4\">]<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9\">EMAIL_USERNAME<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&lt;your email id&gt;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9\">EMAIL_PASSWORD<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">&lt;your email password&gt;<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">TOML<\/span><\/div>\n\n\n\n<p>Your app is now ready to send emails. Here&#8217;s how we send a notification when we successfully capture a windspeed measure.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"# Import email task\nfrom prefect.tasks.notifications.email_task import EmailTask\n\n...\n\n# Create an email_task object. Use all static content here.\nemail_task = EmailTask(\n    subject=&quot;A new windspeed captured&quot;,\n    email_to=&quot;receiver_email@gmail.com&quot;,\n    email_from=&quot;your_email@gmail.com&quot;,\n)\n\nwith Flow(&quot;Windspeed Tracker&quot;, schedule=schedule) as flow:\n    ...\n\n    # Call the email task with variable content.\n    email_task(\n        msg=str(windspeed),\n    )\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #616E88\"># Import email task<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">from<\/span><span style=\"color: #D8DEE9FF\"> prefect<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9FF\">tasks<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9FF\">notifications<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9FF\">email_task <\/span><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> EmailTask<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">...<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Create an email_task object. Use all static content here.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">email_task <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">EmailTask<\/span><span style=\"color: #ECEFF4\">(<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #D8DEE9\">subject<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">A new windspeed captured<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #D8DEE9\">email_to<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">receiver_email@gmail.com<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #D8DEE9\">email_from<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">your_email@gmail.com<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">with<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">Flow<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">Windspeed Tracker<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">schedule<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\">schedule<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">as<\/span><span style=\"color: #D8DEE9FF\"> flow<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    ...<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #616E88\"># Call the email task with variable content.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #88C0D0\">email_task<\/span><span style=\"color: #ECEFF4\">(<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #D8DEE9\">msg<\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #88C0D0\">str<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">windspeed<\/span><span style=\"color: #ECEFF4\">),<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Python<\/span><\/div>\n\n\n\n<p>In the above code, we&#8217;ve created an instance of the EmailTask class. We&#8217;ve used all the static elements of our email configurations during initiating. Then inside the Flow, we&#8217;ve used it with passing variable content.<\/p>\n\n\n\n<p>This configuration above will send an email with the captured windspeed measurement. But its subject will always remain &#8216;A new windspeed captured.&#8217;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prefect Cloud vs. On-Premis Server Deployments.<\/h2>\n\n\n\n<p>We&#8217;ve already looked into how we can start an on-premise server. Because this server is only a control panel, you could easily use the cloud version instead. To do this, we have a few additional steps to follow.<\/p>\n\n\n\n<ol>\n<li>Create a&nbsp;<a href=\"https:\/\/cloud.prefect.io\/\" target=\"_blank\" rel=\"noopener\">Prefect cloud<\/a>&nbsp;account.<\/li>\n\n\n\n<li>Generate a key from the&nbsp;<a href=\"https:\/\/cloud.prefect.io\/user\/keys\" target=\"_blank\" rel=\"noopener\">API Key Page<\/a>.<\/li>\n\n\n\n<li>In your terminal, set the backend to the cloud:\u00a0<code>prefect backend cloud<\/code>.<\/li>\n\n\n\n<li>Also, log in with the generated key:\u00a0<code>prefect auth login --key YOUR_API_KEY<\/code>.<\/li>\n\n\n\n<li>Now, start the agent as usual.&nbsp;<code>prefect agent local start<\/code>.<\/li>\n<\/ol>\n\n\n\n<p>In the cloud dashboard, you can manage everything you did on the local server before.<\/p>\n\n\n\n<p>A big question when choosing between cloud and server versions is&nbsp;security. According to Prefect&#8217;s docs, the server only stores workflow execution-related data and voluntary information provided by the user. Since the agent in your local computer executes the logic, you can control where you store your data.<\/p>\n\n\n\n<p>The cloud option is suitable for&nbsp;performance&nbsp;reasons too. With one cloud server, you can manage more than one agent. Thus, you can scale your app effortlessly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Final Thoughts<\/h2>\n\n\n\n<p>Airflow was my ultimate choice for building ETLs and other workflow management applications. Yet, Prefect changed my mind, and now I&#8217;m migrating everything from Airflow to Prefect.<\/p>\n\n\n\n<p>Prefect is a straightforward tool that is flexible to extend beyond what Airflow can do. You can run it even inside a Jupyter Notebook. Also, you can host it as a complete task management solution.<\/p>\n\n\n\n<p>In addition to the central problem of workflow management, Prefect solves several other issues you may frequently encounter in a live system. Managing teams with authorization controls and sending notifications are some of them.<\/p>\n\n\n\n<p>In this article, we&#8217;ve discussed how to create an ETL that<\/p>\n\n\n\n<ul>\n<li>retries some tasks as configured;<\/li>\n\n\n\n<li>run workflows in a schedule;<\/li>\n\n\n\n<li>accepts run-time parameters and;<\/li>\n\n\n\n<li>sends an email notification when it&#8217;s done.<\/li>\n<\/ul>\n\n\n\n<p>We&#8217;ve only scratched the surface of Prefects capabilities. I recommend reading the official\u00a0<a href=\"https:\/\/docs.prefect.io\/core\/\" target=\"_blank\" rel=\"noopener\">documentation<\/a>\u00a0for more information.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dots\"\/>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>Thanks for the read, friend. It seems you and I have lots of common interests. Say Hi to me on <a href=\"https:\/\/www.linkedin.com\/in\/thuwarakesh\/\" target=\"_blank\" rel=\"nofollow noopener\"><strong>LinkedIn<\/strong><\/a>, <a href=\"https:\/\/twitter.com\/Thuwarakesh\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Twitter<\/strong><\/a>, and <a href=\"https:\/\/thuwarakesh.medium.com\/subscribe\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Medium<\/strong><\/a>. <\/p>\n\n\n\n<p>Not a Medium member yet? Please use this link to <a href=\"https:\/\/thuwarakesh.medium.com\/membership\" target=\"_blank\" rel=\"nofollow noopener\"><strong>become a member<\/strong><\/a> because I earn a commission for referring at no extra cost for you.<\/p>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>I migrated all my ETL work from Airflow to this python orchestration framework.<\/p>\n","protected":false},"author":2,"featured_media":195,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_blocks_custom_css":"","_kad_blocks_head_custom_js":"","_kad_blocks_body_custom_js":"","_kad_blocks_footer_custom_js":"","_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"footnotes":""},"categories":[12,3],"tags":[],"taxonomy_info":{"category":[{"value":12,"label":"MLOps"},{"value":3,"label":"Python"}]},"featured_image_src_large":["https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/prefect-1024x631.png",1024,631,true],"author_info":{"display_name":"Thuwarakesh","author_link":"https:\/\/www.the-analytics.club\/author\/thuwarakesh\/"},"comment_info":0,"category_info":[{"term_id":12,"name":"MLOps","slug":"mlops","term_group":0,"term_taxonomy_id":12,"taxonomy":"category","description":"","parent":0,"count":13,"filter":"raw","cat_ID":12,"category_count":13,"category_description":"","cat_name":"MLOps","category_nicename":"mlops","category_parent":0},{"term_id":3,"name":"Python","slug":"python","term_group":0,"term_taxonomy_id":3,"taxonomy":"category","description":"","parent":5,"count":52,"filter":"raw","cat_ID":3,"category_count":52,"category_description":"","cat_name":"Python","category_nicename":"python","category_parent":5}],"tag_info":false,"_links":{"self":[{"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/posts\/304"}],"collection":[{"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/comments?post=304"}],"version-history":[{"count":1,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/posts\/304\/revisions"}],"predecessor-version":[{"id":938,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/posts\/304\/revisions\/938"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/media\/195"}],"wp:attachment":[{"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/media?parent=304"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/categories?post=304"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/tags?post=304"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}