[Guide] Using ETL-Tools Packages for Workflow Orchestration for Python packages

Warning
I am not a programmer ,if you don't' agree I apologise , code all the way


One non advertised usage of ETL-Tools is to use it for flow orchestration between python scripts , Yes you heard me right , the space occupied by Airflow or Prefect Server. 

Allow me to demonstrate with a photo first and then my observations
 

• Step 5 & Step 6 are both Python script ,Input of 6 comes from 5 
• Like Prefect Server or Airflow ,it's very easy to see from the logs where the mistakes are coming from

 

• You get tonnes of Options , whether script should run async or parallel , either by waiting for script to finish or not
• Options 1/2/3 provide you to run scripts with all your command line options 

My take

• It's easy to get chatGPT to write small script and maintain it, the moment script becomes complicated it starts becoming difficult to troubleshoot . 
• Using above approach , with modular scripts , you can ensure that it's easy to troubleshoot 
• We've slowly moved many things out of Python and moved them to ETL-Tools like mailing with postmark
• It's a wonderful marriage of low code+ no code

Looks very nice and useful

admin wrote:

Looks very nice and useful

You have no idea , for ChatGPT programmers such as myself , this is so easy to manage flow between scripts without learning anything new !