Hello,
I wanted to make the post to share my experience. As of Today, I have spent roughly 40 days into ETL . Happily purchased it as well.
I randomly stumbled across ETL-Tools on Feb 8 , website looks dated but software looked powerful , I asked them few questions regarding functionality , I was told everything's possible, if not , we will add it , which I didn't believe in beginning , but they really do (if you make sense in your request) . Really unique was they sent the trial key with the email too!
Background
1. I had the luxury of working processes , re-creation of those processes in ETL was easier , as my output was very clear and precise.
2. Majority of my data comes via CSV format from SQL . We have license for AUTOSQL with built in scheduler , it works , Although ETL can easily read from SQL ,"if it ain't broke, don't fix it".
3. We are big on python scripts using chatGPT.
Learnings
1. The software is very mature & like photoshop very complex & multiple ways of doing things.
2. The videos are very very helpful in figuring with topics.
3. Documentation is detailed but lacking with visual examples in few things.
4. Forums are very helpful and developers reply to almost everything , I have asked so many silly question , it's beyond my imagination . In fact I would say Forums were a major decision maker to purchase it
5. Version Control are really useful before making major changes
6. You still need to keep saving your transformations from time to time as ATS file , I have had major cases of my transformation gone for random behaviour. I can't put my finger on it yet.
7. The Flow of Packages is very very intuitive , almost magical.
8. Although developers say calculations are slow, I just accepted some lack of speed and my transformations became more compact and much faster (for my brain)
Here's what I have figured so far.
1. Start with Installation of SQL Express from beginning, it's free & syncs all your changes of repository in a single location .
2. I have worked with 3 Connections so Far - Directories with CSV & Excel Files, SQL (from#1) & MySQL Connection, Goes without saying you need to have username/password for SQL & MYSQL connection.
3. Created a scheduler automation which runs at 7 AM , reads list of all employees from CSV file , filters for active employees, updates our internal portal on Wordpress mySQL , so only active members can create tickets.
4, Another which updates list of all active orders and their BOM from csv files, updates our internal portal on Wordpress mysql , so purchase team can create tickets on the same.
5. Run pythons scripts using scheduler.
6. Used Packages to run python scripts , as a step in between transformations.
7. For testing we were running flask server to test few incoming webhooks , using URL Logger we have a more trusted solution , no production solution yet , but work in progress.
8. IN ERP, data entry operator keeps forgetting to enter few optional details in BOM , using a transformations , ill filled items are filtered, then using a packages & scheduler , email is sent to relevant manager so action can be taken daily.
9. Run SQL scripts to truncate a table just before uploading a file.
Future !
Currently I have used maybe 5% of what ETL can do , loads of interesting things to explore in future.
Use URL-Logger for running a webhook to Claude/ChatGPT to outgoing webhook.
Figure Object library
Find if ETL can be used to do data collection using XLS file, use IMAP to receive , use validations and do replies if possible somehow
Thank you