Overnight package run - S3 Storage

Thank you for the example !! 

I think you need to provide a ChatGPT prompt for creating scripts in pascal for us Python peeps.

Resumable File Loop is also good btw 

Question on LOOP FILE !! This functionality is AMAZING . SERIOUSLY . Tonnes of questions 
1. LoopPosition and SavedLoopPosition are new variables now available with FILE LOOP ? What's the difference between them ? 
2. What does #LoopValuesCount# do ? 
3. When do they get updated /incremented when loop starts (new iteration?) or when previous iteration finishes ?
4. The example is really smart , I assume when I close the package window , are they lost ? 
5. What happens if AETL crashes in middle of loop ? Then is the progress lost?
6. Is it possible to also ask user if they wish to resume the loop or simply start from beginning ? If Yes , how ?
 

1. LoopPosition and SavedLoopPosition are new variables now available with FILE LOOP ? What's the difference between them ? 

  LoopPosition was always available as a loop variable, SavedLoopPosition variable is created during example execution

2. What does #LoopValuesCount# do ? 

It holds total count of values in case of file it is number of lines or in case of database total number of records 

3. When do they get updated /incremented when loop starts (new iteration?) or when previous iteration finishes ?

LoopPosition is updated by loop action, SavedLoopPosition is assigned/calculated when package execution starts. During loop execution LoopPosition variable is saved into  #ROOT-FOLDER#\SavedLoopPosition.txt  file

4. The example is really smart , I assume when I close the package window , are they lost ? 

No

5. What happens if AETL crashes in middle of loop ? Then is the progress lost?

No

6. Is it possible to also ask user if they wish to resume the loop or simply start from beginning ? If Yes , how ?

Just delete #ROOT-FOLDER#\SavedLoopPosition.txt  file

Basically the way it works is very simple.
When the package is executed first time the file #ROOT-FOLDER#\SavedLoopPosition.txt  does not exists so loop execution starts from the beginning (SavedLoopPosition=-1)
During loop execution LoopPosition variable is saved into  #ROOT-FOLDER#\SavedLoopPosition.txt  file
If the the  package fails and run it the package loads SavedLoopPosition variable value from  #ROOT-FOLDER#\SavedLoopPosition.txt  file

I hope it is clear now

HI Mike,
Thanks for answering my doubts. It gives me more clarity into how loops run now. I will have more questions I am sure

Below is a WIP video which I will replace with a proper video 
www.loom.com/share/76cb31322e2d46d0ba644...89-900c-0decb390c2cb

1 Currently we able to run a transformation and generate GUID and decide a temp location where file will be downloaded
2. We then run the package which downloads the files to a C:\tmp\Guid.extension and then uploads the same to S3
3. It then proceeds To delete the file downloaded


TODO
1. Introduce writing to CSV file so progress is saved
2. Introduce loop variable to allow resuming of loop