Overnight package run - S3 Storage

Hellos,

I want to attempt a package which runs for 6-7 hours , need best trips and tricks 

Project 
1. I have roughly 5000 files on which I need to move from my website to S3 compatible bucket Telnyx object storage.
2. I already have a list of all 5000 files in a csv files
3. I have generated a GUID for each row so filename can be obfuscated using transformation.
4. Using loop action , each file will be Downloaded with HTTP Get , RENAMED as per GUID and then uploaded to telnyx
5. Object ID will be return copied in CSV and udpated

Question
1. I see a package for Amazon S3 , what about other S3 compatible buckets?
2. With such a long operation , 5-6 hours , I am scared of AETL hanging or crashing midway. One workaround (in my head) was to run this using command line 
3. Any other tips / tricks to make this feasible ? I was consider batching by splitting csv file into having secondary loop to process only 200 files per batch 
 

1. I see a package for Amazon S3 , what about other S3 compatible buckets?

We have not tried but it it is truly S3 compatible all you need to do is to use a different end point.
Please let us know id it works or not

2. With such a long operation , 5-6 hours , I am scared of AETL hanging or crashing midway. One workaround (in my head) was to run this using command line 
3. Any other tips / tricks to make this feasible ? I was consider batching by splitting csv file into having secondary loop to process only 200 files per batch 

Here are some things to consider

1. Using command line is a definitely good idea
2. Test with small number of files first
3. Plan for failure, so you can recover from it.
  For example you should be able to restart the process from the failure point. eg record current file number somewhere than restart from this line

Please keep us posted on you progress, provide some some screenshots etc.

1. HI , it's truly S3 compatible . I changed the

Default end point to below (confirmed working with a python script)
us-central-1.telnyxstorage.com

Error sending data: (12007) The server name or address could not be resolved. This is same with or without any api/key entered

2. We do need to change the terminology here , it's Bucket and not basket
 

3. When we use command line , can we print the progress of the script of it ? if yes how ? See example below with simple python example 
 

4. Yeah restarting front point of failure is good advise. Thinking aloud , I need to run a transformation with validation to skip files with existing object ID before loop starts,
5. I also then need to update object id received to either file or buffer within loop for each iteration. 
6. Question is there a concept of memory buffer within loop to store information after loop breaks  

7. Do we get an object id back when uploading) ?

8. For my progress , besides screenshots,  I think we can do a video of the end product where we create this entire pipeline from scratch  

Can you provide us with connection details please (via email) so we can test it in the office

Good point our spelling skills are not getting any better 

>>>When we use command line , can we print the progress of the script of it ? if yes how ? See example below with simple python example 

If you are talking  about you can write to the log 

etltools.WriteToLog('We are working veryyy hard here')

>>> Question is there a concept of memory buffer within loop to store information after loop breaks?

It does store current(last)  value/values  in variables so you should be able to access it even after loop ends

>>> How do we get an object id back when uploading)

That has to be investigated.


 

>>Can you provide us with connection details please (via email) so we can test it in the office

Will generate and send you

>> If you are talking  about you can write to the log 
I meant since we've a big package , and we run from command line , it' would be great to see update within loop at command line  or message display there in command line (no python)

>>> How do we get an object id back when uploading,That has to be investigated.
OK
 

Thank you for providing connection details i run some tests and it worked absolutely fine