Overnight package run - S3 Storage

More
1 year 5 months ago #25072 by admin
I am a little bit confused about object id...

What it is exactly?

I have heard about Key and Version ID but not Object Id

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
1 year 5 months ago #25073 by prashant
You guys cheat !!It worked absolutely fine when you put the endpoint as 

us-central-1.telnyxstorage.com

But if I put  us-central-1.telnyxstorage.com/ or  us-central-1.telnyxstorage.com/ I get the error !

>> Object ID
When you upload the item to S3 bucket , you get the object ID through which you can construct the URL of the said uploaded object.
The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

More
1 year 5 months ago - 1 year 5 months ago #25074 by admin
I see what you mean now, should be easy to implement

We will let you know once it is ready.

Mike
ETL Architect
Last edit: 1 year 5 months ago by admin.
The following user(s) said Thank You: prashant

Please Log in or Create an account to join the conversation.

More
1 year 5 months ago #25075 by prashant
Any ideas on how to improve the loop resume from failure ?? or is there a hidden loop resume feature I am not aware of ?

1. Step 1 - Duplicate INPUT file (one time, can do check If file exists to ensure) . Introduce a marker column - is_processed (currently blank)
2. Step 2 - Transformation loads both file check , do a lookup or join to get the value of is_processed
3. Do a Validation , to reject items where is_processed = True
4. Load loop , within loop update the duplicated input_file with is_processed=True & with object id
5. If loop breaks or system shuts we can can restart from Step 2 

 

Please Log in or Create an account to join the conversation.

More
1 year 5 months ago #25076 by admin
Please provide example of the file

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
1 year 5 months ago #25077 by admin
FYI

18/04/2025 9.3.5.36

  + Added: Amazon S3 Key variable to Amazon S3 Upload Dynamic Action


 

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

Cookies user preferences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Marketing
Set of techniques which have for object the commercial strategy and in particular the market study.
Google
Accept
Decline
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline
Google Analytics
Accept
Decline
Functional
Tools used to give you more features when navigating on the website, this can include social sharing.
Advertisement
If you accept, the ads on the page will be adapted to your preferences.
Google Ad
Accept
Decline
Save