[Showcase] Using Variables in Transformations - Validator and Transformation

More
1 year 10 months ago #24291 by prashant
HI,
I made a small project with using variables in Transformations. SEE LOOM

Features
1. Reads a CSV file
2. Uses loop to process them 1 by 1
3. Downloads each file using HTTP Get action
4. Uploads the file to Gdrive, outputs the Gdrive ID
5. Runs the transformation which uses validator to 'find' unique id , updates the gdrive value in csv
6. Update the CSV files

Overall fun little project

1. Question, if a file cannot be downloaded or if a file cannot be uploaded to gdrive, what's the best way to write this to another log or text file , using package - script or using transformation ? 
 

Please Log in or Create an account to join the conversation.

More
1 year 10 months ago #24292 by prashant
Another question , if I wish to create an empty text file (csv) but only with headers, is script the only way ?

Please Log in or Create an account to join the conversation.

More
1 year 10 months ago #24295 by admin
Please use package - script action

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
1 year 10 months ago #24296 by prashant
Here's the python script , where I have hardcoded the location of the file with the headers I needed , this can be easily modified for others. 
Code:
import etltools import csv import os csv_file_path = r'C:\Users\Superadmin\Documents\KBIAutomation\asp_upload\gdrive_asp.csv' headers = ['#rid#', '#qb_url#', '#filename#', '#gdrive#'] # Main function to create CSV with headers and a new line for appending data def main():     error_message = ""  # Initialize error message     try:         # Create or overwrite the CSV file and write the headers         with open(csv_file_path, 'w', newline='', encoding='utf-8') as outfile:             # Ensure all fields are enclosed in double quotes             writer = csv.writer(outfile, quoting=csv.QUOTE_ALL)             writer.writerow(headers)  # Write the headers            # Set result value to indicate success         Result.Value = 1     except Exception as e:         # Append all errors to error_message and set in ETL tools         error_message += str(e)         etltools.SetVariable('error_message', error_message)         Result.Value = 0 # Execute main function main()
The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

More
1 year 10 months ago #24308 by Peter.Jonson
Here is much shorter version using Pascal Script

var FileName,Header: String;
begin
 FileName := 'C:\Support\gdrive_asp.csv';
 Header := '"#rid#","#qb_url#","#filename#","#gdrive#"';
 StringToFile(FileName,Header);
 Result:=true;
end;

Peter Jonson
ETL Developer
The following user(s) said Thank You: prashant

Please Log in or Create an account to join the conversation.

More
1 year 10 months ago - 1 year 10 months ago #24312 by prashant
Being jealous of pascal , I asked chatgpt to come up with a shorter script too

Code:
import etltools,csv csv_file_path = r'C:\Users\Superadmin\Documents\KBIAutomation\asp_upload\gdrive_asp.csv' headers = ['#rid#', '#qb_url#', '#filename#', '#gdrive#'] with open(csv_file_path, 'w', newline='', encoding='utf-8') as outfile:     writer = csv.writer(outfile, quoting=csv.QUOTE_ALL)     writer.writerow(headers) Result.Value = 1
Last edit: 1 year 10 months ago by prashant.

Please Log in or Create an account to join the conversation.

Cookies user preferences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Marketing
Set of techniques which have for object the commercial strategy and in particular the market study.
Google
Accept
Decline
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline
Google Analytics
Accept
Decline
Functional
Tools used to give you more features when navigating on the website, this can include social sharing.
Advertisement
If you accept, the ads on the page will be adapted to your preferences.
Google Ad
Accept
Decline
Save