URL Logger how to trigger ETL

prashant
Topic Author
Offline
Platinum Member

More

2 years 6 months ago - 2 years 6 months ago #23243 by prashant

Replied by prashant on topic URL Logger how to trigger ETL

admin wrote:
Yes, text_field_01 can hold more than 255 characters

IT WORKS

I was able to fire a webhook and get a json file on execution !!! WOHOO

Next actionable - Test ETL script provided by you

Thank you for this generous update & all the back and forth

As Promised content of process_webhook.py'

Code:

import sys

# The path to the file where the input will be written
output_file_path = 'C:\\Users\\Superadmin\\Documents\\KBIAutomation\\abc.json'

# Check if at least one argument is provided (first argument is the script name)
if len(sys.argv) > 1:
    # Join all arguments (excluding the script name) into a single string
    input_str = " ".join(sys.argv[1:])
    print("Received input:", input_str)

    # Write the input string to the specified file
    with open(output_file_path, 'a') as file:  # 'a' appends to the file
        file.write(input_str + "\n")
else:
    print("No input provided")

SCRIPT INSIDE URL LOGGER

Code:

const { execFileSync } = require('child_process');

const pythonInterpreter = 'C:\\Users\\Superadmin\\AppData\\Local\\Programs\\Python\\Python312\\python.exe';
const pythonScript = 'C:\\Users\\Superadmin\\Documents\\KBIAutomation\\pythonProject\\process_webhook.py';
const parameters = [pythonScript, JSON.stringify(body, null, 4)];

execFileSync(pythonInterpreter, parameters);

return 200;

Last edit: 2 years 6 months ago by prashant.

Please Log in or Create an account to join the conversation.

admin
Offline
Administrator
Deeds not Words

More

2 years 6 months ago #23245 by admin

Replied by admin on topic URL Logger how to trigger ETL

Thank you for letting us know

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

admin
Offline
Administrator
Deeds not Words

More

2 years 6 months ago #23248 by admin

Replied by admin on topic URL Logger how to trigger ETL

BTW you can use java script to create files so there is no need to use python.
plus this will improve performance

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

prashant
Topic Author
Offline
Platinum Member

More

2 years 6 months ago #23259 by prashant

Replied by prashant on topic URL Logger how to trigger ETL

admin wrote:
BTW you can use java script to create files so there is no need to use python.
plus this will improve performance

Going to try and ask you many more questions in future

Please Log in or Create an account to join the conversation.

prashant
Topic Author
Offline
Platinum Member

More

2 years 6 months ago - 2 years 6 months ago #23262 by prashant

Replied by prashant on topic URL Logger how to trigger ETL

Hi Mike,
The Javascript idea was great. Thank you. with chatGPT n00bs like me can get kickstarted . I have tonnes of option which I wanted. I just need to incorporate a 500 message on this in case of error.

1. JSON file which can be passed to python,
2. JSON which can be polled by ETL

Code:

const fs = require('fs');

// Dynamic file name due to use of logid, we can also timestamp if needed
const logIdString = String(logId);
const jsonFilePath = `C:\\Users\\Superadmin\\Downloads\\director\\${logIdString}.json`;

// Convert the `body` object to a formatted JSON string
const jsonString = JSON.stringify(body, null, 4);

// Write the JSON content to a file synchronously
fs.writeFileSync(jsonFilePath, jsonString, 'utf8');

return 200;

Last edit: 2 years 6 months ago by prashant.

The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

admin
Offline
Administrator
Deeds not Words

More

2 years 6 months ago #23269 by admin

Replied by admin on topic URL Logger how to trigger ETL

I have tweaked your code slightly so it handles errors better

I think this forum thread is too log, Please start a new topic

const fs = require('fs');
try {
// Dynamic file name due to use of logid, we can also timestamp if needed
const jsonFilePath = `C:\\Users\\Superadmin\\Downloads\\director\\${logId}.json`;

// Convert the `body` object to a formatted JSON string
const jsonString = JSON.stringify(body, null, 4);

// Write the JSON content to a file synchronously
fs.writeFileSync(jsonFilePath, jsonString, 'utf8');
return 200;
} catch (err) {
logger.error(`${err.message ? err.message : err}`)
return 500;
}

Mike
ETL Architect

The following user(s) said Thank You: prashant

Please Log in or Create an account to join the conversation.

ETL Forum