URL Logger how to trigger ETL

prashant
Topic Author
Offline
Platinum Member

More

2 years 7 months ago #23205 by prashant

Replied by prashant on topic URL Logger how to trigger ETL

This is great , I will update you once I get Logger up and running.

these examples would be great on WIKI page too . I take it upon myself to provide you a working example post of python as a big fat thanks.

The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

admin
Offline
Administrator
Deeds not Words

More

2 years 7 months ago #23206 by admin

Replied by admin on topic URL Logger how to trigger ETL

these examples would be great on WIKI page too . I take it upon myself to provide you a working example post of python as a big fat thanks.

Please share your scrips here and we will add them to the wiki

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

prashant
Topic Author
Offline
Platinum Member

More

2 years 7 months ago #23214 by prashant

Replied by prashant on topic URL Logger how to trigger ETL

When I run the script

Code:

logger.debug(logId);
logger.debug(method);
logger.debug(originalUrl);
logger.debug(JSON.stringify(body, null, 4));
logger.debug(JSON.stringify(headers, null, 4));

return 200;

URL Logger window gives me

Code:

2024-02-21 06:12:11.168 error:  logId is not defined 2024-02-21 06:12:11.170 info:   ::ffff:122.169.111.227 - e2d82769-f75c-4c6b-b4d1-d4eea8b9898a 500 POST /webhooks/chatgpt - 8.967 2024-02-21 06:12:18.701 error:  logId is not defined 2024-02-21 06:12:18.703 info:   ::ffff:122.169.111.227 - 85e23046-e663-48b2-a84e-be60fe56e8af 500 POST /webhooks/chatgpt - 6.590

---

My understanding now , is I can pass the 'logid' of the webhook to either AETL or to python , I can now then lookup the relevant record in mysql(currently what I am using)
The body or the json is actually not passed to the script , even if I use logger.debug(JSON.stringify(body, null, 4));

-

It might be too much , Can I request you for a working example for both AETL & python . It will really help clarify the usage here.

Assuming my python exists at
c:\python\python.exe
my python script at c:\python\hello.py

Please Log in or Create an account to join the conversation.

Peter.Jonson
Offline
Administrator

More

2 years 6 months ago #23221 by Peter.Jonson

Replied by Peter.Jonson on topic URL Logger how to trigger ETL

Good morning

1 Regarding logId error, I am sorry I forgot to commit my code, please install the latest version.
2 Regarding missing body, it only works with post method, I did test it with postman and it worked fine.
3 I will create two examples for you today, one for aetl another for python

Peter Jonson
ETL Developer

Please Log in or Create an account to join the conversation.

prashant
Topic Author
Offline
Platinum Member

More

2 years 6 months ago - 2 years 6 months ago #23224 by prashant

Replied by prashant on topic URL Logger how to trigger ETL

HI,

1. logID works now . Thanks
2. I am using POST only
3. Will appreciate the scripts.
4. My question still is - can the body be provided directly to either ETL or Python or only logID and then we make a search in mySQl to find relevant entry

I also tried my luck at modifying scripts with chatGPT but URL logger code pages refuses to 'save' this , no errors are given , well you can see the loom video

www.loom.com/share/e34c0173315a4bdc8e23c...04-8995-ee8b50d811fb

Code:

const { execFileSync } = require('child_process');

const pythonInterpreter = 'C:\\Users\\Superadmin\\AppData\\Local\\Programs\\Python\\Python312\\python.exe';
const pythonScript = 'C:\\Users\\Superadmin\\Documents\\KBIAutomation\\pythonProject\\process_webhook.py';
const parameters = [pythonScript, logId];

execFileSync(pythonInterpreter, parameters);

return 200;

Code:

const { execFileSync } = require('child_process');

try {
  const pythonInterpreter = 'C:\\Users\\Superadmin\\AppData\\Local\\Programs\\Python\\Python312\\python.exe';
  const pythonScript = 'C:\\Users\\Superadmin\\Documents\\KBIAutomation\\pythonProject\\process_webhook.py';
  const parameters = [pythonScript, logId.toString()];
  
  const stdout = execFileSync(pythonInterpreter, parameters);
  console.log(stdout.toString());
} catch (e) {
  // Error handling or return removed
}

return 200;

Code:

const { execFileSync } = require('child_process');

try {
  const pythonInterpreter = "C:\\Users\\Superadmin\\AppData\\Local\\Programs\\Python\\Python312\\python.exe";
  const pythonScript = "C:\\Users\\Superadmin\\Documents\\KBIAutomation\\pythonProject\\process_webhook.py";
  const parameters = [pythonScript, logId];

  execFileSync(pythonInterpreter, parameters);
  return 200;
} catch (e) {
  // Handle or log the error as needed
  console.error(e); // This line is for demonstration; replace or remove it as needed.
  return 500; // Indicate an error occurred
}

Last edit: 2 years 6 months ago by prashant.

Please Log in or Create an account to join the conversation.

Peter.Jonson
Offline
Administrator

More

2 years 6 months ago #23225 by Peter.Jonson

Replied by Peter.Jonson on topic URL Logger how to trigger ETL

My question still is - can the body be provided directly to either ETL or Python or only logID and then we make a search in mySQl to find relevant entry

The only way is to use command line but for very complex json it will most likely fail.

So logId is the only option

Peter Jonson
ETL Developer

Please Log in or Create an account to join the conversation.

ETL Forum