URL Logger how to trigger ETL

More
2 years 7 months ago #23145 by admin
Replied by admin on topic URL Logger how to trigger ETL
Thank you for the clarification

I assume that you want to insert the data into your own database or you still want to create CSV files?

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
2 years 7 months ago #23146 by prashant
I would prefer options
1. Run ETL with entry from URL-Logger
2. Add to database of choice (but I was under impression I can already define this from connection type)
3. Maybe run python on the the received data , I know I can run the same with ETL too but options are nice to have.
4. Create CSV file with ETL

Please Log in or Create an account to join the conversation.

More
2 years 7 months ago #23148 by prashant
An actual workflow

URL logger get a incoming request
Requested forwarded to chatGPT for summarisation
Summarised request sent to CRM
The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

More
2 years 7 months ago - 2 years 7 months ago #23195 by Peter.Jonson
As requested we have added support for custom script execution



This script will execute aetl package 
const execFileSync = require('child_process').execFileSync;
try {
// This script might hung so always specify a timeout
const fileToExecute = '"C:\\Program Files (x86)\\DB Software Laboratory\\Advanced ETL Processor Enterprise\\aetlcl.exe"'
const objectId = 6687;
const parameters = [objectId];
const timeout = 60; //seconds
const stdout = execFileSync(fileToExecute, parameters, { shell: true, timeout: timeout*1000 });
} catch (e) {
logger.error(e.message);
return 500;
}
return 200;

PS there is no need need to uninstall the url logger, just install new version on top

Peter Jonson
ETL Developer
Last edit: 2 years 7 months ago by Peter.Jonson.

Please Log in or Create an account to join the conversation.

More
2 years 7 months ago - 2 years 7 months ago #23200 by prashant

As requested we have added support for custom script execution


 
AWESOME, this saves me the step of writing a SQL trigger . If parameter passed here ObjectId here which is used the run the ETL command line from WIKI,


How do I access the JSON object
1. in ETL ,by asking it read from the sql table new object created ? 
2. in external program say non-ETL , python script , by asking it to read from the sql table ?

In both case , JSON object or the incoming request is not being passed to the ETL or program 
Last edit: 2 years 7 months ago by prashant.

Please Log in or Create an account to join the conversation.

More
2 years 7 months ago #23203 by Peter.Jonson
You can pass logId to aetl or python program 
Than you can use it to read record from the sql table

FYI: This script will print all objects available.

logger.debug(logId);
logger.debug(method);
logger.debug(originalUrl);
logger.debug(JSON.stringify(body, null, 4));
logger.debug(JSON.stringify(headers, null, 4));

return 200;

Peter Jonson
ETL Developer

Please Log in or Create an account to join the conversation.

Cookies user preferences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Marketing
Set of techniques which have for object the commercial strategy and in particular the market study.
Google
Accept
Decline
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline
Google Analytics
Accept
Decline
Functional
Tools used to give you more features when navigating on the website, this can include social sharing.
Advertisement
If you accept, the ads on the page will be adapted to your preferences.
Google Ad
Accept
Decline
Save