URL Logger hangs

More
2 years 2 months ago #23906 by prashant
URL Logger hangs was created by prashant
Hellos,
I m noticing a constant behaviour while I am testing that URL logger (when running form EXE file) , after 8-10 messages simply hangs , the webpage refuses to refresh , until I quit and restart the exe file 

Do I email log files ? or shoot a loom video to show this behaviour ? 

Please Log in or Create an account to join the conversation.

More
2 years 2 months ago #23907 by Peter.Jonson
Replied by Peter.Jonson on topic URL Logger hangs
URL logger was designed to write HTTP request into the database as fast as possible

Then another process reads the data and processes it. For example AETL

That way they do not block each other.

Since you are using scripts
URL logger will not respond until script execution is completed. 
it is possible to write a script which will run forever and  it will hang url logger.
So you have to be careful.

Another option is not to block it and do everything in parallel .
That means that you might have 1000's of python.exe's running on a single computer.
This will bring down the server    

Peter Jonson
ETL Developer

Please Log in or Create an account to join the conversation.

More
2 years 2 months ago #23908 by prashant
Replied by prashant on topic URL Logger hangs
Hi,

My idea was to do this in parallel , let the script handoff details to python without waiting for it to finish. We will implement some queuing mechanism in python to ensure multiple python.exe will not run in parallel.

Will this ensure that URL - Logger is stable ? 
 

Please Log in or Create an account to join the conversation.

More
2 years 2 months ago - 2 years 2 months ago #23909 by admin
Replied by admin on topic URL Logger hangs
What exactly is your python code doing?

May be there is another better way of handling it

Mike
ETL Architect
Last edit: 2 years 2 months ago by admin.

Please Log in or Create an account to join the conversation.

More
2 years 2 months ago - 2 years 2 months ago #23914 by admin
Replied by admin on topic URL Logger hangs
Working with web hooks is a battle for milliseconds.

Since you do not want to wait until python script completion it is better to use spawn function with detached parameter.  

const fs = require('fs');
const { spawn } = require('child_process');
try {
    const WORKING_DIRECTORY = 'C:\\Temp\\';
    // Path to the JSON file
    const jsonFilePath = `${WORKING_DIRECTORY}${logId}.json`;
    const jsonString = JSON.stringify(body, null, 4);
    // Write JSON data to a file
    fs.writeFileSync(jsonFilePath, jsonString, 'utf8');
    logger.info(`File written successfully to: ${jsonFilePath}`);
    // Python executable and parameters
    const pythonExecutablePath = `python`;  // Use 'python' from PATH
    const parameters = ["test.py",jsonFilePath];
    // Execute the Python script without waiting for completion within specified working directory
    spawn(pythonExecutablePath, parameters, { cwd: WORKING_DIRECTORY,detached: true })
    return 200;
} catch (err) {
    logger.error(`Caught an error: ${err.message ? err.message : err}`);
    return 500; // Any error caught here results in a 500 status
}

Mike
ETL Architect
Last edit: 2 years 2 months ago by admin.

Please Log in or Create an account to join the conversation.

More
2 years 2 months ago #23915 by admin
Replied by admin on topic URL Logger hangs
Puthin script I've used for testing

import sys
f = open("args.txt", "w")
f.write(sys.argv[1])
f.close()

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

Cookies user preferences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Marketing
Set of techniques which have for object the commercial strategy and in particular the market study.
Google
Accept
Decline
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline
Google Analytics
Accept
Decline
Functional
Tools used to give you more features when navigating on the website, this can include social sharing.
Advertisement
If you accept, the ads on the page will be adapted to your preferences.
Google Ad
Accept
Decline
Save