Processing JSON from URL Logger

prashant
Topic Author
Offline
Platinum Member

More

2 years 1 month ago #23810 by prashant

Processing JSON from URL Logger was created by prashant

This is a question regarding performance

I have created a URL which receives Webhooks from telegram always

When processing all I want is
username
last file_id
last file_unique_id

Should this tinkering be done in CODE section to extract relevant JSON file and write it , or this processing should be done outside of URL Logger (in Advance ETL or python)

Code:

{   "update_id": 744444728,   "message": {     "message_id": 13,     "from": {       "id": 85449808,       "is_bot": false,       "first_name": "Prashant",       "last_name": "M",       "username": "Prashant_M",       "language_code": "en"     },     "chat": {       "id": -421638518,       "title": "pypy",       "type": "group",       "all_members_are_administrators": true     },     "date": 1716735693,     "photo": [       {         "file_id": "AgACAgUAAxkBAAMNZlNOzeptxSS54JIkcWX2O8EhWIQAAqy_MRtbMphWIoj6SdRjvH8BAAMCAANzAAM1BA",         "file_unique_id": "AQADrL8xG1symFZ4",         "file_size": 998,         "width": 90,         "height": 48       },       {         "file_id": "AgACAgUAAxkBAAMNZlNOzeptxSS54JIkcWX2O8EhWIQAAqy_MRtbMphWIoj6SdRjvH8BAAMCAANtAAM1BA",         "file_unique_id": "AQADrL8xG1symFZy",         "file_size": 11420,         "width": 320,         "height": 170       },       {         "file_id": "AgACAgUAAxkBAAMNZlNOzeptxSS54JIkcWX2O8EhWIQAAqy_MRtbMphWIoj6SdRjvH8BAAMCAAN4AAM1BA",         "file_unique_id": "AQADrL8xG1symFZ9",         "file_size": 12412,         "width": 398,         "height": 211       }     ],     "caption": "This is my caption on image"   } }

The following user(s) said Thank You: Peter.Jonson

Please Log in or Create an account to join the conversation.

Peter.Jonson
Offline
Administrator

More

2 years 1 month ago #23811 by Peter.Jonson

Replied by Peter.Jonson on topic Processing JSON from URL Logger

In my opinion it is better to do it in CODE, sending large amount of data between applications will reduce performance.

Peter Jonson
ETL Developer

Please Log in or Create an account to join the conversation.

prashant
Topic Author
Offline
Platinum Member

More

2 years 1 month ago #23812 by prashant

Replied by prashant on topic Processing JSON from URL Logger

Hellos,
Few questions, for my non programming brain for code here

Is this Javascript or is this Node.js ?
When there are errors in code , how do I know about them ?

---

Chatgpt with some prodding gave me a fairly complex code for me to extract text or photos present in telegram message , hope this is useful for all.

This code now extracts username and
if text message - the text
if photos message - the file_id of the photo for downloading it & caption

Code:

const fs = require('fs');
try {
// Dynamic file name due to use of logid, we can also timestamp if needed
const jsonFilePath = `C:\\Users\\Superadmin\\Downloads\\director\\${logId}.json`;

const username = body.message.from.username;
const caption = body.message.caption || body.message.text || ''; // Use caption if available, otherwise use text, default to empty string
let lastFileId = null;
let lastFileUniqueId = null;
if (body.message.photo && body.message.photo.length > 0) {
    const lastPhoto = body.message.photo[body.message.photo.length - 1]; // Get the last photo object
    lastFileId = lastPhoto.file_id;
    lastFileUniqueId = lastPhoto.file_unique_id;
    }

// Prepare the extracted data
const extractedData = {
username: username,
caption: caption
};
if (lastFileId && lastFileUniqueId) {
  extractedData.last_file_id = lastFileId;
  extractedData.last_file_unique_id = lastFileUniqueId;
  }
const jsonString = JSON.stringify(extractedData, null, 4);
fs.writeFileSync(jsonFilePath, jsonString, 'utf8');

return 200;
} catch (err) {
logger.error(`${err.message ? err.message : err}`)
return 500;
}

Please Log in or Create an account to join the conversation.

prashant
Topic Author
Offline
Platinum Member

More

2 years 1 month ago #23813 by prashant

Replied by prashant on topic Processing JSON from URL Logger

Team ETL,

I propose we should release this as a guide for showcasing the power of URL Logger , how to get telegram for incoming message / photos etc & then processing it in ETL if needed. I think it would be highly useful for users

The following user(s) said Thank You: Peter.Jonson

Please Log in or Create an account to join the conversation.

prashant
Topic Author
Offline
Platinum Member

More

2 years 1 month ago #23814 by prashant

Replied by prashant on topic Processing JSON from URL Logger

OK , I think I got all variables now including text, photos,video,audio and files. For any explanation what I've done , we need to ask chatgpt ,

This works when you've created a bot and invited to a group .

Code:

const fs = require('fs');
try {
// Dynamic file name due to use of logid, we can also timestamp if needed
const jsonFilePath = `C:\\Users\\Superadmin\\Downloads\\director\\${logId}.json`;
// Convert the `body` object to a formatted JSON string

const username = body.message.from.username;
const caption = body.message.caption || body.message.text || ''; // Use caption if available, otherwise use text, default to empty string
let lastFileId = null;
let lastFileUniqueId = null;
if (body.message.photo && body.message.photo.length > 0) {
        const lastPhoto = body.message.photo[body.message.photo.length - 1]; // Get the last photo object
        lastFileId = lastPhoto.file_id;
        lastFileUniqueId = lastPhoto.file_unique_id;
    } else if (body.message.video_note) {
        const videoNote = body.message.video_note; // Get the video_note object
        lastFileId = videoNote.file_id;
        lastFileUniqueId = videoNote.file_unique_id;
    } else if (body.message.voice) {
        const voice = body.message.voice; // Get the voice object
        lastFileId = voice.file_id;
        lastFileUniqueId = voice.file_unique_id;
    } else if (body.message.document) {
        const document = body.message.document; // Get the document object
        lastFileId = document.file_id;
        lastFileUniqueId = document.file_unique_id;
    }

// Prepare the extracted data
const extractedData = {
username: username,
caption: caption
};
if (lastFileId && lastFileUniqueId) {
  extractedData.last_file_id = lastFileId;
  extractedData.last_file_unique_id = lastFileUniqueId;
  }
const jsonString = JSON.stringify(extractedData, null, 4);
fs.writeFileSync(jsonFilePath, jsonString, 'utf8');

return 200;
} catch (err) {
logger.error(`${err.message ? err.message : err}`)
return 500;
}

The following user(s) said Thank You: Peter.Jonson

Please Log in or Create an account to join the conversation.

Peter.Jonson
Offline
Administrator

More

2 years 1 month ago #23815 by Peter.Jonson

Replied by Peter.Jonson on topic Processing JSON from URL Logger

Looks good

Peter Jonson
ETL Developer

The following user(s) said Thank You: prashant

Please Log in or Create an account to join the conversation.

ETL Forum