How to normalize Cyrillic look alike characters in file name and XML content

More
4 months 3 weeks ago #26005 by downing
Hi,I’m looking for a solution to handle Cyrillic look alike characters in an ETL process.We receive two files per document:
  • one PDF
  • one XML
The issue:
  • both file names may contain Cyrillic characters that look like Latin ones (e.g.
    Code:
    вто
    instead of
    Code:
    BTO
    )
  • in the XML file, the same Cyrillic characters also appear inside the XML content
Although everything looks correct visually, this breaks matching, routing and validations.I’m looking for a clean approach to:
  1. Normalize the file names of both PDF and XML files
  2. Normalize the XML content by replacing Cyrillic look alike characters with Latin ones, before further processing

Files look like:
pinv_1010_560279_вто.-2600048.pdf
pinv_1010_560279_вто.-2600048.xml

XML Content:
<Invoice>
    <OrgId>1edacd05-b0cf-52c0-a7c4-a53283485675</OrgId>
    <DocumentName>pinv_1010_560279_ВТО.-2600048.pdf</DocumentName>
    <SourceType>EMAIL</SourceType>
    <LastChangeDate>2026-02-18</LastChangeDate>
    <LastChangeDateTime>2026-02-18T11:13:34</LastChangeDateTime>
    <InvoiceNum>ВТО.-2600048</InvoiceNum>
    <InvoiceDate>2026-02-04</InvoiceDate>
    <PostingDate>2026-02-18</PostingDate>
...

Thx for your Ideas
Thomas
 
The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

More
4 months 3 weeks ago #26006 by admin
This is rather easy.

Please watch the second video on this page.
www.etl-tools.com/wiki/advanced-etl-proc...sources/file-system/

All you need to do is to use a different Replace function called Replace Characters.
www.etl-tools.com/wiki/advanced-etl-proc.../replace-characters/

 

Mike
ETL Architect
The following user(s) said Thank You: downing

Please Log in or Create an account to join the conversation.

More
4 months 3 weeks ago - 4 months 3 weeks ago #26007 by downing
A great piece of software.

Thank you, Mike.
Last edit: 4 months 3 weeks ago by downing. Reason: bad format

Please Log in or Create an account to join the conversation.

More
4 months 3 weeks ago #26008 by downing
Hello Mike,I’d like to add a filter for the file extension .xml in the [File Data] step of the transformer.It has already corrupted two PDF files.
Do you think this can happen, or was it just a coincidence?Thomas
The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

More
4 months 3 weeks ago #26009 by admin
Good Morning

I think in your case you need two transformations

1 XML

Replace Cyrillic characters inside the file
Replace Cyrillic characters inside the file name

2 PDF
 
Replace Cyrillic characters inside the file name only

Replacing inside the PDF file content may corrupt the file

Please keep us posted on your progress

 

Attachment reader.png not found


Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
4 months 3 weeks ago #26010 by downing
Hello Mike,
Funny thing: when I reopen the Replace Character function in the transformation after closing it,
it shows BTOlace Chara instead of BTO.Does this have any impact?

 

Please Log in or Create an account to join the conversation.

Cookies user preferences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Marketing
Set of techniques which have for object the commercial strategy and in particular the market study.
Google
Accept
Decline
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline
Google Analytics
Accept
Decline
Functional
Tools used to give you more features when navigating on the website, this can include social sharing.
Advertisement
If you accept, the ads on the page will be adapted to your preferences.
Google Ad
Accept
Decline
Save