Stripping Unicode

More
4 years 7 months ago #21467 by bruce.gibbins
AETLP x32 v6.3.7.30

Hi

We are getting a file that has a Unicode character in it which I need to strip (or replace with blanks). I have attached some screen shots of me trying a RegEx that works on a RegEx Testing site I use but it doesn;t seem to give the desire result in AETL.

Any ideas?

I could go the other way and specify the characters I need to keep. But thought it would be good to see if I can do itt hrough exclusion first.


Attachments:

Please Log in or Create an account to join the conversation.

More
4 years 7 months ago #21468 by admin
Replied by admin on topic Stripping Unicode
Can you attach a file example please

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
4 years 7 months ago - 4 years 7 months ago #21469 by bruce.gibbins
Replied by bruce.gibbins on topic Stripping Unicode
Hi.

I have attached a sample csv file and an AETL .ATS file as well to show the transformation elements I have tried including Replace RegEx

Without going into specifics (and back story) the value that is coming through is actually a Bullet Character which are popular in PDF files.

The easy workaround is to use the Keep Characters Transformer and specify all of the ASCII Printable characters. this is easy and reusable.

One of the RegEx Replace Transformation Actions gives an odd result in that the expression [\x00-\x1F]|[\x7F-\xFF] just about does what I need except that one portion of the input string is not 'replaced'.

Attachments:
Last edit: 4 years 7 months ago by bruce.gibbins.

Please Log in or Create an account to join the conversation.

More
4 years 7 months ago #21472 by admin
Replied by admin on topic Stripping Unicode
Open data reader properties and set source file encoding to UTF8
Then use the delete characters function to remove •
That would be the fastest option


Mike
ETL Architect
Attachments:

Please Log in or Create an account to join the conversation.

More
4 years 7 months ago #21475 by bruce.gibbins
Replied by bruce.gibbins on topic Stripping Unicode
Thanks.

Yep that would be the quickest and most intuitive. But how did you get the Bullet point in the text box?

I ended up copying from your post via the browser dev tools

cheers

Please Log in or Create an account to join the conversation.

More
4 years 7 months ago #21478 by admin
Replied by admin on topic Stripping Unicode
I used notepad ++. I think should be also possible to copy it from data reader grid

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

Cookies user preferences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Marketing
Set of techniques which have for object the commercial strategy and in particular the market study.
Google
Accept
Decline
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline
Google Analytics
Accept
Decline
Functional
Tools used to give you more features when navigating on the website, this can include social sharing.
Advertisement
If you accept, the ads on the page will be adapted to your preferences.
Google Ad
Accept
Decline
Save