Uploading of jpeg image files as Base64 field in json records

Hi there

We seem to have a problem with the processing of Base64 image data -
Currently we have been running trials to extract and send member data from a Sydney site to a Azure Cloud based system based in Europe - I believe Holland.

The data is uploaded into a database and used by a mobile phone app - accessed by members here in Australia.
All the non-image data fields upload fine. It is a small number of the upload records where the image data that has been converted to a Base64 string that displays with corrupted images.

The current upload data is assembled as an Excel spread sheet, with one of the fields (Column) being a converted Jpeg colour image file. The typical size of the covered file is 15K to 25K. For some images we have to resize the image and then used as a web based service to convert to a Base64 string - see following link -
www.base64decode.net/base64-image-decoder

Would you recommend a different Base64 tool?

In the attachments, the Word document page 1 contains a screen shot of the first and second screens displaying the first 20 records from an upload batch of 35 records.
In the attached zip file there are 4 x folders, 3 x Excel files and the ETL Pro Repository backup Ver:6.1.0.0

The 3 Excel files that are the ETL input Excel spread sheets match the 3 x folders that contain the ET Pro log files, Json input records that have been built by the ETL Transformer process.

The initial upload of 35 member details, including the member's jpeg image file converted to Base64, can be found in the folder ETL_Log_P2
The Excel input file is in the root folder Tjen_Elixr_1710231 named CMbo171014_PF2
- the Excel file has the member records highlighted that have image errors after upload

After the image corruption we ran individual uploads of the 2 x members with the single entry Excel spread sheets CMbo171021_ImageCorrectionPA & CMbo171023_ImageCorrectionCT

From the 2 x single member record upload files we took a copy of the ETL generated json file and copied the Bse64 image from the Excel spread sheet to see if there was a difference in Base64 character string -

For the json files 2T.json. and 2TRun2.json and 42.T.json

In the Word document after the on page two there is a screen shots for each of the json files with the extra Base64 data string showing where the files start to have a difference in the character strings -

For the file 2TRun2.json there is a second comparison string from a second upload run to confirm that error in the image Base64 string of the json file was the same character variation as seen in the first upload run comparison.

On the second page of the word document he there are three screen shots of the comparison to illustrate the use of the line of "====" characters to mark where the initial strings are the same.
At the bottom left corner is the character position in the string of the last matching character -
for the 2T.json record it is position 12,207 while the position in 42T.json it is 4,338

Let me know if there is any further information that would assist

Best Wishes

Tral

Hi Tral

Thank you for such a long and detailed post.
if you have a look at base 64 value for ID 42 you will see that F14 was replaced with 0

Now have a look at this picture.



Looks familiar is it not?

To avoid issues use long values for example use F14XXXXXXXXX instead of F14.

Hi Mike
Thank you for the quick response –
Your suggested change did resolve the image issue with the Base64 character string image errors.
I still do not fully understand why the need to have such a long name in the Replace map field.
Early in the project we did make a change to ensure that they were all the same length and that fixed some problems we had struck.

I also wanted to check on what is best practice for setting up the template for the json record with regard to the field names and their data component – it is to do with the use of a space between the field name and the “:” character and then to the actual data.
In the attached document there is a screen shot of the template where I have pointed to two different layouts that have been used. Is it best to have a space or not?

Regards

Tral

1) I still do not fully understand why the need to have such a long name in the Replace map field.

Here is an example

Values are replaced multiple times.

JSON
"Field1": "F01"
"Field2": "F02"

Values

F01= 10
F02= F010

Replacing values first time

"Field1": "10"
"Field2": "F010"

Replacing them again

"Field1": "10"
"Field2": "1010"

2) Regarding spaces this might be useful to you
stackoverflow.com/questions/4150621/are-...nsignificant-in-json