Data Warehousing and Data integration

ETL stops responding when reading large xml files

More
5 years 1 month ago - 5 years 1 month ago #21259 by dickels112
XML file of 70 mb continuously fails to load (we even left it alone for over 1,5 hours).
Data restriction load does not work on loading the file itself.
Is there anything we are missing?

Example of the xml file is stored here: we.tl/t-KAd7dClky0https://we.tl/t-KAd7dClky0

For transposing the xml we use this transform script:
Code:
<?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <!--Specify over which table to loop--> <xsl:template match="DFMS_DiamondFMSToDataConnectCompanyValuesExport"> <DFMS_VariableValues> <!--Specify over which record set to loop over--> <xsl:for-each select="DFMS_VariableValues/DFMS_VariableValue"> <DFMS_VariableValue> <!--Specify over which attributes of the record to map to which fields--> <DFMS_PeriodDimensionID><xsl:value-of select="@DFMS_PeriodDimensionID"/></DFMS_PeriodDimensionID> <DFMS_PeriodTypeID><xsl:value-of select="@DFMS_PeriodTypeID"/></DFMS_PeriodTypeID> <DFMS_PeriodTypeValue><xsl:value-of select="@DFMS_PeriodTypeValue"/></DFMS_PeriodTypeValue> <DFMS_PeriodYear><xsl:value-of select="@DFMS_PeriodYear"/></DFMS_PeriodYear> <DFMS_PeriodValue><xsl:value-of select="@DFMS_PeriodValue"/></DFMS_PeriodValue> <DFMS_ValueNumeric><xsl:value-of select="@DFMS_ValueNumeric"/></DFMS_ValueNumeric> <DFMS_VariableID><xsl:value-of select="@DFMS_VariableID"/></DFMS_VariableID> <DFMS_VariableValueID><xsl:value-of select="@DFMS_VariableValueID"/></DFMS_VariableValueID> </DFMS_VariableValue> </xsl:for-each> </DFMS_VariableValues> </xsl:template> </xsl:stylesheet>
Last edit: 5 years 1 month ago by dickels112.

Please Log in or Create an account to join the conversation.

More
5 years 1 month ago #21260 by admin
1) I am unable to download the example.
2) We do not know which software you are using or version or edition or is it actually our software

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
5 years 1 month ago #21261 by dickels112
1) we.tl/t-KAd7dClky0
2) Advanced ETL Processor Enterprise (64 bit) version: 6.3.7.26

Please Log in or Create an account to join the conversation.

More
5 years 1 month ago #21262 by admin
What are you trying to do with your file?
Are you trying to load it into the database or convert it into another XML format?

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
5 years 1 month ago #21263 by dickels112
Hey Mike

As it is an XML where the records contain attributes, so I need to transform them.
The issue however is not related to this XSL transformation. When I try to read the XML (even without the XSL transformation) ETL freezes and after 1,5 hours I need to end the process in order to continue. This is not only on my machine.

Please Log in or Create an account to join the conversation.

More
5 years 1 month ago #21264 by admin
We have reproduced the problem and we need time to investigate the issue.

The reason for my question is if you loading data into the database you can just convert XML file into CSV and load it instead.
That would be a much faster option.

You can use the XLST for conversion.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl=" www.w3.org/1999/XSL/Transform ">
<xsl:output method="text" encoding="UTF-8"/>
<xsl:template match="DFMS_VariableValues">
<xsl:text>@DFMS_PeriodDimensionID,@DFMS_PeriodTypeID,@DFMS_PeriodTypeValue,@DFMS_PeriodYear,@DFMS_PeriodValue,@DFMS_ValueNumeric,@DFMS_VariableID,@DFMS_VariableValueID
</xsl:text>
<xsl:text>&#xA;</xsl:text>
<xsl:for-each select="DFMS_VariableValue">
<xsl:value-of select="@DFMS_PeriodDimensionID"/>
<xsl:text>,</xsl:text>
<xsl:value-of select="@DFMS_PeriodTypeID"/>
<xsl:text>,</xsl:text>
<xsl:value-of select="@DFMS_PeriodTypeValue"/>
<xsl:text>,</xsl:text>
<xsl:value-of select="@DFMS_PeriodYear"/>
<xsl:text>,</xsl:text>
<xsl:value-of select="@DFMS_PeriodValue"/>
<xsl:text>,</xsl:text>
<xsl:value-of select="@DFMS_ValueNumeric"/>
<xsl:text>,</xsl:text>
<xsl:value-of select="@DFMS_VariableID"/>
<xsl:text>,</xsl:text>
<xsl:value-of select="@DFMS_VariableValueID"/>
<xsl:text>&#xA;</xsl:text>
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>

On my computer, it takes 14 seconds to convert.
The CSV file is much smaller than the original XML (6Mb vs 36Mb)

Please also have a look at this example
www.etl-tools.com/automation/0010-how-to...-delimited-file.html

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

Cookies user preferences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Marketing
Set of techniques which have for object the commercial strategy and in particular the market study.
Google
Accept
Decline
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline
Google Analytics
Accept
Decline
Functional
Tools used to give you more features when navigating on the website, this can include social sharing.
Advertisement
If you accept, the ads on the page will be adapted to your preferences.
Google Ad
Accept
Decline
Save