Common task - loading files into the database not so simple as you might think
When we thought about introducing this functionality we decided to use the file system as a data source or target.
For example, File system as a data source provides the following fields
- [Directory]
- [File Name]
- [Size]
- [Creation Date]
- [Read Only]
- [Hidden]
- [Archive]
- [File Data]
The file system as a data target uses just two fields
- [File Name]
- [File Data]
The next task actually loading files data into blob fields. We had to perform an incredible amount of testing.
For example, data can be loaded into Oracle Database in 4 ways
- Via ODBC,
- Via OLE DB,
- Via Direct path loading
- And Conventional path loading
Plus there are different versions of ODBC drivers and OLE DB Providers
And of course different versions of the Oracle database.
And Various data types Raw, Long Raw, CLOB, NCLOB, Blob
and it has to be repeated for every database type our ETL software supports
Once the file is loaded into the blob field we export it back into a different directory and compare files using MD5
Here is the package we used for testing


We started with small files then we moved to large ones, at some stage we were loading setups of our ETL software into the database.
The largest file we loaded was about 100 Meg.
All I can say forget about programming, just use "Advanced ETL Processor".
Mark Brown
MIS Consultant