Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1

TOPIC: Load files from etl for transformation

Load files from etl for transformation 1 month 3 weeks ago #19027

Hello


Currently I'm using the transformation tool on a local folder to transform a QVD file to MySQL table and it work ok, now I need to load that QVD file from an FTP/STP/SAMBA server so I did set up the connection but I don't see it listed on the connection type of the transformation object properties.

I did set up the FTP server as a windows connection but the ETL tools wan't able to access it that way.

So How can I use a FTP folder to load some files for transformation process?

Thanks

Please Log in or Create an account to join the conversation.

Load files from etl for transformation 1 month 3 weeks ago #19028

  • admin
  • admin's Avatar
  • Offline
  • Moderator
  • Moderator
  • Deeds not Words
  • Posts: 7510
  • Karma: 33
  • Thank you received: 409
Hello

So How can I use an FTP folder to load some files for the transformation process?

You download the files fist and then load them into the database
Mike

Please Log in or Create an account to join the conversation.

Load files from etl for transformation 1 month 3 weeks ago #19029

Holla


Hi Admin,

Thanks for reply, so whats the purpose of that FTP connection on the ETL tool?


I don't think to download the file and then load it into the application in a stand-alone process isn't practique so I will use a third party app to map the FTP server as a windows Drive unit with letter assigned and setup the ETL tools to load from that drive as a local folder...



Thanks

Please Log in or Create an account to join the conversation.

Load files from etl for transformation 1 month 3 weeks ago #19030

  • admin
  • admin's Avatar
  • Offline
  • Moderator
  • Moderator
  • Deeds not Words
  • Posts: 7510
  • Karma: 33
  • Thank you received: 409
The FTP connection is there to perform FTP actions
www.etl-tools.com/wiki/aetle/packages/ftp_action
You can also use to load a list of FTP files
www.etl-tools.com/wiki/aetle/processing_data/reader/ftp

Based on my personal experience I can say that FTP protocol is unreliable.
You may not agree with me however what if you are loading 10 million records and it fails in the middle?
All ETL tools I know download the files first then load them into the database.

Regarding using a third party tool to map FTP as a drive.
Please bear in mind that mapped drives do not work from windows services, therefore, you will not be able to use execution agent.

Please post screenshots if you have any further questions it is very difficult to provide support without them
Mike
The following user(s) said Thank You: montolla

Please Log in or Create an account to join the conversation.

Load files from etl for transformation 1 month 3 weeks ago #19032

Hello


Hi Admin,
thanks for your reply.
I'm Agree with you but that same server can serve as samba, sftp,ftp,ftpS file server so any of them can work well.
I'm using windows scheduler task to execute the task, since the only think it does is transform the QVD to MySQL, but then a php script must executes to take that mysql data, then split it into packages/lotes of 500 records each and send it using Json REST API CALL to a SugarCRM on a remote server.

Thanks

Please Log in or Create an account to join the conversation.

  • Page:
  • 1

We have 195 guests and 2 members online