Automation of Daily CSV File Import Process

More
11 months 3 weeks ago #25436 by TARUNTHAKUR
As per our business requirement, we receive daily CSV files for data import into our database. We would like to automate the process with the following steps:
  1. Monitor a designated input folder for CSV files in the format 
    Code:
    upload_dd-mm-yyyy.csv
    .
  2. Upload and import the data from each file into the database.
  3. Upon successful import, move the file to the "successful" folder. If the import fails, move the file to the "failed" folder.
  4. Maintain a daily log file in the format 
    Code:
    log_dd-mm-yyyy.txt
     to record the status of each file processed (success or failure), along with timestamps and error details if any.
We request support in creating a scheduled job or automation script to handle this process efficiently.
The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

More
11 months 3 weeks ago #25437 by admin
We will create an example for you, It should not not take log.

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
11 months 3 weeks ago - 11 months 3 weeks ago #25438 by admin
I am struggling to understand why do you need the log?
(Everything is logged by AETL itself already)

Maintain a daily log file in the format log_dd-mm-yyyy.txt

Mike
ETL Architect
Last edit: 11 months 3 weeks ago by admin.

Please Log in or Create an account to join the conversation.

More
11 months 3 weeks ago #25442 by TARUNTHAKUR
If a separate log is available for the job, then it has worked.
The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

More
11 months 3 weeks ago #25444 by admin
Please read the following WIKI article and watch video tutorial. It was created especially for you

www.etl-tools.com/wiki/advanced-etl-proc...s/directory-monitor/

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
11 months 3 weeks ago #25445 by admin
Please see repository backup attached


 

Mike
ETL Architect
Attachments:

Please Log in or Create an account to join the conversation.

Cookies user preferences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Marketing
Set of techniques which have for object the commercial strategy and in particular the market study.
Google
Accept
Decline
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline
Google Analytics
Accept
Decline
Functional
Tools used to give you more features when navigating on the website, this can include social sharing.
Advertisement
If you accept, the ads on the page will be adapted to your preferences.
Google Ad
Accept
Decline
Save