clean up log files

More
7 years 9 months ago #14098 by thatisfun
clean up log files was created by thatisfun
Our ETL is in Production for a while, it generate a lot of logs. Like several Millions of log files.
We found we cannot create system backup if we don't delete logs from monitor tab. it just frozen the system if we create Repository backup.
Here is the question:

1. How to create Repository backup, if we have a lot of logs in monitor?
2. Does ETL has any tool or feature to clean old logs?

Thanks

Please Log in or Create an account to join the conversation.

More
7 years 9 months ago #14099 by Peter.Jonson
Replied by Peter.Jonson on topic clean up log files
What is your repository type?

Peter Jonson
ETL Developer

Please Log in or Create an account to join the conversation.

More
7 years 9 months ago #14100 by thatisfun
Replied by thatisfun on topic clean up log files
MS SQL Server

Please Log in or Create an account to join the conversation.

More
7 years 9 months ago - 7 years 9 months ago #14101 by Peter.Jonson
Replied by Peter.Jonson on topic clean up log files
You can use MS SQL Server backups as alternative
If you want to clear logs you can delete it by runnig script

delete from QUEUE_HISTORY
delete from QUEUE_ACTIONS_HISTORY
or
truncate table QUEUE_HISTORY
truncate table QUEUE_ACTIONS_HISTORY

www.etl-tools.com/wiki/knowledgebase:repository_tables

Peter Jonson
ETL Developer
Last edit: 7 years 9 months ago by Peter.Jonson.

Please Log in or Create an account to join the conversation.

More
7 years 9 months ago #14102 by thatisfun
Replied by thatisfun on topic clean up log files
What about the several million log files under C:\ETL\logs folder?
What about we want to keep a backup of current logs for a while for trouble shooting?

Do we have the stop ETL Scheduler . before we do backup? running live database backup scares me.(it is usually for Disaster recovery )
Does ETL have best practice for log management? when do clean it. how to clean it. how to back it up,
Repository Backup looks like just Don't do the job, when logs are big.

Thanks

Please Log in or Create an account to join the conversation.

More
7 years 9 months ago #14104 by John
Replied by John on topic clean up log files
If you have million of logs files you probably have a lot of data in QUEUE_HISTORY, QUEUE_ACTIONS_HISTORY tables
If you delete data from them your repository will become much smaller

Regarding the log file you can always move them into different folder using file operation action.
For example you can move files older than 10 days and compress them

Attachments:

Please Log in or Create an account to join the conversation.