The Problem:

My media server runs a rsync job via ssh to another server every night between 2230 and 0600. Every time the scripts runs, it generates a new time-stamped log file. Eventually there are quite a few log files that require manual cleanup. I want to automate this process and cleanup the log file generation.

The log files

The bash script is kept in my home directory (for easy, unscheduled syncs) and is executed using cron.

My crontab file contains:

30 22 * * * /home/wargus/rsync.sh >/dev/null 2>&1
0 6 * * * killall rsync >/dev/null 2>&1

Contents of script:

#!/bin/bash
rsync –bwlimit=450 –delete –protect-args –size-only –copy-dirlinks –log-file=/var/log/rsync/log.`date +”%Y%m%d_%H%M%S”` -avPe ssh “/path/to/files/” “user@host:/path/to/files/”

I won’t go into the details of the rsync command above, suffice to say it works and limits bandwidth to something reasonable for a slow, home ADSL connection. I expect that will change when NBN will finally (if ever) arrive at my off-site location. For this to work, I did have to generate ssh keys to allow the job to execute without user intervention.

The Solution:

The addition of two line lines to my rsync.sh above the rsync command script did the trick:

find /var/log/rsync/ -mtime +8 |xargs -I % sh -c ‘rm -f %’;
find /var/log/rsync/log.* |xargs -I % sh -c ‘tar -rf /var/log/rsync/rsync.1.tar %; rm -f %’;

The first line finds anything older than 8 days, then using the list output by find input, deletes all the files. On first run it deleted all my older log files, but going forward, it will remove the archive after 8 days.

The second line fill find every log file in that directory and appends it to an archive, if it exists, or creates the archive first if it does not.

Now, when the script executes, I have no problem knowing what the newest log is and in case I want to check older ones, I can open the archive and have a look.