[ale] Duplicity very slow

[Edward Holcroft]

Nice, thanks James. If I can get it to run faster, I may use this.

ed

On Tue, Apr 21, 2015 at 1:35 PM, James Sumners <james.sumners at gmail.com>
wrote:

> I use the script included in this email to run backups. It's really just a
> call to duplicity after some sanity checks. Run on a system that hasn't
> been backed up before, where that system is about 1.7GB of data, it takes
> about 8 minutes to run.
>
> ````
> #!/bin/bash
>
> # Pre-execution check for bsfl
> # Set options afterward
> if [ ! -f /etc/bsfl ]; then
>   echo "Backup script requires bsfl (https://code.google.com/p/bsfl/)."
>   exit 1
> fi
> source /etc/bsfl
>
> ### Options ###
>
> # Set to the desired logfile path and name
> LOG_FILE="$(dirname $0)/logs/runbackup-$(date +'%m-%d-%Y').log"
>
> # Set to the file that contains backup exclusions (format = line separated
> paths)
> EXCLUDES="$(dirname $0)/excludes"
>
> # Set to the NFS mount point
> # Be sure to configure /etc/fstab appropriately
> NFS_DIR="$(dirname $0)/remote"
>
> # Set to test string for testing NFS mount success
> NFS_MOUNT_TEST="^//bigspace.*/this_system_dn"
>
> # Set to the remote backup container directory
> # Backups will be stored in subdirectories of this directory
> BACKUP_DIR="${NFS_DIR}"
>
> # Set to the email address that will receive notifications
> # of backup failures
> ERROR_EMAIL_ADDR="admin at example.com"
>
> # Set to the list of switches you want passed to Duplicity
> # Add '--time-separator .' if using an old version of Duplicity
> DUPLICITY_SWITCHES="-v4 --no-encryption --exclude-globbing-filelist
> ${EXCLUDES} --exclude-other-filesystems --volsize 150 --full-if-older-than
> 30D"
>
>
> ### Begin actual script ###
>
> function notify {
>   mail -s "Backup failure on $(hostname)" ${ERROR_EMAIL_ADDR} < ${LOG_FILE}
> }
>
> # Turn on bsfl logging support
> LOG_ENABLED="yes"
>
> # We need to be root to 1) read all files and 2) mount the NFS
> USER=$(whoami)
> if [ "${USER}" != "root" ]; then
>   notify
>   die 2 "Backup must be run as root."
> fi
>
> log "Mounting NFS"
> mount ${NFS_DIR}
>
> NFS_MOUNTED=$(cat /proc/mounts | grep ${NFS_MOUNT_TEST})
> if [ ! $? -eq 0 ]; then
>   notify
>   umount ${NFS_DIR}
>   die 3 "Could not mount remote share."
> fi
>
> # Let's make sure we have enough room on the remote system
> STAT_INFO=$(stat -f --format='%b %a %S' ${NFS_DIR})
> TOTAL_BLOCKS=$(echo ${STAT_INFO} | awk '{print $1}')
> FREE_BLOCKS=$(echo ${STAT_INFO} | awk '{print $2}')
> BLOCK_SIZE=$(echo ${STAT_INFO} | awk '{print $3}')
> # (1024bytes * 1024kilobytes) / (x bytes) = (1 megabyte [in bytes]) / (x
> bytes)
> # => number of blocks in 1 megabyte = y
> REMOTE_FREE_BYTES=$(echo "${FREE_BLOCKS} / (1048576 / ${BLOCK_SIZE})" | bc
> -l)
> log "Remote free bytes = ${REMOTE_FREE_BYTES}"
>
> STAT_INFO=$(stat -f --format='%b %a %S' /)
> TOTAL_BLOCKS=$(echo ${STAT_INFO} | awk '{print $1}')
> FREE_BLOCKS=$(echo ${STAT_INFO} | awk '{print $2}')
> BLOCK_SIZE=$(echo ${STAT_INFO} | awk '{print $3}')
> LOCAL_USED_BYTES=$(echo "(${TOTAL_BLOCKS} - ${FREE_BLOCKS}) / (1048576 /
> ${BLOCK_SIZE})" | bc -l)
> log "Local used bytes = ${LOCAL_USED_BYTES}"
>
> REMOTE_HAS_ROOM=$(echo "${REMOTE_FREE_BYTES} > ${LOCAL_USED_BYTES}" | bc
> -l)
> if [ ${REMOTE_HAS_ROOM} -eq 0 ]; then
>   log_error "Remote system does not have enough free space for the backup."
>   notify
>   umount ${NFS_DIR}
>   die 4 "Remote system does not have enough free space for the backup."
> else
>   log "Remote system has enough room. Proceeding with backup."
>   log "===== ===== ===== ====="
>   log ""
> fi
>
> # Send today's delta over
> duplicity ${DUPLICITY_SWITCHES} / file://${BACKUP_DIR} 2>&1 1>>${LOG_FILE}
>
> # Remove unnecessary data over 6 days old
> duplicity remove-older-than 6D --force ${DUPLICITY_SWITCHES}
> file://${BACKUP_DIR} 2>&1 1>>${LOG_FILE}
>
> log ""
> log "===== ===== ===== ====="
> log "Backup complete."
>
> umount ${NFS_DIR}
> ````
>
> On Tue, Apr 21, 2015 at 1:21 PM, Edward Holcroft <eholcroft at mkainc.com>
> wrote:
>
>> I'm trying without encryption. It still seems the same, not constant hard
>> drive i/o going on. Will let it run for a while and see what happens.
>>
>> ed
>>
>
>
> --
> James Sumners
> http://james.sumners.info/ (technical profile)
> http://jrfom.com/ (personal site)
> http://haplo.bandcamp.com/ (band page)
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
>


-- 
Edward Holcroft | Madsen Kneppers & Associates Inc.
11695 Johns Creek Parkway, Suite 250 | Johns Creek, GA 30097
O (770) 446-9606 | M (770) 630-0949

-- 
MADSEN, KNEPPERS & ASSOCIATES USA, MKA Canada Inc. WARNING/CONFIDENTIALITY 
NOTICE: This message may be confidential and/or privileged. If you are not 
the intended recipient, please notify the sender immediately then delete it 
- you should not copy or use it for any purpose or disclose its content to 
any other person. Internet communications are not secure. You should scan 
this message and any attachments for viruses. Any unauthorized use or 
interception of this e-mail is illegal.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20150421/8a56b3b6/attachment.html>