Posts Tagged 'linux'

DIY backup system

I’ve got tons of satellite images and GIS data that I don’t want to loose. But hard drives are not eternal.

My home made backup system is a simple external usb2 hard drive: PC hard drive are becoming cheap, and you can by for some bucks an usb2 external case to plug it in. Now, the question is only how to manage that.

On any *nix system (Unix, Linux, Mac OS X, …) you can use rsync to make a fast backup of your repositories.
My backup hard drive is mounted on /Volumes/Archive on my Mac.
I wrote a very simple bash script:

#!/bin/sh -l
rsync -E -a -x -S --delete --progress --exclude-from=/Users/bubuitalia/exclude_from_rsync.txt /Users/bubuitalia /Volumes/archive/save

This script was saved in my home directory (/Users/bubuitalia) as backup.sh (do not forget to do a chmod u+x backup.sh this script to make it executable). You just have to change the path for your own installation. To run it, type ./backup.sh

This rsync command synchronizes the data in /Users/bubuitalia/ with the archive directory (/Volumes/archive/save).
The rsync commands line has the following options:

  • -E : copies extended files attributes.
  • -a : archive mode
  • -x : don’t cross file system boundaries (omits all mount-point directories from the copy)
  • -S : try to handle sparse file in an efficient way
  • –delete : delete extraneous files on the receiving system: if you delete something on your original data set, it will be deleted on the archive at the next synchronization. Use this function if you want to maintain a mirror copy of your system. It is worth to use it to avoid your archive size to get too large with time.
  • –progress : show progress during transfer.
  • –exclude-from=FILE : read exclude pattern from FILE

Don’t forget: if you run the script, any change in the original data will be applied to the backup. So, if you delete a file and want to restore it, do not run the script! First retrieve the data from the archive (any data deleted on the source, will be deleted on the archive after each synchronization).

I made another file, /Users/bubuitalia/exclude_from_rsync.txt, where I listed (1 entry per line) the directories I don’t want to save:

Music/
Movies/
Library/Caches/
.Trash/

To adapt this example to your own system, simply change the source and target directories.
The rsync page is on http://rsync.samba.org/
and you can find some other examples on http://rsync.samba.org/examples.html

Creating dummy (empty) files

For testing shell tips (linux and cygwin) it is often handy to

  • work a test directory
  • make some (tons of!) files
  • Ok, I suppose you can create directories (mkdir dirname). Now, you can use touch to create (empty) files:

    touch a b c d

    will create files a, b, c and d.

    To create 200 file starting their names with file_, followed with a number and ending with .img, do

    mkdir source
    cd source
    for ((num=0;num<200;num+=1)); do touch file_${num}.img ; done

    Now you can tests the shell tips (example).

    200 files made with touch command and a bash loop

    Batch renaming files

    Whether you are using linux (bash or shell) or cygwin, renaming a (large) set of files is really easy. Because renaming a file and changing its location is the same thing for linux and cygwin, you’ve got a large set of possibilities.

    First example. Say you have 200 files in a directory that must receive a prefix like new_. Here is what you can write from the prompt.

    for f in * ; do mv $f new_$f ; done

    and ALL files in the current directory have their names starting with new_ now.
    The star * tells to consider any file. For each file found, its name is stored in the variable f and the command mv $f new_$f is executed. The $f is replaced by each filename.
    Moving the files to another place is trivial. Say you have a directory named source and another named target, you can:

    cd source
    for f in * ; do mv $f ../target/new_$f ; done
    cd ..

    (you can use cp instead of mv to copy the original files to another named file).

    Second example. Now you want to change the file extension. It comes that bash lets you do some operations on the variables. For example, if f contains a string that ends with .img, say my_image.img, the instruction ${f%.img} would remove the extension .img from the end of the string (more manipulation in future posts). Renaming files ending with .img with .ida would be:

    for f in *.img ; do mv $f ${f%.img}.ida ; done

    More on files manipulation in future posts!