Backup script

lovedaddy
lovedaddy
Have been working for a few hours on a backup script to be called from Cron to do an incremental dump, between the last dump and latest revision. Works with multiple repos in the same basedir. [code]#!/usr/bin/python # script to backup SVN repos at a given time from the last revision number # Greg Loscombe import os import sys debug = 0 # build a list of projects and what the last revision backup basedir = "/home/subversion/basedir" basebackup = "/var/log/svnbackup" revlogfile = "/var/log/svnbackup/last_rev" projects = {} something_dumped = 0 def get_youngest(proj): youngest_cmd = "/usr/bin/svnlook youngest " + basedir + proj if ( debug == 1): print "command:", youngest_cmd cmdout = os.popen( youngest_cmd ) rev_youngest = cmdout.read() rev_youngest = int( rev_youngest.strip() ) cmdout.close() return rev_youngest def dump_incremental(base, proj, rev_from, rev_youngest): location = base + proj + ".dump." + str( rev_from ) + "-" + str( rev_youngest ) if ( debug == 1): print "Location dumped too", location incremental = "/usr/bin/svnadmin dump --incremental " + basedir + this_proj + " -r " + str( rev_from ) + ":" + str( rev_youngest ) + " > " + location cmdout = os.popen( incremental ) cmdout.close() if ( debug == 1): print "Command incremental dump:-", incremental def update_log(): revfile = file( revlogfile, "w") for this_proj in projects.iterkeys(): revfile.write( this_proj + " = " + str(projects[this_proj]) + "\n" ) if ( debug == 1): print "New log:-", this_proj + " = " + str(projects[this_proj]) + "\n" print this_proj + " = " + str( projects[this_proj] ) revfile.close() def update_error_log(message): errorfile = file( "/var/log/svnbackup/error_log", "a") errorfile.write( message ) #errorfile.write( proj + ":" + str( updated_rev ) + ":" + message ) #add more stuff here soon errorfile.close() #def update_access_log(message): #this will print things like last run etc # main part now, lets get all the old revision numbers we backed up revfile = open( revlogfile ) while 1: line = revfile.readline() if line == "": break if ( ( line.startswith("[") or line.startswith("#") ) is False ): proj, last_rev = line.split(' = ') last_rev = last_rev.strip() projects[proj] = int(last_rev) revfile.close() # lets get the new youngest and compair, if its newer, then do a incremental dump # if its older than the last log, we have a problem, log it # if its the same nothing happens # if its never been dumped, time for a full dump for this_proj in projects.iterkeys(): rev_youngest = int( get_youngest(this_proj) ) rev_old = int( projects[this_proj] ) if ( debug == 1 ): print "this project", this_proj, "old revision", rev_old, "youngest", rev_youngest if ( rev_old > rev_youngest ): errmessage = "logged backed up revision " + str(rev_old) + " > youngest current revision " + str(rev_youngest) + "\n" #update_error_log(this_proj, rev_youngest, errmessage) update_error_log(errmessage) elif ( rev_old == 0 ): print "First time dump, do it all for", this_proj, "upto", rev_youngest dump_incremental(basebackup, this_proj, 0, rev_youngest) something_dumped = something_dumped + 1 projects[this_proj] = int(rev_youngest) elif ( rev_old < rev_youngest ): print "Update for project", this_proj, "upto", rev_youngest rev_from = rev_old + 1 dump_incremental(basebackup, this_proj, rev_from, rev_youngest) something_dumped = something_dumped + 1 projects[this_proj] = int(rev_youngest) elif ( rev_old == rev_youngest ): print "Revisions have not changed for project", this_proj # if something changed then its time to update the last logged file if ( something_dumped > 0 ): print something_dumped, "repos have changed since last dump" update_log() else: print "Nothing has changed, nothing done" # the end[/code] The last_rev file located in /var/log/svnbackup would have something like this in it: [code]MyProject = 0[/code] To start with. When you run the py file it will update that to the last commit from that project when you ran the script. To add another project just add it to the bottom of the last_rev file and set it equal to zero. (to redump from scratch, delete the .dump. files and set = 0 again) Its nowhere near finished (error trapping, callbacks etc) - but works for me today, and might help a few out. Any suggestions, please feel free...[/code]

Last updated

dserodio
dserodio
Small corrections It's a great script, thanks for sharing it. But I could only make it work by making [b]basedir[/b] and [b]basebackup[/b] end in a slash. I'm working on an improved version of this script, which optionally compresses the generated dumps. I'll post it here when I'm done.
lovedaddy
lovedaddy
ending in a slash This might be to do with the last_rev log file... I put / at the start of every project... eg:- /Shared/Game = 15437 /Shared/Tech = 18565 /Projects/FVC = 377 etc. Any updates would be great, I've still never really had time to improve it, the next thing I'll put together is a way of automatically loading the files into a clean repos. (achieved it the other day with a bit of ordered ls, grep trickery, but a real script might be nice =)
Manuzhai
Manuzhai
You should probably os.path.join() projects and repository instead of just concatenating them, that will solve problems with trailing/starting slashes.
dserodio
dserodio
Improved version of the script I improved on this script a little, compressing the resulting dump and replacing "if(debug): print" with "logging.debug", as well as normalizing the paths: [code] #!/usr/bin/env python # Script to incrementally backup SVN repos at a given time from the last revision number # Written by Greg Loscombe # Found at [url]http://www.svnforum.org/forum/viewtopic.php?t=183[/url] # # Improved by Daniel Serodio import os import sys import logging # build a list of projects and what the last revision backup basedir = "/var/lib/svn/" basebackup = "/var/backups/svn/" revlogfile = "/var/backups/svn/latest_rev" logfile = "/var/log/svn_backup.log" compress = "bzip2" # if set, the program to compress the dumps projects = {} something_dumped = 0 logging.root.setLevel(logging.INFO) def get_youngest(proj): youngest_cmd = "/usr/bin/svnlook youngest " + basedir + proj logging.debug( "command: %s", youngest_cmd ) cmdout = os.popen( youngest_cmd ) rev_youngest = cmdout.read() rev_youngest = int( rev_youngest.strip() ) cmdout.close() return rev_youngest def dump_incremental(base, proj, rev_from, rev_youngest): location = os.path.normpath("%s/%s_%s-%s.dump" % (base, proj, str( rev_from ), str( rev_youngest ))) logging.debug( "Location dumped to %s", location ) incremental = "/usr/bin/svnadmin dump --incremental " + basedir + this_proj + " -r " + str( rev_from ) + ":" + str( rev_youngest ) + " > " + location cmdout = os.popen( incremental ) cmdout.close() logging.debug( "Command incremental dump: %s", incremental ) return location def update_log(): revfile = file( revlogfile, "w") for this_proj in projects.iterkeys(): revfile.write( this_proj + " = " + str(projects[this_proj]) + "\n" ) logging.debug( "New log: %s = %s\n", (this_proj, str(projects[this_proj]))) logging.info( "%s = %s", (this_proj, str( projects[this_proj] ) )) revfile.close() def update_error_log(message): errorfile = file( logfile, "a") errorfile.write( message ) #errorfile.write( proj + ":" + str( updated_rev ) + ":" + message ) #add more stuff here soon errorfile.close() #def update_access_log(message): #this will print things like last run etc # main part now, lets get all the old revision numbers we backed up revfile = open( revlogfile ) while 1: line = revfile.readline() if line == "": break if ( ( line.startswith("[") or line.startswith("#") ) is False ): proj, last_rev = line.split(' = ') last_rev = last_rev.strip() projects[proj] = int(last_rev) revfile.close() # lets get the new youngest and compair, if its newer, then do a incremental dump # if its older than the last log, we have a problem, log it # if its the same nothing happens # if its never been dumped, time for a full dump for this_proj in projects.iterkeys(): rev_youngest = int( get_youngest(this_proj) ) rev_old = int( projects[this_proj] ) filename = None logging.debug( "this project=%s, old revision=%s, youngest=%s" % (this_proj, rev_old, rev_youngest )) if ( rev_old > rev_youngest ): errmessage = "logged backed up revision " + str(rev_old) + " > youngest current revision " + str(rev_youngest) + "\n" #update_error_log(this_proj, rev_youngest, errmessage) update_error_log(errmessage) elif ( rev_old == 0 ): # logging.info( "First time dump, do it all for %s up to %s", (this_proj, rev_youngest )) filename = dump_incremental(basebackup, this_proj, 0, rev_youngest) something_dumped = something_dumped + 1 projects[this_proj] = int(rev_youngest) elif ( rev_old < rev_youngest ): logging.info( "Update for project %s up to %s", ( this_proj, rev_youngest )) rev_from = rev_old + 1 filename = dump_incremental(basebackup, this_proj, rev_from, rev_youngest) something_dumped = something_dumped + 1 projects[this_proj] = int(rev_youngest) elif ( rev_old == rev_youngest ): logging.info( "Revisions have not changed for project %s", this_proj ) if ( something_dumped and compress != None ): logging.info( "Compressing %s with %s",( filename, compress )) cmdout = os.popen( "%s %s" % (compress, filename )) cmdout.close() # if something changed then its time to update the last logged file if ( something_dumped > 0 ): # logging.info( "%s repos have changed since last dump", something_dumped ) update_log() else: logging.info( "Nothing has changed, nothing done" ) # the end [/code]
indee
indee
Hi, First of all thanks for the script. I am getting this error message [code]Traceback (most recent call last): File "/usr/lib/python2.3/logging/__init__.py", line 674, in emit msg = self.format(record) File "/usr/lib/python2.3/logging/__init__.py", line 567, in format return fmt.format(record) File "/usr/lib/python2.3/logging/__init__.py", line 362, in format record.message = record.getMessage() File "/usr/lib/python2.3/logging/__init__.py", line 233, in getMessage msg = msg % self.args TypeError: not enough arguments for format string [/code] I can see the dump is created. Here are the version details Apache: Apache/2.0.52 subversion-1.2.3-1.rhel4 subversion-devel-1.2.3-1.rhel4 subversion-perl-1.2.3-1.rhel4 subversion-perl-1.3.2-1.rhel4 subversion-python-1.2.3-1.rhel4 Python 2.3.4 OS RHEL 4
crazytoon
crazytoon
backup script for multiple repositories I wrote a backup script which does multiple repositories and uses hotcopy which is available from svn devs [url]http://crazytoon.com/2007/08/16/subversion-svn-how-do-you-set-up-backup-for-your-subversion-repositories/[/url]

1-7 of 7

Reply to this discussion

You cannot edit posts or make replies: You should be logged in before you can post.

Post a reply
11897 views