subversion checkout on distrubuted filesytem

Hi Folks,  We are using svn 1.8.11 and trying checkout on a distrubuted filesystem(DFS) like lustre. However it bails out. Mounting lustre filesystem with flock works to avoid filelocking with .sqllitedb but performance is very bad. Is there any solution for parallel/faster checkout on DFS?  svn: E200030: sqlite[S10]: disk I/O error, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA foreign_keys=OFF;PRAGMA locking_mode = NORMAL;' svn: E155007: '/b/noormohamed/svn_test/DEV_COMMON_BRANCH/src' is not a working copy svn: E200030: sqlite[S10]: disk I/O error, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA foreign_keys=OFF;PRAGMA locking_mode = NORMAL;' svn: E200030: sqlite[S10]: disk I/O error, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA foreign_keys=OFF;PRAGMA locking_mode = NORMAL;' svn: E155007: '/b/noormohamed/svn_test/DEV_COMMON_BRANCH/src' is not a working copy svn: E200030: sqlite[S10]: disk I/O error, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA foreign_keys=OFF;PRAGMA locking_mode = NORMAL;'

Last updated 4 years ago

Looking here: http://grokbase.com/t/subversion/users/125n432e2v/cant-execute-svn-commands-through-the-network The discussion included at least 2 SVN committers (Philip Martin and Daniel Shahaf) - pay attention to their comments. It would appear in that case that byte-level locking was not properly supported.  I'd run the same type of experiment that Philip Martin requested: 'strace svn co ...' and see what is causing the failure.  If it's just the performance of the checkout that's the concern (when mounting DFS with flock support), then perhaps doing a checkout to a local FS and copying the result into the DFS? The subsequent "svn st -u" will still likely be a poor performer...

Thanks Doug for the pointer. That's was our alternate plan to use local FS and copy to DFS.

It that possbile to checkout only the meta data[sqllite db] in a local FS and svn export of sources in DFS and later create symlink to handle as a workspace?

The sqllite db is not the only "private" part of the working copy. There is also the "pristine store" and some other files. They're all in the ".svn" directory at the working copy root.  I just did a quick test by moving the .svn directory to a different location and putting a symlink in its place: did NOT work. The alternative would be to have the .svn directory there but symlinks to everything else - and that just can't work properly since symlinks are legal constructs in that directory.  2 other possibilities: A. Try copying the ".svn" directory to a local FS and then mount that local FS over the ".svn" directory in the DFS (if mounting is allowable). B. Ask the SVN OpenSource community for help.

I asked the advice of the OpenSource community and, if your problem is poor performance of SQLLite locking on DFS then you might want to look into SQLLite locking configuration changes. Per Philip Martin:

 If SQLite locking works at all then the client-side config option     --config-option config:working-copy:exclusive-locking=true  may help.  It makes SQLite take less granular locks which improves performance, particularly on network filesystems, by reducing the SQLite overhead.  The performance gain can be substantial.    https://sqlite.org/pragma.html#pragma_locking_mode  Exclusive locking also reduces the concurrency of Subversion operations: generally only one Subversion operation can run on a working copy at any one time, but lots of users never notice this.

   Another committer, Branko Čibej, noted that you can actually move the SQLLite database out of the ".svn" folder and onto an "ext4" file system and leave a symlink behind. Something like: 

 mv .svn/wc.db /DFS/mywcdb/. ln -s /DFS/mywcdb/wc.db .svn/wc.db

   Finally, per another committer, Julian Foad, it's interesting to note that you can copy the ".svn" directory tree to some other location and then re-populate that location as a new working copy. I mention this because you might be able to "pre-populate" a working copy on your DFS by tar'ing the ".svn" directory up from an ext4 location, and then do an "svn update" to have it re-constitute itself. This might be helped by the suggestion by Philip Martin above.