Copyright 2006 by Rules | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ContentsWhat it isScreenshotRequirementsLicenseDownloadPreface to installation and runningBinary installationInstallation from SourceSetting up to runUsageScriptingFAQ: How does Xfiles differ from Rsync?Appendix: Version logAppendix: Sample xfiles.py scriptAppendix: Link/Alias detectionAppendix: TroubleshootingAppendix: Installation from Source using native link detectionWhat it isXfiles is an interactive utility for comparing and mergingone file tree with another over a network. It supports freeformwork on several machines (no need to keep track of what filesare changed on which machine). Xfiles can also be used as a cross-validating diskdisk backup strategy(portions of a disk may go bad at any time, with no simple indication of which files were affected. Cross-validate against a second disk before backup to make sure you aren't backing up bad data).A client/server program (GUI on the client) traverses a file tree and reports any files thatare missing on the server machine, missing on the client machine, or different.For each such file, the file size/sizes and modification date(s)are shown, and a comparison (using Unix diff) can be obtained. For files that are missing from one tree, `similarly named' files in that tree are reported.Inconsistent files can then be copied in either directionor deleted on either machine.The file trees do not need to be accessible via nfs.Files checksums are computed in parallel, so largely similar trees canbe compared over a slow network link. The client and server processes can also be run on the same machine.See the topic 'How does Xfiles differ from Rsync' below formore information.ScreenshotRequirementsXfiles1.4 requires Java jdk1.2 or above.To see if you have java installed, type java -version,which should report something like > java -version java version "1.3.0"There are various free implementations of jdk1.2 and above.www.javalobby.org has links to someimplementations. The Sun java versions are at java.sun.com under "Products and APIs". www.blackdown.org has versions that are better tuned to Linux,though their improvements are also integrated into recent Sun versions.IBM also has a free JDK1.3 implementation with an alternatehigh-performance virtual machine.LicenseDownloadRead the LICENSE.txt file first. Xfiles is released under the GPL,but it optionally uses jpython for scripting;jpython has its own open sourcelicense, see LICENSE.txt.Download the latestxfilesBinary or xfilesSource or archive.Optionally download jpython.jar if you want to do scripting (see scripting section below). [Note: in netscape download this by right-clicking on the link,or by shift-button 1. Simply clicking on the linkcauses the file to be read into the netscape window,and sometimes causes the browser to hang if you have java turned on]jpython.jar is also needed to recompile the source, though small changes to the source will let it recompile without it.jpython.jar is not needed if you do not want to script.Download the nativeFile archive only if you want native Unix link detection (see discussion at end). You probably do not need this.Source and binary archives for older versionsPreface to installation and runningJava tends to be poorly integrated with the underlying operating system.If you're new to java take care to read the directions below --there are not yet conventions for where and how java programs (norJava itself) should be installed, so the installation outlined below isnot as streamlined as a Linux rpm or similar. The individual steps areeasy however.If you don't already have Java you'llneed to select some install location such as /usr/local/java; any directory will work. Similarly, the Xfiles program can live anywhere. If you follow theinstallation below you will need to launch the program from the directorywhere it resides (this does not restrict its function). Changes to theshell files to make it run from any directory are evident.The program and this documentation refer to client and server machinesand directories. These are interchangable -- the server merely refers tothe machine that the server is running on (see below).Binary InstallationFor both client and server machines, do the following steps:Put the Java bin directory on your path. If using scripting put the jpython.jar file in the samedirectory as xfiles.jar and the xfilesClient, xfilesServer scripts.Installation from Source(The following instructions are for Unix/Linux)Put the Java bin directory on your path. The programs javac, java, and rmic will be needed.To compile the source you must download jpython.jar and place it in the source directory.makeSetting up to runThe following files comprise the program, verify that theseexist in the current directory:xfiles.jar java archive containing the programxfilesClient shell program to launch the client guixfilesServer shell program to launch the serverjpython.jar optional, needed for scriptingNetworking must be configured and turned on even if bothclient and server are running on the same machine. I believe thatJava RMI needs the system service that converts a hostname into an IP address,and uses this even if both the client and server are running on the samemachine. There is discussion of this point on some of the java mailing lists.On my Redhat system, all I do is runnetcfg and turn on oneof the configured interfaces. Doing /sbin/ifconfig <interface>up would probably work too.You may need to enable the hosts in the Unix .rhosts file.If you know networking please send me more authoritative(spell?) instructionsfor what's needed here. Launch the Xfiles server. On the server machine, run: xfilesServerThis should print out the server hostname,then "XfilesServer is running".If you get an error that says "java.lang.ClassNotFoundException: <nameof class>", one of the paths in the xfilesServer file is not setcorrectly.Launch the Xfiles client on the client machine, giving the server hostname (spelled as printed by xfilesServer)and the roots of the client and server file trees: xfilesClient duckpond /usr/jfk/devl /home/jfk/DevlThe command above will compare the file tree starting at /usr/jfk/devlon the client machine with the file tree /home/jfk/Devl on the machine 'duckpond'.UsageWhen the GUI comes up, select a directory and press the start button.The client displays the client file tree at startup to allowyou to select a sub-tree of the specified root if desired.If you select nothing the entire tree will compared when you hit the start button.To save you time Xfiles first scans the whole (sub)tree before reporting any differences (this may take a while);all differences are then reported consecutively.After synchronizing one directory, you can select another in the GUIand press start again. Currently the GUI file tree does not update to reflect deletions in earlierruns, however (see the TODO section).The following are optional command-line arguments:-fastThis causes Xfiles to ignore files whose size and date are identical,so the scan is much faster. This will not detect disk corruption however.-exclude substring1 substring2 ...Arguments up to the next flag argument indicate files to be excluded.Any file that contains one of these arguments as a substring willbe skipped.-clientonlyFocus on the client: files that are older on the server or are missingon the server are not reported.-serveronlyFocus on the server: files that are older on the client or are missingon the client are not reported.-scriptargsArguments following this are not examined; all arguments are passedto the optional xfiles.py setup(args) function, so scripting arguments should be passed following this flag.Xfiles writes a file XFILES.LOG listing the selected actions.ScriptingFile selection and interaction with a revision control system suchas RCS can be handled by scripting using jpython.To enable scripting, download the file jpython.jar and place it in the client and server launch directories.Then create a file xfiles.py, which also must be copied into both theclient and server launch directories.xfiles.py can define the following functions:setup(args) This function, if defined, will be called with the command line arguments. Arguments following -scriptargs are ignored by the application, but all arguments (not just those following -scriptargs) are passed to the xfiles.py script.pathFilter(path) This function should return 1 if Xfiles should process path, 0 if Xfiles should ignore it. preCopy(path) this function is called before Xfiles writes this file. It can be used (for example) to check out the file from RCS. postCopy(path) this function is called after Xfiles writes the file. It can be used to check the file into RCS.It is not necessary to define all of these functions, however,if a function is defined it should be correct -- if thefunction call generates an error Xfiles will quit.A sample xfiles.py file is contained with the distributionand listed at the end of this document.FAQ: How does Xfiles differ from Rsync?Rsync works automatically, and assumes the source directory is correct.If that disk has corruption, good files on in the backup directory may be overwritten with corruptedfiles from the source disk. A similar scenario is when yousave a file that was accidentally edited somehow.Again, the newer file is not the desired one.Xfiles is also allows relatively unstructured workin several places, without needing to think of onelocation as the 'master' -- in this case each directory may have files that are obsolete, as well as ones that should be mirrored. An automated algorithm will duplicate the files that need to be deleted, resulting in extra work later. Xfiles ask you if you wantto copy (and in which direction), and gives the option to delete.In the case of text files it also gives a "diff" (On unix) to help in the decision.In the case where both files have desirable parts, the file mustbe manually merged; Xfiles doesn't help here (except to identify the situation). Adding something like the Unix xdiff visual mergewould be a nice addition.In rsync's favor, rsync is much faster. (Although, if you add -cto force checksums on all files, rsync gets a lot slower).AuthorPrimary author: J.P.Lewis scribblethink.orgContributors: Peter Gadjokov, Wolfgang Lugmayr, Dan SchmittPlease e-mail problems, successes, fixes, and fears to: zilla@computer.orgSend email with subject line XFILES to be notified of updates.To doIt should be possible to rename files.The client side gui should update to reflect the changed file treePortable replacement for diff.If both client, server files are ascii it would be betterto transmit a diff and do a patch on the other end. Appendix: Version log1.4 jul00Added -clientonly, -serveronly, -fast flags;preserve dates when copying (assumes jdk1.2),update files to assume jdk1.2 (JFC no longer required on classpath),xfiles.py has setup() call.1.3 28mar99Added JPython scripting to control what filesare visited and to script interaction with a revision controlsystem such as RCS. 1.3.1: improved documentation.1.2 22mar99External rmiregistry no longer needed(simplified usage and troubleshooting, also helps if running over a firewall),paths are transmitted from client to server as java.io.Files rather than Strings sothat path separator translation is done at unserialization time --should help if client and server are running on different O/Ss.1.1 7jan99 Rewritten to not require native (non-java) link detection;program is more portable to non-Unix environments;installation is simplified; other improvements.1.0.2improved definition of 'similarly named' files,warn if deleting on the wrong side, copy direction is color coded.1.0.1 5jan99 fixes: stop did not work always, bad window sizing,prohibit copying in the wrong directionby accident - e.g. client to server if the server file exists and the client does not.1.0 3jan99 initial versionAppendix: Sample xfiles.py scriptThe following example script implements these functions:setup(args)- Just prints out the command-line argumentspathFilter(path)- Ignores all object files, libraries, .so files, RCS files, java class files preCopy(path)- Checks out a file from RCS if needed before you copy over it. postCopy(path)- Checks a newly copied file back into RCS with a message saying that Xfiles created the revision.Although jpython is quite compatible with regular python, this file is written using a mixture of python and java calls(for example, java string functions are used rather thanthe python string.split routine).This is done to limit the dependence on jpython to theone file jpython.jar.import javafrom java.io import Fileimport java.lang.Runtimeruntime = java.lang.Runtime.getRuntime()# ignore files that end with these stringsskipextensions = ['RCS', ',v', '.o', '.so', '.a', '.class', '.jar']# grab the command line arguments. # You could pass in the skipextensions here for example.def setup(args): print 'setup got args:' for arg in args: print arg# return 1 if xfiles should visit this path, else 0#def pathFilter(path): print 'pathFilter(%s)' % path if path[len(path)-1] == '~': # emacs backup file return 0 if path == 'so_locations': return 0 spath = java.lang.String(path) for ext in skipextensions: if spath.endsWith(ext): return 0 return 1# called before copying over a file# (check out from RCS if appropriate)#def preCopy(path): name = filename(path) spath = filedir(path) spath = spath + '/RCS/' print 'name = %s' % name if exists(spath): # RCS/ exists spath = spath + name + ',v' print 'spath = %s' % spath if exists(spath): # RCS/file,v exists docmd('co -l -f %s' % path)# called after copying over a file# (check in to RCS if appropriate)#def postCopy(path): name = filename(path) spath = filedir(path) spath = spath + '/RCS/' print 'name = %s' % name if exists(spath): # RCS/ exists spath = spath + name + ',v' print 'spath = %s' % spath if exists(spath): # RCS/file,v exists docmd('ci -u -f -mXfiles_copy_checkin %s' % path)# helper commandsdef docmd(cmd): if 1: print cmd pid = runtime.exec_(cmd) pid.waitFor()def filedir(path): result = File(path).getParent() if not result: if isabs(path): result = path # Must be root else: result = "" return resultdef filename(path): return File(path).getName()def exists(path): return File(path).exists()def isabs(path): return File(path).isAbsolute()Appendix: Link/Alias detectionThis section is largely obsolete because Java link detection appears to work adequately.Because Xfiles traverses a directory tree, it needs to be able to distinguishbetween "real" files and links (aliases) so as to avoid an infiniteloop in the case where a link points to a directory above itself. There are two approaches to this, and you need to selectwhich one you will use: A built in method An approach that relies on a native (non-java) library nativeFile.Xfiles will call the native function if it exists in the launch directory.Installation with the native function is a bit more work, and itdoes not exist for non-Unix operating systems yet.For most purposes it will probably be fine to use the built-in code.Read the appendix Links/Aliases/Shortcuts in Java for more details on this issue.Troubleshooting GuideConnection refused to host: 127.0.0.1The errorjava.rmi.ConnectException: Connection refused to host: 127.0.0.1this often has something to do with the /etc/hosts file.On one Redhat6.1 machine there was a /etc/hosts file that looked like127.0.0.1 localhost.localdomain localhost [Qserver]xx.xx.xx.xx Qserver [localhost]where [] means optional.When "Qserver" is present on the first line and the rmi serveris running on this host, the client gets the `rmi connection' refused error.When Qserver is removed from from the first line, the rmi connectionworks, but now the machine hangs on startup at "starting sendmail"for several minutes (but then continues ok).The sendmail hang issue is discussed in one of the /usr/doc/HOWTOdocuments, but I do not understand that discussion.In another case this seemed to happen because the unix DISPLAY variable was not set before running the server, or because"host +" was issued before starting the X server.java.rmi.UnknownHostException: Unknown host: scruffI think this happened on linux when /etc/HOSTNAME orsomething else set the name "scruff", but scurff was not found in /etc/passwd.'import exceptions' failed; using string-based exceptionsThis jpython warning is not a problem.java.lang.NoClassDefFoundError: org/python/core/PyObjectMeans that jpython.jar was not found on the CLASSPATH at runtime.This is not a problem if you are not scripting xfiles.XfilesScript.java:22: cannot resolve symbolsymbol : class PythonInterpreterThis means that jpython.jar is not found during compilation.Download jpython.jar, put it in the compile directory, andcheck that the Makefile adds it to the CLASSPATH.Installation from Source, using native link detectionPut the Java bin directory on your path. The programs java, javac, and rmic will be needed.To compile the source you must download jpython.jar and place it in the source directory.Download and extract the nativeFile archive.Build the nativeFile library - in the Nativefile subdirectoryedit the Makefile CFLAGS to pointto the location of your java installation's include files and the JNI includefiles. The latter are in a subdirectory of the java include, probably calledeither "genunix" or the name of your OS.Build the nativeFile library - build the java binding to the nativeFile:make nativeFile.classBuild the nativeFile library - compile the .so: make nativeFile.soCopy nativeFile.class and libnativeFile.sointo the directory containing Xfiles,go to that directory.Examine the LD_LIBRARY_PATH variable in xfilesClientand xfilesServer and modify if needed.makeback to home| --BEGIN VALIDATION CODE-- K gOsX eSzPvXaPxNnOnR gTsHgJzAqCnM eG nBn TvDnU kT eIjFvQ iNcGjOyWoGqO qVc YnXkNnJhKoXdZt CeD hWrI aDeHoGxPfGeAhJwI kTmMe KzVsX xC wZjT oPs T uAkYqOlF gUsMoBmAtTgJ xMoWgUx U gC cO qJ s OuTjQ yNvN lYwN gF h ScOzY nWdW wO lJf YgLwJmB sHgPgR lQjRhOrMbRcZ bQqYxQeUzKgRpY iUdYoLgM yK hohositeX2006 --END VALIDATION CODE-- |
