Universal Resource Library
Version: 0.2
Author: Yarda Tulach, NetBeans
- Abstract:
- The Universal Resource Library implements the concept of
universal file resource abstraction.
It provides an API that clients can use to access
files stored in local filesystems, JAR archives, CVS repositories, etc. It
also defines an SPI for resource providers, so additional pluggable filesystem
implementations can be created to access other types of file-like resources.
Client implementation is simplified because the storage of resources is
transparent to the client.
This document contains basic information how to download and and examples
describing some of the ways the library can be useful.
- Contents:
- 1. What is this?
- 2. Download instructions
- 3. APIs
- 4. Examples
- 5. Conclusion
The
Universal Resource Library is an implementation of the
NetBeans Filesystems API for the
NetBeans IDE. It is a set of
simple interfaces that encapsulate different sources of file-like
resources (local files, JAR archives, CVS repositories, etc.) and
provides a contract between the parts of the client that work on such
resources (in the case of NetBeans, the editor, compilers, debuggers,
javadoc generators, obfuscators)
and the providers of the resources (version control repositories,
network file shares providers, etc.). It can be used as a standalone
library independently of NetBeans for other Java projects.
Effectively, it gives you completely virtualized access to file data,
in which the underlying storage mechanism is tranparent and replacable.
Additionally, with the included MultiFileSystem implementation, it gives you
the ability to have two separate file hierarchies appear to be
superimposed over each other (this is particularly useful for
situations where data may need to be non-destructively overridden,
as in the case of user-specific settings, as detailed below).
Filesystems works on the same concept as Unix mount points - a set
of resources are mounted, while the communication protocol
is transparent.
The advantages
gained by using this library instead of java.io.File for file access
are:
- You automatically get support for .zip/.jar archives -
with read-write capability. A zip or jar is mounted
as if it were a directory, and all aspects of file
operations are handled transparently.
- The access to resources is cached - common queries like
does a resource with name
java/awt/String.class exist?
are cached and the number of accesses to disk or network is thus reduced.
- File storage is completely abstracted -
third parties can create support for access to file-like objects
stored in an arbitrary manner, such as in a database,
version control system or networked filesystem.
- You automatically get support for
CVS
and FTP -
by including the necessary classes
which are also part of NetBeans.
- You automatically get support for an XML filesystem
- where the filesystem is actually an XML document. File content within the
XML filesystem can be explicitly provided or supplied indirectly via a URL in
the document
- A Service Provider Interface (SPI) - for writing your own filesystem implementations
- Any new filesystem type that is available can be plugged in -
for example, if someone writes an NFS or IMAP filesystem
using the NetBeans filesystems SPI, it will work with your application.
- It supports listening for change events if a file is externally modified -
handy when writing complex applications where one part may modify a
document but does not necessarily notify another part that is also
interested in that document
- MultiFileSystem - One of the most compelling aspects of all,
this allows you to merge two mountpoints as a single
filesystem. Extremely useful for shared settings.
- Ability to add arbitrary "status" data to a file object and be notified of changes on it
- Support for actions on file objects, allowing you to define what actions are possible on a given file (and
dynamically update these), and then expose those actions through your user interface.
- Client code is simplified - because you do not have to write
code to support multiple types of file access.
- Applications using this library are more extensible as new
storage providers are developed.
There are several ways to get the library. Each of them has some advantages
and some disadvantages. The choice of download method
depends on how you intend to use the library.
If you need a new feature not available in the zipped distribution, you
can download the latest development
build of NetBeans and extract the following files:
- org-openide-fs.jar
- org-openide-util.jar
Add these archives to your classpath
and you are ready to start your standalone application that
uses the Filesystems API.
If you would like to read or hack the sources to this library, you
can get them either by anonymous
CVS or by
downloading
a tarball of the sources and
building
appropriate modules. If you make any useful modifications,
please contribute them back to the NetBeans project.
Note that you will also need the
nbbuild module and
extra libraries also available on the NetBeans project download page.
The
Filesystems API is a one part of the whole
API set of the NetBeans IDE. It is
possible to download it or browse its javadoc
online.
NetBeans is an open source project licensed under the
Sun Public License (a variant of the Mozilla license - a diff between the
two is available on the website). This library can be used in either closed-
or open-source projects.
If you are familiar with using the Filesystems API within NetBeans
The main difference when using the library as a standalone is that it includes
only the packages
org.openide.filesystems and
org.openide.util.*
plus the
org.openide.ErrorManager class. Because of this,
a few things need to be done differently:
- To access
ErrorManager one should write
org.openide.ErrorManager.getDefault ()
and be prepared that the returned value may be null.
- To access the entire repository of filesystems one should call
org.openide.filesystems.Repository.getDefault ()
The following are common scenarios that should demonstrate
the power of this library.
Obfuscators take compiled java class files and replace information such
as method names with semanticly meaningless text in order to deter
reverse engineering.
So, an obfuscator needs access to the
.class files
produced by compilation. These are typically local files,
but can also be packaged in JAR/ZIP files.
Mounting the current classpath in the filesystems repository
The first thing that the obfuscator using our library should do is it to setup
the filesystem repository based on the current classpath, so that it can find
class files it may be interested in. This means mounting all of the directories
and JAR archives it finds on the class path.
Repository r = Repository.getDefault (); //get the root of all mounted filesystems
for each elem in CLASSPATH { //iterate through each item in the classpath
if (elem.isFile ()) { //if it's a file we'll assume it's a jar archive
JarFileSystem fs = new JarFileSystem (); //create a new filesystem object for it
fs.setJarFile (elem); //point our object at the jar file
r.addFileSystem (fs); //add it to the repository (the set of all mounted filesystems)
continue;
}
if (elem.isDirectory ()) { //if it's a directory
LocalFileSystem fs = new LocalFileSystem (); //create a local filesystem object for it
fs.setRootDirectory (elem); //point our object at the directory
r.addFileSystem (fs); //and add it to the repository
continue;
}
System.err.println ("Non-existing classpath element: " + elem); //if it doesn't exist, complain
}
XXX these examples are obsolete as of NB 4.0; use masterfs + FileUtil.toFileObject
This is the only place the code needs to differentiate between the JAR
filesystem and other types of filesystem - when the filesystem is first being
mounted.
Locating file objects in the repository
So if the obfuscator needs to find a file of given name it can just call
FileObject fo = Repository.getDefault ().findResource ("folder/subfolder/name.ext");
and then test
fo != null to find out if that resource exists or not.
The content of the filesystem is cached in memory and is highly optimized
for the performance of findResource queries. So to find out whether a resource
really exists or not, it is often not necessary to access the background storage
at all. The results are much faster than
new java.io.File ("folder/subfolder/name.ext").exists ().
On the other hand the cache also uses weak references, so if something cached is not
used for long time, the cache releases its data structures to keep the
memory footprint small.
Creating new file objects in the repository
The output of our obfuscator will be another set of files, so we need
a way to create new resources:
FileSystem outputFileSystem = ....; //wherever we want to put the results
FileObject data = FileUtil.createData ( //create a new data file
outputFileSystem.getRoot (), "folder/targetfolder/name.ext" //specify a location for it
);
// get exclusive access to the resource
FileLock lock = data.lock ();
OutputStream os = data.getOutputStream (lock);
// write some data to os
lock.releaseLock ();
os.close ();
It is similar to existing code that uses
java.io.File, but has
the advantage (aside from the caching) that it is completely independent
from the resource provider. So one can easily set up the
Repository
to contain an
FTP filesystem and the obfuscator
will seamlessly work over FTP. Is that not nice?
Incremental obfuscation - listening for changes in the repository
A possible improvement is based on the idea of incremental obfuscation.
The input of the obfuscator is output of a compiler. So a better solution
would be to have our obfuscator listen to the resources it's interested in
(the .class files and folders where new ones may be created) and when
a .class file is created or updated, go to work? I believe
jikes has
such a feature, called
incremental compilation.
Let us design something like this for our obfuscator. Whenever the obfuscator
reads an important file, it can attach a listener to it.
FileChangeListener listener = new ObfuscatorListener ();
FileObject important = repository.findResource ("...");
important.addFileChangeListener (listener);
important.getInputStream ();
// do anything with the file
Using the listener, the obfuscator could cache information about what it has
done and only re-obfuscate what has been changed, so we have
incremental obfuscation.
class ObfuscatorListener implements FileChangeListener {
// ....
public void fileChanged (FileEvent ev) {
// file ev.getFile () has changed
// invoke the incremental update
}
// ....
}
The code is not complicated and changes the original obfuscator into
a complex processor of files that can become a member in a chain of other
tools - preprocessor, compiler, postprocessor, jar packager.
Imagine how much work it would require to write all this logic
and how it would complicate the code of the obfuscator? All the
caching, checking for file changes, uniform
access to JAR/ZIP and regular files would complicate the code tremendously.
With the Filesystems library the heart of the code, the obfuscator can be
coded more cleanly and be much more powerful.
Anyone writing a program that can be installed and then shared by
multiple users encounters a common set of problems: How to maintain default
settings, and how to merge the settings set up by default or by a superuser
with user settings.
The Unix approach is to provide shared settings in the
/etc directory and override those with user settings in
$HOME/.filename. The application itself must find these files and
determine what to do - merge the contexts, ignore the defaults if the user
file exists, etc. How many times has this logic been written over the
history of Unix?
Managing menu configurations using a filesystem hierarchy
The Filesystems API allows for a much cleaner, more flexible solution to this problem.
Take, for an example, the organization of a menu that will appear in an
application. The goal is to design a solution that allows the system
administrator to design the menu, and each user to customize it. But if the
system administrator changes it after the user has customized it, those
changes should be propagated too.
To solve the problem let's construct the menu from the file resources
(if this seems like a strange approach, recall how much more useful the File
abstraction became in UNIX when a file stopped necessarily being a file).
Imagine that each menu item is represented by one file, each submenu represented
by one directory. Define that there is the root folder called
"Menu" and its content defines whole menu. Let us also say
that the name of a menu item is defined by a name of its file and that the file
should have extension menu. The content of the file contains
instructions what to do when the menu item is invoked:
/Menu/
/Menu/File/
/Menu/File/Open.menu
/Menu/File/Save.menu
/Menu/File/Exit.menu
/Menu/Edit/
/Menu/Edit/Cut.menu
/Menu/Edit/Copy.menu
/Menu/Edit/Paste.menu
It is not hard to write an application that checks the content of the
Menu folder and composes the actual menu. Also it is not hard
to make the application listen for changes on the disk and update the menu
when new item is created or existing deleted. These are all features discussed
in the previous example about obfuscator, but there is one important
difference when using this library: MultiFileSystem.
Using MultiFileSystem to create a layered filesystem
MultiFileSystem takes an array of other
FileSystems
(called
layers) and merges their content.
For example, if the content of shared menu is:
/Menu/
/Menu/File/
/Menu/File/Exit.menu
/Menu/Edit/
/Menu/Edit/Cut.menu
/Menu/Edit/Copy.menu
/Menu/Edit/Paste.menu
and the user layer contains:
/Menu/
/Menu/File/
/Menu/File/Open.menu
/Menu/File/Save.menu
the result of the merge will be the same as in the
first example. This is the architecture
used in NetBeans and it has proven very useful.
As usual, the code that works with the
MultiFileSystem does not need to know anything about the storage of this
data. After initialization, all operations are transparent:
FileSystem sharedFS = new LocalFileSystem (...);
FileSystem userFS = new LocalFileSystem (...);
FileSystem settings = new MultiFileSystem (new FileSystem[] {
userFS, sharedFS
});
Any code accessing the
MultiFileSystem can do so just as it
would any other filesystem. If the client creates new data, it will be
created in the first layer (userFS in the above example).
We now have exactly the behavior we wanted. If the superuser changes the
default configuration of menu, those changes are immediately propagated
to all users, but the users can still modify the defaults by changing
the content of their own layer.
Moreover the solution can be extended to more than just
two layers. By just changing the initialization code we can devide the
users into more groups with different settings such as
department, project, team, etc.
Since the menu is merged from all of these layers the client code
that constructs the actual menu in the user interface remains unchanged!
There is one more piece of the puzzle, which is how to provide defaults
reliably. We'll cover that now, and look at how to use XML to transparently
provide defaults for those settings via XMLFilesystem.
As mentioned earlier, when a program is installed on
UNIX, it adds a file to the /etc
directory with the default settings it needs.
Then the superuser edits the file and changes the configuration to better
suit the system. This works but has some limitations.
What if a new version of the program is installed? It comes with another set of
default settings. So should it replace the existing file (possibly
modified by superuser) or not? Neither solution is a good fit for all cases,
but this is another place where MultiFilesystem offers
a solution:
FileSystem[] settings = new MultiFileSystem (new FileSystem[] {
userFS, // user layer for writing of own settings
groupFS, // group layer
sharedFS, // shared layer provided by superuser
defaultFS // default settings provided by application
});
As opposed to the previous example, there is another layer,
defaultFS, at the bottom,
containing default settings. These can be modified at the
sharedFS layer by superuser to reflect the system configuration.
The defaultFS filesystem is never modified, so it can be replaced when a new
version is installed without affecting changes made in other layers.
There are still some questions: How should the resources for defaultFS
be distributed? The application could, for example, create a JAR file with the
defaults:
defaultFS = new JarFileSystem ();
defaultFS.setJarFile ("/etc/defaults/ObfuscatorConfig.jar");
This would work, but what happens if we update to a new version of, say,
our obfuscator? This is a recipe for ending up with situations such as:
Warning: .cf file is out of date: sendmail 8.10.1 supports version 9, .cf file is version 8
where a configuration file from an older version of a program is used with
a newer version of the same program, which can cause unpredictable behavior.
Using XMLFileSystem to provide virtual settings files in an XML document
The solution is the
XMLFilesystem, also part of this
library.
XMLFilesystem defines an XML format which can be used as a
source of resources. So instead of reading settings from individual files
on disk, an XML document
is a filesystem; the others are layered on
top of it.
To solve the settings problem, simply include an XML document defining
default settings in the application (for example, in
/usr/lib/Obfuscator.jar). Voila!
The default settings are always carried with the application and can't
get out of synch, but users and administrators don't lose the ability to
customize. To all users, the XML file is read-only, but when the
application JAR is updated the defaults are updated too.
An example of such a file could look like:
<!DOCTYPE filesystem PUBLIC "-//NetBeans//DTD Filesystem 1.0//EN" "http://www.netbeans.org/dtds/filesystem-1_0.dtd">
<filesystem>
<folder name="Menu">
<folder name="File">
<file name="Open.menu" /> <!-- empty content -->
<file name="Save.menu" >
content written directly into the XML file
</file>
<file name="Exit.menu" url="../contents/Exit.txt" /> <!-- external reference to another URL -->
</folder>
</folder>
</filesystem>
The file can be empty, can be explicitly specified, or
one can use a URL to refer to include content from elsewhere.
For further details about XMLFileSystem see the
API documentation.
We hope that you find this library interesting and useful. If you have questions
or comments, please join the
nbdev
mailing list (nbdev@netbeans.org).
