Master FileSystem Redesign - Architecture Review Opinion

Issue: 51551
Submitter: Radek Matous
Inception Date: 2004/11/24
Final Date: 2005/01/04
Reviewers: Jaroslav Tulach, Trung Duc Tran, Tomas Pavek, Pavel Buzek
History: available in CVS

Content

Summary

The master file system (originally reviewed as issue 40958) started to be an important part of NetBeans 4.0. This prominent possion resulted in its increased usage and discovery of certain flaws that should be addressed in order to address different operating system specifics and make it more usable and reliable on various operating systems.

The current flaws include UNC problems on Windows, automount of /net on unix systems, symlinks and other ways how to get access to the same physical path by different logical names. The goal is to fix them for promotion E (while we will need to keep support for VCS filesystem) as well as promotion F (where the VCS filesystem may no longer be needed). Details on the goals can be found in [1].

Decision

Accepted with one TCA.

Opinion

The need for masterfs

The original need for masterfs (to plug into itself other filesystems) seems no longer be true or will not be true in near future. The mounting API has friend stability and is being used only by VCS, which will very likely soon switch its functionality to non-filesystem based implementation. Then the mounting API can completely disappear (probably promotion F).

However there still seems to be good reasons for existance of the masterfs module. It cleanly separates the filesystem APIs (org.openide.filesystems) from the implementation and allow us to solve problems that would be hard to address in the org.openide.filesystems part (like possible usage of JNI, less stable SPI interfaces, etc.).

The future direction is to move most of the java.io.File file object wrapping implementation into the masterfs module and slowly discontinue the use of SPI classes as AbstractFileSystem, LocalFileSystem, etc.

java.io.File relationship

The goal for masterfs has always been to provide one to one mapping with java.io.File, this is still a reasonable goal as it fits into the spirit of filesystem APIs (so we do not need to change it) and satisfies most of our needs. There however exists a need to provide certain improvements with respect to file uniquiness to prevent data loss. These should be addressed by improving the locking mechanism.

The reviewers agreed that the team shall write a new implementation of a filesystem for local files independent from AbstractFileSystem that will not share AbstractFileSystem troubles. This solves problems with /net and to certain extent addresses the high memory usage (see advisory information [1]).

At the time of promotion E, the MasterFileSystem implementation shall remain the same (allow pluging of VCS filesystems), just instead of the old org.openide.filesystem.LocalFileSystem let it use the new implementation. In promotion F (when the mounting will no longer be needed) a merge of the local fileobjects with master fs ones, further lowering the memory requirements, shall happen.

Windows UNC problems

In some sence the UNC files are special (one cannot list them in advance as A:/, C:/ and other roots), in some ways they just make certain problems more visible (a FileObject created on a laptop in work may not be found on the laptop running at home [advisory section [2]).

That is why the UNC roots should appear dynamically as chidren of the root of master filesystem when accessed. They can be represented by the newly implemented filesystem for local files or its subclass, in order to separate the UNC only hacks from the rest of java.io.File wrapping code. However the team is requested to make sure that problems are filled and that the JDK team is properly notified about importance of these issues (talk to Trung about that).

The identity problem

Resolving the symlinks problem by delegation of certain operation to a cannonical FileObject looks like a briliant idea. However in certain environments this is not sufficient as logical paths can lead to the same file even without using symlinks. This can cause possible data loss as the same file can still be opened in editor twice and edited more times at once.

There were a lot of ideas how to solve the problem from relaying on File.getCanonicalFile (known to be unsuffitient), to using java.nio.channels.FileLock (could possibly prevent other applications like ant to access the file - see advisory section [3]), using JNI (we do not want to do it), exposing some SPI for others to provide better ways to discover the identity (possible in future), or simulating what other editors do.

The reviewers suggest to try the last option and simulate emacs (advisory section [4]) for now and see if this is enough. If found unsufficient, other ways can be explored later. The possible problems found till now include the unability to lock a file this way if one does not have permition to create the lock file in a directory (just lock it without the file) or prevent the locking of a file if the file is itself referenced by symlink (try to traverse the symlink, or just do not solve this), but none of them seems to be too critical to not choose this solution.

As a result the file will get opened in editor more times, but when user types in both of them, he will be notified (beep, message in status line) that the file is already locked.

Minority Opinion

In the email discussion two opinions arised:

Advisory Information

  • In order to lower the memory requirements, please test and measure the amount of memory needed to open one file in editor (without its siblings being needed) and optimize for this case.
  • Implement proper serialization and deserialization. Make sure the fileobject can be deserialized even its java.io.File does not exists for proper handling of dynamic environment (laptop working at home/work/home/elsehwhere).
  • Check (possibly by having an automated test) the behaviour of nio FileLock and its suitfulness or unsuitfulness to be used in locking files while discovering their real identity.
  • Emulate the emacs locking - when somebody calls FileObject.lock() create a temporary file along the original (its name shall be changable as a private property to accommodate possible problems on various platforms) and delete the file on lock release. Check CLIHandlerTest for a possible ways how to write and test code that needs to synchronize on creation and modification of files between more independent applications, however the full crash detection and strict locking between various programs is not the goal number one. The goal is to make this working internally in one NetBeans based product.
  • Appendixes

    Appendix A: Technical Changes Required

    None.

    Appendix B: Technical Changes Advised

    53801 - test and measure the amount of memory needed to open one file in editor

    Appendix C: Reference Material

  • Submitted document (its version 1.4 that was reviewed).
  • Discussion on nbdev@netbeans.org
  • Project Features

    About this Project

    openide was started in November 2009, is owned by Antonin Nebuzelsky, and has 20 members.
    By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20160708.bf2ac18). © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
     
     
    Close
    loading
    Please Confirm
    Close