Datasystems Redesign Plan

This is preliminary plan for Datasystems redesign in core platform. The plan expects that you read following documents:

• Problem overview
• Solution overview
  

The plan first defines highlevel areas, then goes into more details for each area, and at the end it defines milestones in which the plan can be implemented. The plan is still quite high level and needs to go into more details if realistic estimates are needed. At the moment it just captures all what has to be done, all what is unknown and needs to be analyzed and defines dependencies between these tasks.

If there is no status or owner mentioned it means that no work was done yet and task is not planned yet.

At the moment plan does not contain any tasks for performance evaluation. They have to be added.

Highlevel Areas

The Datasystems redesign plan can be split into these highlevel areas:

• A1: Create Settings API
  Implement Registry and Convertors APIs. This step will eliminate part of the DS (eg. InstanceDataObject, XMLDataObject) which are used for settings.
  

• A2: Migrate to Settings API
  All usages of DS for settings management must be rewritten to use new settings system.
  

• B1: Create Filesystems Extension API
  Implement Filesystems Extension API. There already exist some implementation. The size of the API is relatively very small. The implementation should be consulted with projects team which already using this concept and any overlapping API should be merged into platform if possible.
  

• B2: Migrate to Filesystems Extension API
  Migrate from old DS to Filesystems Extensions API. Cleanup of org.openide.loaders package.
  

• C1: Extend or reuse existing subsystems
  All existing APIs which are using Datasystems must be extended to work smoothly without them. In practice it means that they must be able to work with FileObject only. Part of this area is also extending these subsystems with features in the past provided by old DS
  

Areas in more detail

A1: Create Settings API

A1.1: create Registry API
    A1.1.1: create proposal & prototype for the module and its public API
    A1.1.2: discuss them on mailing list and incorporate feedback
    A1.1.3: full implementation, tests, documentation
Status: DONE.
Comment: The Registry API is available for use with one restriction that for object persistence are still used current Datasystems. After the Convertors API is finished the Registry impl will be changed to use Convertors API instead. This will have significant impact on lifecycle of instances created by clients. See A1.3.
IsueZilla: 32679
Links: Registry API
Owner: dkonecny@netbeans.org

A1.2: create Convertors API
    A1.2.1: create proposal & prototype for the module and its public API
    A1.2.2: discuss them on mailing list and incorporate feedback
    A1.2.3: full implementation, tests, documentation
Status: DONE.
IssueZilla: 34671
Links: Convertor API
Owner: dkonecny@netbeans.org

A1.3: rewrite default implementation of Registry to use Convertors API
The major difference is that while old Datasystems saved modified objects automatically the new system requires to explicitly bind object back to Registry to make changes permanent.
This task has external dependency on new projects. They are already using Registry and are using old core/settings convertors and possibly rely on the autosaving behaviour of DS. They have to do two things in one step immediatelly after this task is implemented: #1 rewrite existing convertors to new Convertors API and #2 explicitly save all modified objects
Status: in progress.
IssueZilla: 35050
Owner: dkonecny@netbeans.org

A1.4: create Registry implementation over XML document
Current Registry implementation works still over the Filesystems layer. This tasks defines new format of module layer which is not constrained by XML Filesystem DTD, which is validatable by XML Schema, which uses XML Namespaces, and which is more natural for registration of instances then Filesystems DTD. The format will take advantage of Convertor API. This will eliminate Filesystems layer from settings handling.
    A1.4.1: define structure of new XML module layer
    A1.4.2: implement Registry for that XML
    A1.4.3: write runtime conversion from old module layer to new format
    A1.4.4: write binary cache of merged module layers

Dependencies:
A1.3 depends on A1.2.

A2: Migrate to Settings API

A2.1: API enhancements requests and support
Plan some time for API support and API enhancements requests which will surely come after clients start using the APIs.
Status: in progress.
IssueZilla: 34114, 34274, 34353, 34342, 34424, 34551, 34019 and others might come.
Owner: dkonecny@netbeans.org

A2.2: create support for merged Registry Contexts
This is needed to fully replace SystemFS by default Registry. That will make NetBeans backend configuration system independent on Filesystems and will allows us to extend format for declarative registration.
    A2.2.1: create analysis: requirements, masking of deleted bindings, etc.
    A2.2.2: full implementation, tests, documentation
    A2.2.3: change default registry to be merged Registry Context of installFS context and customizationFS context
Status: A2.2.1 finished; A2.2.2 in progress
IssueZilla: 34057
Links: Merged Contexts chapter on this page
Owner: rmatous@netbeans.org

A2.3: create replacements for several existing openide support classes
    A2.3.1: support for Menu based on top of the Registry Context
    A2.3.2: support for Toolbar based on top of the Registry Context

A2.4: rewrite everything in core and openide to use settings system directly or the helper classes from A2.3
The detailed scope of this issue is not known at the moment. It is possible that "something" what was not expected will appear.

A2.5: default lookup
Default look will be based only on META-INF lookup. The content of /Services folder from SystemFS will be removed from it. These services will be accessed directly by Registry API.
Links: Some thoughts about this task.

A2.6: reimplement settings presentation
Settings presenter (ie. options dialog) must be internally reimplemented to use Registry. The major difference is that modified objects are not persisted automatically. There are two types of objects under UI/Services which will be distinguished: Node instances and JavaBeans instances. The Node instances must take care of persisting objects they are visualizing when some property has changed. The JavaBeans instances (eg. SystemOptions, services, etc.) will be wrapped into a special Node which will listen on bean properties and which will bind modified object back to the Registry.
    A2.6.1: create ContextLook as replacement of DataFolder.FolderNode. Can be reused by TemplateWizard.

A2.7: resolve access to files on SystemFS
The SystemFS contains also files which clients need to access and which are at the moment not accessible by Registry API, eg.: DTDs, parser databases, templates, etc. For more details see analysis of this task below. Originally this was thought to be difficult problem, but after analysis only two quite small issues remain. See the analysis for more.
Links: analysis of this task

A2.8: migration of other NetBeans modules
    A2.8.1: prepare cookbook of what and how should be migrated
    A2.8.2: plan some time for support of clients

Dependencies:
A2.4 depends on A2.3
A2.6 depends on A1.3
A2.7 does not depend on anything
A2.8 depends on A1.3 and better wait till most of the A2 tasks are finished

Most of the A2 tasks (apart from A2.3 and A2.4) can be done in parallel by different people.

B1: Create Filesystems Extension API

B1.1: create Filesystems Extensions API
    B1.1.1: resurrect work already done in this area; merge API overlapping with project ones
    B1.1.2: discuss them on mailing list and incorporate feedback
    B1.1.3: finish implementation, tests, documentation
Status: B1.1.1 in progress
IssueZilla: 35051
Links: Filesystems Extension
Owner: pnejedly@netbeans.org

B1.2: evaluate compatibility bridge
Part of the already done work is also skeleton of possible compatibility bridge. The scope of what is possible and what not must be evaluated and properly documented.

Dependencies:
B1 does not depend on A1 nor A2 and can be done in parallel with them
B1.2 does not depend on B1.1, but preferably should be done after that

B2: Migrate to Filesystems Extension API

B2.1: cleanup of loaders package
    B2.1.1: deprecate several APIs without replacement: ConnectionSupport, OperationListener, LoaderTransfer and maybe others
    B2.1.2: allocate some time for potential troubles which will appear
Detailed analysis of this task should be done. Needs to be analyzed.

B2.2: Create autoload openide-loaders.jar.
Separate org.openide.loaders.* package and necessary content around into its own JAR file and make the rest of openide.jar independent from it. After that the openide-loaders.jar can be turn into regular autoload module and slowly (after all known modules stop using it) be removed from distribution and just for backward compatibility left on an autoupdate center.
Status: almost DONE
Links: issue 32143
Owner: jtulach@netbeans.org

B2.3: migration of other NetBeans modules
    B2.3.1: prepare cookbook of what and how should be migrated
    B2.3.2: plan some time for support of clients

Dependencies:
This area is not much clear. All tasks depend on completion of C1 area. Also to complete migration from old DS the A2 must be finished too.

C1: Extend or reuse existing subsystems

Detailed analysis about this area must be done first. Below is list of known tasks, but their concrete size and impact is not know. It is possible that some other tasks will appear. Needs to be analyzed.

C1.1: Filesystems library issues
    C1.1.1: evaluate and enhance MIME type recognition process
    C1.1.2: deprecate or create replacement for FileObject.setImportant(boolean)

C1.2: Editor library issues
    C1.2.1: create replacement for DataObject.modified property and DataObject.Registry
    C1.2.2: create FileObject based replacement of DataEditorSupport.
    C1.2.3: create replacement for OpenSupport based on FileObject
    C1.2.4: check whether the FileObject identity must be kept. see issue 28497.
All these tasks touch whole architecture of editor support in NetBeans which itself it buggy and hard to use and might need to be completely rethink as part of this issue.

C1.3: Wizard library issues
    C1.3.1: TemplateWizard shall be replaced by a new system.
This task needs to be analyzed. Existing DS functionality for Template Wizard must be moved to wizard library. The placement of template files depends on A2.7. The templates should be registered in the system in some Registry Context as instances of a "WizardIterator" class.

C1.4: use Looks for visualization
It needs to be analyzed carefully with the HIE team how the presentation of object in NB should look like and according to that this issue can be implemented including the grouping of files.

Dependencies:
This area is not much clear. Most of the tasks in this area are for different libraries and so they can be done in parallel by owners of the libraries. Most the these tasks (apart from C1.1.*) depends on completion of B1, but their analysis can start immediatelly.
C1.3 depends on A2.7 and might depend on A2.6.1

Milestones

The above tasks can be grouped into these milestones.

M1: Registry API - basic implementation ready
- finish A1.1
- status: FINISHED

M2: Registry API & Convertor API - full implementation ready
- finish A1.2
- finish A1.3
- depends on M1
- status: IN PROGRESS

M2B: New validatable module layer format
- finish A1.4

M3: Migration to new Settings API in core platform
- finish A2
- depends on M2
- status: IN PROGRESS

M4: Filesystems Extension API ready
- finish B1.1
- status: IN PROGRESS

M5: MIME type classification system optimized
- finish C1.1

M6: Editor support enhanced
- finish C1.2
- depends on M4

M7: Wizard system enhanced with templates
- finish C1.3
- depends on M3

M8: Visualisation of objects
- finish C1.4
- depends on M4

M9:  Datasystems redesign complete
- finish B1.2
- finish B2
- depends on all previous milestones

According to milestone dependencies it is clear that M2 can run in parallel with M4 and M5. The M6, M7 and M8 can also run in parallel after their dependencies are satisfied.

Current plan is to finish M2 and M4 as soon as possible. And them continue with the rest. Depends on the release schedule.