Datasystems Redesign Plan
This is preliminary plan for Datasystems redesign in core platform. The plan expects that you read following documents:The plan first defines highlevel areas, then goes into more details for each area, and at the end it defines milestones in which the plan can be implemented. The plan is still quite high level and needs to go into more details if realistic estimates are needed. At the moment it just captures all what has to be done, all what is unknown and needs to be analyzed and defines dependencies between these tasks.
If there is no status or owner mentioned it means that no work was done yet and task is not planned yet.
At the moment plan does not contain any tasks for performance evaluation. They have to be added.
Highlevel Areas
The Datasystems redesign plan can be split into these highlevel areas:- A1: Create Settings API
Implement Registry and Convertors APIs. This step will eliminate part of the DS (eg. InstanceDataObject, XMLDataObject) which are used for settings.
- A2: Migrate to Settings API
All usages of DS for settings management must be rewritten to use new settings system.
- B1: Create Filesystems Extension API
Implement Filesystems Extension API. There already exist some implementation. The size of the API is relatively very small. The implementation should be consulted with projects team which already using this concept and any overlapping API should be merged into platform if possible.
- B2: Migrate to Filesystems Extension API
Migrate from old DS to Filesystems Extensions API. Cleanup of org.openide.loaders package.
- C1: Extend or reuse existing subsystems
All existing APIs which are using Datasystems must be extended to work smoothly without them. In practice it means that they must be able to work with FileObject only. Part of this area is also extending these subsystems with features in the past provided by old DS
Areas in more detail
A1: Create Settings API
A1.1: create Registry APIA1.1.1: create proposal & prototype for the module and its public API
A1.1.2: discuss them on mailing list and incorporate feedback
A1.1.3: full implementation, tests, documentation
Status: DONE.
Comment: The Registry API is available for use with one restriction that for object persistence are still used current Datasystems. After the Convertors API is finished the Registry impl will be changed to use Convertors API instead. This will have significant impact on lifecycle of instances created by clients. See A1.3.
IsueZilla: 32679
Links: Registry API
Owner: dkonecny@netbeans.org
A1.2: create Convertors API
A1.2.1: create proposal & prototype for the module and its public API
A1.2.2: discuss them on mailing list and incorporate feedback
A1.2.3: full implementation, tests, documentation
Status: DONE.
IssueZilla: 34671
Links: Convertor API
Owner: dkonecny@netbeans.org
A1.3: rewrite default implementation of Registry to use Convertors API
The major difference is that while old Datasystems saved modified objects automatically the new system requires to explicitly bind object back to Registry to make changes permanent.
This task has external dependency on new projects. They are already using Registry and are using old core/settings convertors and possibly rely on the autosaving behaviour of DS. They have to do two things in one step immediatelly after this task is implemented: #1 rewrite existing convertors to new Convertors API and #2 explicitly save all modified objects
Status: in progress.
IssueZilla: 35050
Owner: dkonecny@netbeans.org
A1.4: create Registry implementation over XML document
Current Registry implementation works still over the Filesystems layer. This tasks defines new format of module layer which is not constrained by XML Filesystem DTD, which is validatable by XML Schema, which uses XML Namespaces, and which is more natural for registration of instances then Filesystems DTD. The format will take advantage of Convertor API. This will eliminate Filesystems layer from settings handling.
A1.4.1: define structure of new XML module layer
A1.4.2: implement Registry for that XML
A1.4.3: write runtime conversion from old module layer to new format
A1.4.4: write binary cache of merged module layers
Dependencies:
A1.3 depends on A1.2.
A2: Migrate to Settings API
A2.1: API enhancements requests and supportPlan some time for API support and API enhancements requests which will surely come after clients start using the APIs.
Status: in progress.
IssueZilla: 34114, 34274, 34353, 34342, 34424, 34551, 34019 and others might come.
Owner: dkonecny@netbeans.org
A2.2: create support for merged Registry Contexts
This is needed to fully replace SystemFS by default Registry. That will make NetBeans backend configuration system independent on Filesystems and will allows us to extend format for declarative registration.
A2.2.1: create analysis: requirements, masking of deleted bindings, etc.
A2.2.2: full implementation, tests, documentation
A2.2.3: change default registry to be merged Registry Context of installFS context and customizationFS context
Status: A2.2.1 finished; A2.2.2 in progress
IssueZilla: 34057
Links: Merged Contexts chapter on this page
Owner: rmatous@netbeans.org
A2.3: create replacements for several existing openide support classes
A2.3.1: support for Menu based on top of the Registry Context
A2.3.2: support for Toolbar based on top of the Registry Context
A2.4: rewrite everything in core and openide to use settings system directly or the helper classes from A2.3
The detailed scope of this issue is not known at the moment. It is possible that "something" what was not expected will appear.
A2.5: default lookup
Default look will be based only on META-INF lookup. The content of /Services folder from SystemFS will be removed from it. These services will be accessed directly by Registry API.
Links: Some thoughts about this task.
A2.6: reimplement settings presentation
Settings presenter (ie. options dialog) must be internally reimplemented to use Registry. The major difference is that modified objects are not persisted automatically. There are two types of objects under UI/Services which will be distinguished: Node instances and JavaBeans instances. The Node instances must take care of persisting objects they are visualizing when some property has changed. The JavaBeans instances (eg. SystemOptions, services, etc.) will be wrapped into a special Node which will listen on bean properties and which will bind modified object back to the Registry.
A2.6.1: create ContextLook as replacement of DataFolder.FolderNode. Can be reused by TemplateWizard.
A2.7: resolve access to files on SystemFS
The SystemFS contains also files which clients need to access and which are at the moment not accessible by Registry API, eg.: DTDs, parser databases, templates, etc. For more details see analysis of this task below. Originally this was thought to be difficult problem, but after analysis only two quite small issues remain. See the analysis for more.
Links: analysis of this task
A2.8: migration of other NetBeans modules
A2.8.1: prepare cookbook of what and how should be migrated
A2.8.2: plan some time for support of clients
Dependencies:
A2.4 depends on A2.3
A2.6 depends on A1.3
A2.7 does not depend on anything
A2.8 depends on A1.3 and better wait till most of the A2 tasks are finished
Most of the A2 tasks (apart from A2.3 and A2.4) can be done in parallel by different people.
B1: Create Filesystems Extension API
B1.1: create Filesystems Extensions APIB1.1.1: resurrect work already done in this area; merge API overlapping with project ones
B1.1.2: discuss them on mailing list and incorporate feedback
B1.1.3: finish implementation, tests, documentation
Status: B1.1.1 in progress
IssueZilla: 35051
Links: Filesystems Extension
Owner: pnejedly@netbeans.org
B1.2: evaluate compatibility bridge
Part of the already done work is also skeleton of possible compatibility bridge. The scope of what is possible and what not must be evaluated and properly documented.
Dependencies:
B1 does not depend on A1 nor A2 and can be done in parallel with them
B1.2 does not depend on B1.1, but preferably should be done after that
B2: Migrate to Filesystems Extension API
B2.1: cleanup of loaders packageB2.1.1: deprecate several APIs without replacement: ConnectionSupport, OperationListener, LoaderTransfer and maybe others
B2.1.2: allocate some time for potential troubles which will appear
Detailed analysis of this task should be done. Needs to be analyzed.
B2.2: Create autoload openide-loaders.jar.
Separate org.openide.loaders.* package and necessary content around into its own JAR file and make the rest of openide.jar independent from it. After that the openide-loaders.jar can be turn into regular autoload module and slowly (after all known modules stop using it) be removed from distribution and just for backward compatibility left on an autoupdate center.
Status: almost DONE
Links: issue 32143
Owner: jtulach@netbeans.org
B2.3: migration of other NetBeans modules
B2.3.1: prepare cookbook of what and how should be migrated
B2.3.2: plan some time for support of clients
Dependencies:
This area is not much clear. All tasks depend on completion of C1 area. Also to complete migration from old DS the A2 must be finished too.
C1: Extend or reuse existing subsystems
Detailed analysis about this area must be done first. Below is list of known tasks, but their concrete size and impact is not know. It is possible that some other tasks will appear. Needs to be analyzed.C1.1: Filesystems library issues
C1.1.1: evaluate and enhance MIME type recognition process
C1.1.2: deprecate or create replacement for FileObject.setImportant(boolean)
C1.2: Editor library issues
C1.2.1: create replacement for DataObject.modified property and DataObject.Registry
C1.2.2: create FileObject based replacement of DataEditorSupport.
C1.2.3: create replacement for OpenSupport based on FileObject
C1.2.4: check whether the FileObject identity must be kept. see issue 28497.
All these tasks touch whole architecture of editor support in NetBeans which itself it buggy and hard to use and might need to be completely rethink as part of this issue.
C1.3: Wizard library issues
C1.3.1: TemplateWizard shall be replaced by a new system.
This task needs to be analyzed. Existing DS functionality for Template Wizard must be moved to wizard library. The placement of template files depends on A2.7. The templates should be registered in the system in some Registry Context as instances of a "WizardIterator" class.
C1.4: use Looks for visualization
It needs to be analyzed carefully with the HIE team how the presentation of object in NB should look like and according to that this issue can be implemented including the grouping of files.
Dependencies:
This area is not much clear. Most of the tasks in this area are for different libraries and so they can be done in parallel by owners of the libraries. Most the these tasks (apart from C1.1.*) depends on completion of B1, but their analysis can start immediatelly.
C1.3 depends on A2.7 and might depend on A2.6.1
Milestones
The above tasks can be grouped into these milestones.M1: Registry API - basic implementation ready
- finish A1.1
- status: FINISHED
M2: Registry API & Convertor API - full implementation ready
- finish A1.2
- finish A1.3
- depends on M1
- status: IN PROGRESS
M2B: New validatable module layer format
- finish A1.4
M3: Migration to new Settings API in core platform
- finish A2
- depends on M2
- status: IN PROGRESS
M4: Filesystems Extension API ready
- finish B1.1
- status: IN PROGRESS
M5: MIME type classification system optimized
- finish C1.1
M6: Editor support enhanced
- finish C1.2
- depends on M4
M7: Wizard system enhanced with templates
- finish C1.3
- depends on M3
M8: Visualisation of objects
- finish C1.4
- depends on M4
M9: Datasystems redesign complete
- finish B1.2
- finish B2
- depends on all previous milestones
According to milestone dependencies it is clear that M2 can run in parallel with M4 and M5. The M6, M7 and M8 can also run in parallel after their dependencies are satisfied.
Current plan is to finish M2 and M4 as soon as possible. And them continue with the rest. Depends on the release schedule.