Modularization of the Platform
$Revision: 1.48 $
Changes: available in
CVS
Status: issue
19443
and colors in the text -
red means not done yet,
yellow means doing it now,
green is for done items.
Some paragraphs are marked as
grey which means
they can be finished later in trunk.
Abstract:
The heart of any application based on NetBeans is build around openide and core
which (althrough the overall NetBeans architecture is very modularized and
general)
are themselves more monolithic than they should be. The current state
complicates certain things in infrastructure, makes other hard to achieve
and finally contributes to poor module developer experience.
The modularization effort described in this document is going to solve these
problems and improve current situation.
Requirements
- Compile independently independent parts of the
application. This increases stability and improves the code base, as it
avoids unwanted hacks used by indepenedent parts to work around some kind
of communication problem (for example explorer calling into loaders
or filesystems to handle some special state).
- Independent JavaDoc - the logical pieces
of openide has to form independent modules in API reference in the same
way as for example project modules do. Then the module writer
will not be confused by thinking openide is something special.
- Package independent functionality into independent
JAR files. This improves the customization - JARs can be removed, added more
easily than to do any changes in code. This is needed in order to satisfy
(oposite) requirements from different products builds on top of NetBeans
platform and given them a bit of freedom in adapting new functionality in
favor of deprecated one (for example replacement of loaders, remove of
TopManager, replacement of old actions system).
- Standard Infrastructure - there
is a default infrastructure to build NetBeans module in nbbuild and also
in apisupport. Currently it is not used to build openide and various
hacks treating openide as something special are spread in unexpected
places. By structuring openide parts in standard way we can remove those
hacks and safe many infrastructure maintanance nightmares.
- Independent JARs shall be converted into
real modules and not be statically added to classpath. This allows step by
step deprecation of obsoleted functionalities. If packaged as module
autoload JAR, such functionality is enabled only when needed (there is a
module requesting this deprecated one), thus increasing the compatiblity
without preventing future improvements and deprecations.
- The heart of NetBeans shall be really
small and include only minimum of classes needed to boot the module system.
This is the final phase of making JARs independent which gives the configurators
of a system absolute freedom in selection of the functionality that they really
want to use from NetBeans Platform.
Content
- Compile Time Separation
- Independent JavaDoc
- Independent JAR files
- Modularization and Miniaturization
The first step, useful of its own, but definitively necessary for the other
requirements to happen is to separate the compilation of opende/src into
separate parts and thus ensuring that there are no hidden interdependencies
between them. The status of this part is tracked under
issue 34758
and is available in branch compile_34758 of openide.
Compile time separation (phase I)
Let's modify the openide/build.xml to compile sources
not all at once, but chunk by chunk. This is possible by passing
javac an argument --sourcepath ''
which disables javac's search for
files to be compiled. Benefit: we will understand and separate the openide
to pieces without breaking other people ability to check in (work will
be done on a branch, but as the only modified file will be build.xml, it
will be easy to merge the branch, not as big problem as when doing
openide/loaders).
CVS separation (phase II)
When phase one is finished, the CVS will be closed for one day and all
sources moved to new locations. That will be easy, as the locations will have been
defined by the build.xml "chunks".
While moving the sources, we will loose history, but it will remain accessible
in the old CVS location, if really needed. Moreover as other moves we did (like loaders) show,
the history is usually needed just for one release back, rarely more. So loosing
it is not that big pain comparing to current problems that it is going to solve.
The sources will be formated using jalopy to follow coding standards
(visit issue 57941
to see the patch that does the move and formating). Reformating
usually hurts history, but because
we loose it anyway, we can exploit this chance to improve our current
mess in source files.
For each openide subproject a standard NetBeans project type will be created,
initially creating its JAR in extra cluster, as the current openide/build.xml
will stay and package the final openide.jar file as we do now. The main
build script in nbbuild will contain new targets for openide subproject, but
the all-openide will remain there and will depend on all of the
subtargets.
All of this will be integrated to trunk at once during a night to not
disturb the majority of developers working on openide.
Together with the source code CVS changes, the openide javadoc generation
will be moved to the separated parts. This will allow us to remove special
hack support for
openide in nbbuild/javadoctools and will provide consistent
view of all NetBeans javadocs to module developers at
reference API page
.
Also the indexing and overview presentation of all javadocs will be allowed
to be enhanced as there will now be a common and standard way how a
module delivers its API packages and documentation.
As part of this split we have to remove the huge introduction page in current
javadoc and move its appropriate sections into individual arch-*.xml
documents. If put into arch-what section, they will appear in the
reference API page
.
The apichanges.xml will have to be separated into individual documents
each for one submodule. The spec version of each created module will start at
6.2.
nbbuild/build.properties will be modified to build javadoc from
openide submodules and not the whole openide one.
We introduced new naming scheme. Each API can be referenced by using
@org-netbeans-code-base-name@ of the module. That is why we need to get
rid of the old @MODULE/SUBMODULE@ links in API docs. We will do it by
reviewing all javadocs and fixing broken links. Owners has been informed
to fix their javadocs.
As soon as all javadocs are fixed, we will change the build-javadoc to verify
that the whole javadoc of all modules is consistent and without broken links.
Otherwise the build fails. That way we prevent regressions in future.
At some point (not necessarily now), I think we should move e.g.
openide/arch/arch-openide-windows.xml to openide/windows/arch.xml. However we
also need to clean these up seriously: many of them describe not just the API,
but also the impl in core. There should be e.g. a separate core/windows/arch.xml
that describes what the impl module does.
When every source will have its own new place and the logical parts will be defined
the huge openide.jar can be split to smaller parts (e.g.
org-openide-filesystems.jar, org-openide-nodes.jar, etc.). Beyond
improving the customization by allowing
individual JARs to be removed, this jarification
will be the major step
towards turning the JARs into modules.
This is likely a step with very small effect, but huge affect. Every build
script refering to openide.jar will have to be updated to refer to
all or only appropriate parts of openide-*.jar it needs.
Based on the previous work in
issue 36494 a FixDependencies ant task was created
that makes it easy to replace and reduce dependencies for every
projectized module. Just execute
ant fix-dependencies and your nbproject/project.xml
will be upgraded. Still we need to improve the
task to modify only dependencies that are used for compilation, not
those that are there only for runtime.
Move fix-dependencies to common.xml module so other applications based
on NetBeans and using the apisupport harness can invoke it as well.
Just splitting openide.jar into smaller JARs has potential performance
impact. The startup will be slower (last time we tried it was by 10%).
The performance guys
agreed that this is ok, if it gets resolved by release. And it will because
the modularization and miniaturization described later will return the
time back to previous values. If not, we will return back to one gigantic
openide.jar.
When the JARs are separated it is time to turn them into modules. This however
requires that there are not hidden mutual dependencies between those JARs as
they will be loaded by own classloaders and that the other part of platform the
core.jar does not depend on them or can be turn into modules as
well.
The ultimate goal of all these efforts is to spit out a miniature part of the
core in NetBeans (the module system and few other utilities) and create a
modular application platfrom bootstrap.
This requires changes to module system and also to startup sequence that can
contribute to its clean up and clarification. The result will be small
microkernel bootstrap with nearly no overhead capable of bootstraping any
NetBeans application customized to absolute degree.
We already have core/bootstrap which contains classloaders
for openide and core, and the plan is to move into it also the module system.
Together the openide/util, openide/modules, openide/fs and boot.jar would
stay on dynamic or even real classpath and the rest of the openide modules
plus core would be turned into real, dynamically loaded modules. The current
thinking about the layout of platform6 directory is:
lib/boot.jar contains what it has now,
together with part of what is now org.netbeans.core.modules.*
the part that is already marked as being independent of other NB APIs
.
lib/org-openide-util.jar will next to boot.jar to provide
the basic infrastructure used by boot.jar.
lib/org-openide-modules.jar provides the APIs that the
boot.jar implements.
core/core.jar will be loaded dynamically
by the boot.jar and will contain most of the startup and the
rest of org.netbeans.core.modules. For its implementation it will need
filesystems which will be in the same folder as well.
core/org-openide-filesystems.jar the filesystem APIs needed
for the standard launch code to read module config files and listen
on their changes.
modules/org-openide-nodes.jar and other openide libraries
will be turned into real modules.
modules/org-netbeans-core.jar will also be a module
and will contain the rest of core, or if possible just parts of it
if it gets split even more (like moving the UI part to core/ui, etc.).
Currently there is core/org-netbeans-swing-plaf.jar in the
dynamic classpath. It is not too nice and maybe it is not that necessary.
Consider calling its methods later and making it real module. This could
be done by adding new methods to CoreBridge, implementing them in CoreBridgeImpl
and calling the methods in the plaf library. The core method would then be
called after initialization of module system.
Sources should be rearanged to reflect the logical partitioning in the launcher.
There should be three sets of classes, each in their own package (we can
move them, as they are not part of supported API). Having them in own
packages will prevent linkage errors when two classes loaded by different
classloaders try to access their package private methods. The sets are:
- independent module system - placed on java classpath in directory
lib in boot.jar. All the sources will be hosted
in CVS module core/bootstrap and in addition to existing ones, they will
also contain the generic part of core module system. The list of these
classes is defined in
core/build.xml in target nb-modules. They are
currently in org.netbeans.core.modules package, but
as that package is going to be split in half, those classes
shall be repackaged to org.netbeans, so all classes
in boot.jar are in the same package.
- NetBeans specific module system - the part of NetBeans launcher
that recognizes NetBeans format of modules, knows their locations,
shows splash screen with progress and does other NetBeans specific hacks
during startup. The list is again defined in
core/build.xml as selector
nb-modules. They are spread
in various packages with entry point in org.netbeans.core.Main.
They shall be moved to one separate packages (
org.netbeans.core.startup, org.netbeans.core.startup.modules,
org.netbeans.core.startup.layers)
to prevent linkage errors with
other classes. The choosen name is org.netbeans.core.modules
and the name for their final jar file is core/core.jar. The
CVS module for these classes is going to be core/startup.
The only possible problem is the move of the Main class, as some applications
might depend on it, but as it was not part of public API, we should not
really care. We should still do grep on our sources to search for usages
of org.netbeans.core.Main and org.netbeans.core.Plain
and making sure they continue to work.
-
The rest of the original core classes will remain in the same places
where they are (for now, in future they can be moved more) and will be
contained in
modules/org-netbeans-core.jar module.
TestModuleDeployer was changed to class with static methods and is
kept in org-netbeans-core.jar.
The communication between boot.jar and core.jar is done using property
netbeans.mainclass that will point to org.netbeans.modules.Main.
The communication between core.jar and rest of the modules is
handled thru CoreBridge calls, regular ModuleInstall
and later thru RunLevel interface.
The ParseXMLModule would be changed to correctly insert compile time classpath
entries for everyone depending on org.openide.
Meaning that keeping a dep on "org.openide" would result in compilation
against all split JARs (so it would succeed), yet the JAR would be
created with the old OpenIDE-Module-IDE-Dependencies as before, thus
still producing a runtime warning. We would
still need to do a batch conversion of project.xml's for trunk modules
to state minimal specific deps.
There is a ModuleAutoDeps still in effect for anyone missing
an org.openide dep, since the module system currently assumes the
equivalent of "IDE/1 > 0.0" if not otherwise stated, for compatibility.
This will have to be changed (and modules which are really libs
and do *not* use any org.openide.** will no longer need to state
the phony dep). We will have to get rid of the compat trick used
in
http://www.netbeans.org/download/dev/javadoc/OpenAPIs/org/openide/doc-files/upgrade.html#3.5i-sep
but this shouldn't matter much since the modules referred to by the ModuleAutoDeps are long gone.
We should probably get rid of OpenIDE-Module-IDE-Dependencies altogether - deprecate the keyword.
A warning will be printed if the tag appears in the module's manifest, the
ParseProjectXML will not deal with that tag anymore. Remove all but the last
auto dependency in openide/modules. Add URL to the warning about openide ide split.
Sometimes we have used IDE/1 versions to signify changes not in any class libraries,
but
in the module JAR format itself. We will define special token and
use Requires vs. Provides for this.
We use:
OpenIDE-Module-Requires: org.openide.modules.ModuleFormat1
and let the module system just automatically provide e.g.
org.openide.modules.ModuleFormat1, org.openide.modules.ModuleFormat2, and
org.openide.modules.ModuleFormat3 for all modules, if it were currently
at rev 3 of the module format.
Performance measurements show that the
startup time exhibits less then five percent
regression. This was not merge stopper
as there are things that can and will be done
to improve the startup time: We got rid of
org.openide.compat, then org.openide.explorer
and org.openide.util were made sealed packages
and the startup, also thanks to removal of old
vcs modules got under control.
One has to check that autoupdate continues to work, that we can enable, disable
module. Disabling of a module is part of commit validation,
autoupdating of ant docs with old dependencies seems to
work fine, dependencies upgraded, ant help included in our
help system.
Unit tests and commit validation has to pass successfully.
Describe startup sequence somewhere, probably in core/arch/arch-launcher.xml.
Rewrite upgrader.jar to use plain swing. Now it relies on JOptionPane.
Comments to
nbdev@netbeans.org please.
