The applet's first matrix shows four groups of modules, each in its own color. As can be seen the modules are grouped by its expected clusters. There is a platform at the bottom, ide above it, followed by java support and then apisupport. As can be noticed, there are certain dependency problems, as some of the java modules are inserted in middle of the ide group. These problems need to be solved resolved prior to split of ideX cluster as our build system does not allow cyclic dependencies:

• lib/jsch - depends on Ant module. Needs to be removed, obvious fix is to split the module into two parts, but probably a better one is to let this module do some kind of declarative registration (maybe library for Ant?) and let some other module find such declaration. This would on one hand create a soft dependency, on the other it would keep the declaration next to the library and could be useful in other places as well. Fixed[77489]
• xml/tools - provide support for generating skeleton for SAX parser, etc. this indeed needs to be part of the java cluster. The solution is to split the module into two parts, make the one with the tool eager depending on the xml/tools module and java infrastructure. [77491]
• apisupport/project - has implementation dependency on xml/tax. Need to change to friend, at least. [77492]
• httpserver - btw. the httpserver contains some old and obsolete copy of tomcat. I guess tomcat 3. Independent, but should be fixed, if possible the httpserver could be moved to enterprise cluster where guys could integrate well with never version of tomcat. The problem however is that currently XSL module is depending on httpserver and as such it would be neccessary to split the module into api and impl, probably. [77493]

The visualization shown in the applet is based on artificial commit verification files generated by special build script which defines the actual groups of modules by use of Ant's selectors. This build script generates golden files into a directory which is then read by the applet. The most important files there include the list of modules and clusters they belong to and also the list of dependencies between individual modules.

Why this split is good?

This split is good as it satisfies most of the critical. Let's see. Let's assign value 9 to every critical which is High, 3 to those that are Medium and 1 to all Low. If the proposed solution solves the problem stated by the critical, we get the points, otherwise we get negative amount of points. Here is a table evaluating the state:

As can be seen from the total sum, the current proposal gets 28 points from 32 possible and that means it does not satisfy just low priority criticals. That is the reason why it is believed that this proposal is good enough to be implemented.

Clusterization of NetBeans IDE

From a set of currently available modules we generate set of golden files describing dependencies and other important global characteristics. See CVS for text format of those files or visualize the current dependencies by following applet applet:

The applet provides interactive layout and visualization of modules and their dependencies. There are two views. The dynamic one presents each module as represented a colored box. Modules in the same cluster share the same color. Dependencies between modules are draw as lines between modules and cause the modules to be attracted together. One can drag and drop module boxes to rearange the layout and make it more reasoanable. The matrix one shows individual modules as lines, sorted by default according to their dependencies and clusters they belong to, and allows various inspections and manipulations on top of the data. Both views are connected and changes made in one, shall be visible in the other as well.

Functional dependencies between different parts should be expressed on a level of individial modules (Java Debugger depends on Java Support and on Debugger Core), but is itself included in a Debugger cluster which may work in a product without Java support (aka C/C++ studio). This is fine, as the clusters main goal is to group chunks of related functionality that will be developed and maintained as a whole and will be kept compatible as a whole. For that reason there may be mutual interdependencies between clusters - module in one cluster can depend on module in another one while there is another pair of modules with reverted dependency. This is fine and probably unavoidable, the only necessary thing is to remember the transitivity problem and avoid it.

Solution A: RPM as functional pack

Please note that based on discussion in installation proposal document it is not possible for an RPM package to span over multiple (compatibility) clusters. E.g. RPM containing Java cannot include part of XML support. This immediatelly implies that we cannot satisfy the extensibility requirement. On the other hand it is easy to package one functional pack into one downloadable file as RPMs are already archives.

Also from the nature of RPMs, it seems that they should more likely be managed by native operating system management tool (not sure what that should be on Windows) by system administrator. That delegates the responsibility of (un)installation and restoring to the operating system support. Probably restorable updates could be done via solaris patches, which keep previous version and thus support restore. Enabling/Disabling of some functionality would probably need to be supported by NetBeans directly even it is not clear whether it would not be better to abadon it completely in favor of (operating system supported) uninstallation.

This kind of solution seems to reuse a lot of implementation from operating system and thus save us a lot of work, based on assumption that there will be some kind of packaging system available for Windows. It nearly removes the need to have AutoUpdate support in the product at all. On the other hand it limits the possibilities choosing functional pack (cannot be bigger than compatiblity cluster) and does not cope well with temporary disabling of some pack.

Solution B: Module based approach

The UI shall just show only regular modules (e.g. functional packs) and hide autoloads and eager ones as they will be the implementation detail. There should probably be an expert mode allowing manipulation of the (usually invisible) modules allowing power users to turn autoload modules into real ones and completely configure their system.

The beauty of this system is that it is completely orthogonal to the compatibility clusters and gives absolute configuration power not only to skilled users, but also for product assemblers as they can preconfigure their own regular modules in groups they like and change the status of others provided to them by other groups, which was not possible in previous proposal.

This proposal easily satisfies requirements of multiple files in a pack (by declaring dependencies on them), it has no problems with enabling/disabling such packs. It however does not simply satisfy one downloadable file as the actual module JARs are packages in various RPMs (remember the orthogonality) and only thing one could do is to ZIP all necessary RPMs that form the functional pack and let user unzip them (but of course they would very likely contain a lot of other unrelated stuff), so this is not really good solution. The uninstallation and restoring of a feature would be probably better to leave to operating system and its administrators. Only they could uninstall or restore RPMs with certain functionalities using native tools.

Extensibility is where this proposal excels. Not only it gives absolute freedom to product assemblers to create functional packs of their own, but also allows third parties to extend such a defined packs. For example: if there is a module defining XMLSupport (which depends on xml/tree, xml/edit, etc. autoloads) one can define an eager module depending on the XMLSupport and enabling the third party functinal modules. That way whenever a user enables XMLSupport, not only things defined originally but also any other extension willing so, will be enabled. This seems like very desirable behaviour for a dynamic and modular system like NetBeans.

Verification Framework

Goals

• [lessimpldep] We need to minimize the amount of implementation dependencies - they are hard to increase after a release (because they are always done in some specific way)
• [noimpldepbetweenclusters] We need to prevent implementation dependencies between clusters - this limits the amount of combinations we can do with clusters and as they are supposed to be compiled separately, we cannot have such restrictions
• [nocyclesbetweenclusters] We need to prevent cyclic dependencies between clusters - otherwise we will not know how to compile them (module system can handle this, but the build system does not and probably should not)
• [comparegraphs] We need to compare two dependency graphs together - otherwise we will not find out that something has changed
• [fileslayout] We need to be informed about changes in files layout - the change of layout is usually sign of some bigger changes somewhere else and is one of the most useful inputs for any review
• [dailytest] We need to run the comparing test dailly - probably as part of daily build and we need some kind of notification that it failed
• [personaltest] We need to run the test as part of every developer build - this can catch unwanted changes early, commiters cannot complain that they did not know
• [allowchanges] We need to allow changes of module dependecies - of course certain changes are sometimes acceptable, and there should be a way how to enabled them
• [education] We all need to understand that dependecies are important - the knowledge of the importance of dependecies between modules does not seem to be too big and this restricts usability of our product (html editing requires mdr), we need to help everyone to understand how important this is
• [simpleincrement] We need to simplify update of dependencies - currently the dependencies take significant amount of time to be updated when a new release branch is created. This is due to non-standardized way implentation dependencies are used and also due to non-trivial way how to increase certain digit in specification version of each module in release.
• [listpublicpackages] We need to check list of public-packages - if we decide to eliminate implementation dependencies, we will need to replace them by something, that something means real API and this means to leads to opening certain module packages and declare them as public. This shall not get out of control as it is an API and needs careful review.
• [publicpackagesapis] We need to check api changes in public-packages - right now we do sigtests just on official namespaces, but apis in public packages are important as well (not that important, but important). We especially need to prevent accidental changes there.
• [genereatedocs] Architecture documentation needs to be in sync with the reality. Documentation is an important part of API and it would help a lot to automate generation of module dependencies information into arch documents. Otherwise we are reviewing what people tell us they did, not what they have really implemented.

Proposed Solution

The general idea is to implement a special ant task in the nbbuild CVS module that will be configurable using its properties and will generate output that can then be compared using regular text tools to find out whether or not there are some differences.

The input of the task will be a set of module JAR files or possibly also list of projects from where the task will read its dependency informations. The list of JAR files is useful for checking the dependencies on the produced build and will have to be implemented since begining, reading of the project.xml can be useful for genereatedocs goal, but as that is not the highest priority, it can wait for a while.

The task will be configurable to produce various output formats and write them into given output file. The possible outputs will include list of all public packages (useful for listpublicpackages and publicpackagesapis, if the list is then fed to a signature processing tool), list of all modules and their versions (this will notify us that there is a new module added), list of all modules and their dependencies including implementation ones (it will not help directly to have lessimpldep but at least the amount will not grow, and also will notify us about changes in dependencies).

The task will have to understand clusters in order to check for cluster dependencies noimpldepbetweenclusters and prevent nocyclesbetweenclusters. Probably the task shall be able to output dependencies on the level of clusters and not only modules.

The output format will have to be stable during runs, so we can use plain diff to check for differences and comparegraphs this means that the tool will need to sort the elements by something immutable. For example by module name.

In order to enable personaltest the test will be performed as part of build (or commit validation) and the results will be compared to a golden files stored in CVS. If a difference is found the build will fail to warn the developer that he is doing something imporant. But as we need to allowchanges, the location of the golden file will be printed and the developer allowed to modify it as well. This will help with education and improve the understanding that modification of modules dependencies in an important part of API.

For the version that does the dailytest a special property overriding the location of golden files will be given and instead of CVS the test will compare the changes to the result of previous daily build. If the test fails, the report will be sent to api-changes@netbeans.org, so everyone interested is notified about the changed made during the day.

The check for fileslayout is similar but does not need the special task. It can be composed from standard ant tasks that will generate the file with the layout and compare it to the golden file in ide/golden/files-layout.txt.

The simpleincrement goal is desirable, but addressed just indirectly by preventing the number of implementation dependencies to grow. In order to simplify the increment after branching, it is suggested to include in each manifest a special postfix of specification version. So the version would be composed from regular part 1.2 and a special one ${specversion.postfix}. This would be by default empty. After creating a branch we would change nbbuild/build.properties to define the value as specversion.postfix=.1 and this would immediatelly increment all spec versions in all modules on the branch. However the increment in trunk would still have to be done by hand.