NetBeans - API & Module Versioning Policy

History: available in CVS

Abstract:

If there were no external modules that could run on NetBeans, and we did not have the Update Center to let users mix and match modules, then handling versioning in NetBeans would be very easy: nothing more than labelling when every module was built. However, we do have these concerns. So some guidelines are required to make sure that the module versioning system is capable of matching our practice. Issues include labelling of APIs according to version numbers; specifying levels of compatibility; depending on other modules; marking releases for Auto Update; providing APIs from modules; and so on.

Contents:

1. Versioning of APIs

1.1. What is an API change?

1.2. How to make an API change

1.3. How to use an API change

1.4. Providing a module API

1.5. Keeping track of API changes

1.6. Using a module API

2. Versioning of user data

3. Numbering Scheme for Updates

Versioning of APIs

The Modules API includes a detailed description of how versions and dependencies work technically. This documented is intended to give a policy for how to use them on netbeans.org.

What is an API change?

The simplest case of an API change is anything that changes the public or protected signature of a public or protected class, that is a signature change which would appear in Javadoc and possibly affect clients of the API. These may be compatible or not, depending on whether any client code would be required to change in order to continue to run in binary form; and in order to compile in source form. Note that changing the set of unchecked exceptions documented to be thrown by a method, or changing whether a method is permitted to accept or return null, counts as an API change and may be compatible or not.

More subtle changes include behavioral changes where the behavior is specified in API documentation. For example, if a method is documented to call some other protected method inside its dynamic scope while holding a particular lock, ceasing to hold that lock or ceasing to call that other method would be an API change, potentially incompatible. These kinds of changes are harder to evaluate, so be careful to read the existing documentation; and when adding new documentation be careful to say exactly what you intend the behavior to be, and if the documentation includes anything that you expect could change and should not be relied on, say so. For example:

/**
 * Open this widget.
 * Will be called while holding widget control lock.
 * <p>In the current implementation, this uses {@link #createWidget}
 * but that may be changed in the future.
 * @return the opened widget
 */

How to make an API change

API changes must not only be documented, they must also be matched to API versioning, so module authors can programmatically depend on them.

Compatible change on the trunk

The safest possible sequence of steps for making a backwards-compatible API change is this:

1. Go through an API review and get approval for the change.
2. Make sure you have a CVS working directory of the appropriate module(s) checked out - do not commit changes until later. Do not make changes in client module code to use the new API yet, if you were planning to - at least keep a copy of the existing module source for the IDE. This is to ensure that a standard set of modules continues to work with the changed API without themselves being changed.
3. Make the change in your working copy of sources. If the change adds a new class, method, etc. which will be visible in Javadoc (public or protected), or changes the behavior of a documented object, please make sure you document what it is supposed to do in Javadoc (its contract, not details of implementation).
4. Increase the specification version in your module's manifest. If the previous version was 1.3, change it to 1.4, i.e. always increase the last number in the version. Remember that the version after 1.9 is 1.10, not 2.0.
5. If the API change involved adding a class, method, etc. to the APIs that will appear in Javadoc, add a @since tag mentioning the new module name and specification version. For example: @since org.netbeans.modules.foo/1 1.4. If the documented behavior of an existing object is being changed, mention this as well, for example: @since org.netbeans.modules.foo/1 1.3; as of 1.4, resulting list may also be modified. If an object is deprecated, say when, e.g. @deprecated As of org.netbeans.modules.foo/1 1.4, the other constructor is preferred.
6. If there is prose API documentation describing the API you are modifying at a higher level, please consider updating this as well, if it needs it.
7. Use Build | Generate Javadoc from the IDE to build documentation for the module and view it. Look over the changed docs.
8. Update your apichanges.xml to mention the new API change. Insert an entry with the appropriate API and class name, label it with the date and new specification version, and give a description of the change and any suggestions for how or why to use it.
9. Build and test the whole IDE - note this is with the API change made but no one yet using it.
10. For changes in client modules to use the new API, see below.
11. Run cvs diff to verify all changes. If the output is messy and hard to understand (e.g. unrelated parts of code reformatted for no reason), stop! Revert all unneeded changes, and again build and test the IDE, and diff again.
12. Commit the API change in one CVS commit: all affected source files, the changed manifest, apichanges.xml, and any other affected documentation.

Compatible change on a branch

For changes made on experimental branches to test whether a new API can support other development on that branch, there are no special requirements: change what you like, but remember to follow the steps listed above when merging into the trunk.

API changes in release (stabilization) branches are discouraged and should only be made when they are required for some priority bugfix. They should of course be made in the trunk as well. The procedure is similar to that listed above for trunk changes; however:

• You will be creating a different specification version on the branch, e.g. 1.3.3 from 1.3.2.
• Mention both the branch and trunk versions in all places where a version number is requested above. E.g. @since 1.4 and 1.3.3.

Incompatible change

Of course you should avoid making incompatible API changes unless really necessary. But, if you do, do it right. Do all steps needed for compatible changes, and also:

1. Make sure you have an API review that authorized the incompatible change.
2. Increase the major release number in the module's manifest; for example your code name could change from org.netbeans.modules.foo/1 to org.netbeans.modules.foo/2. The specification version should conventionally be increased e.g. from 1.10 to 2.0 as a mnemonic.
3. If there are any other modules on netbeans.org which depend on your module's API, change them as well in source. Ask for help from other module owners as needed, but you must make the physical change.
4. Build and test the whole IDE, from scratch (clean build), and be careful.
5. Commit all changes (to your module and to other modules depending on it) in one CVS commit.
6. Notify dev@openide.netbeans.org of the change, and how clients of your module's API should be changed to work with the new version.

How to use an API change

A module should in general explicitly declare the version of every API-providing module it requires in its manifest. It is a developer's responsibility to maintain the accuracy of this dependency at all times. For example, your project.xml might list:

<dependency>
    <code-name-base>org.apache.tools.ant.module</code-name-base>
    <build-prerequisite/>
    <compile-dependency/>
    <run-dependency>
        <release-version>3</release-version>
        <specification-version>3.12</specification-version>
    </run-dependency>
</dependency>

to request version 3.12 or higher of the Ant module API. The IDE will forbid a user to install it if an older version of the Ant module is present (or if the module is missing altogether).

If you have made a compatible API change according to the steps above, you may now use it in your module. Make any code changes to use the new API. Also change your project.xml to require the new version. Build and test the IDE including your module with its new changes, run cvs diff, and commit the code changes and project.xml in one CVS commit.

Avoid increasing your dependency on the API version arbitrarily, as it would prevent a user interested in trying out a new version of your module from running it in an older build (such as the last release version). Of course, if you are not sure which new APIs you might be using, to be safe request the newest API version.

Providing a module API

In order to provide an API from your module for the use of other modules, please follow these steps:

1. Make sure your module code name has a major release version. For example, OpenIDE-Module: org.netbeans.modules.foo/1.

   This ensures you have a mechanism for indicating any incompatible changes later. If you forget, no major release version is similar to -1.

   Convention is to initially use /0 for a freshly introduced API. Clients should depend on your.module/0-1. If you stabilize it in a subsequent release, change it to /1. If you find it was mistaken and have to break compatibility in a subsequent release, change it to /2.

2. Make sure your module declares a specification version. For example, OpenIDE-Module-Specification-Version: 1.7. (You should use the Versioning tab of your project properties to manage this.)
3. Decide on some subset of your module's classes that will form an API. Clearly the smaller and simpler this subset, the better.
4. Place all API-related classes into a special package or package structure in your module that is clearly distinguished from the rest. The convention is to name the package to include api or spi, and to indicate degree of binding to the module implementation. For example, if the private implementation of your module is in org.netbeans.modules.foo (and subpackages), you may use these packages (and their subpackages):

   org.netbeans.api.foo

   Client APIs for the general functionality you provide. Such APIs are assumed to not be closely tied to the implementation of your module, i.e. a quite different implementation could in principle (or fact) support them.

   org.netbeans.spi.foo

   As above, but service-provider APIs, and supports (common implementation bases and defaults).

   You may wish to host support classes separately from "pure" SPIs.

   org.netbeans.modules.foo.api

   Client APIs which are bound in meaning to specific services your module provides. Consider exposing these only as a "friend" API to a selected set of modules.

   org.netbeans.modules.foo.spi

   As above, but service-provider APIs.

5. Physically restrict other modules from using packages outside your API area; see the Modules API for details on how to do this. Use <public-packages> or <friend-packages> in your project.xml.
6. Write clear and complete Javadoc comments for all publically accessible members in the API package.
7. If additional specifications of module behavior beyond the Javadoc are necessary, use package.html and/or doc-files/*.html as needed. You can keep such documentation in your main source tree if you like. samples/ or some such subdirectory can contain example code demonstrating use of the API.
8. Keep an apichanges.xml file, listing specification versions, dates, and changes made. If registered in project.properties it will be automatically displayed in your module's Javadoc.
9. Make sure your module's API is published in nbbuild/build.properties.

Keeping track of API changes

Each module should have an apichanges.xml and list of generated changes in order to track the progress of development of its APIs. Here are the steps you should take to get such list:

1. Copy nbbuild/javadoctools/apichanges-template.xml to your own module, e.g. foo/apichanges.xml.
2. Replace all CHANGEME strings with the correct path or token.
3. Edit your apichanges.xml:
   1. edit <apidefs> as needed (your module might have only one API area)
   2. add <change>s
4. Generate Javadoc for the module and check it.

Upgrade Guide

Significant changes in APIs which require the attention of users of your API should be documented in a separate Upgrade Guide document: currently there is only one, at openide/api/doc/org/openide/doc-files/upgrade.html. The document should summarize what is necessary to do to switch to a new API, what are the advantages of using the new API, performance implications, etc.

Using a module API

To use another module's API in your module, you must declare a dependency on that module in your project.xml. Now you may import and use API classes from the "foo" module in your module's code, e.g. org.netbeans.api.foo.FooCookie. Use of non-API classes is not permitted as your module might break unexpectedly.

If the "foo" module adds new APIs in version 1.8 which you wish to use, you must increase your dependency in the manifest to 1.8 at the same time as you make code changes to use the new APIs, and commit these together in one CVS commit. If the "foo" module changes incompatibly to e.g. org.netbeans.modules.foo/2 1.0, it will be necessary to make any needed code changes in your module, as well as to change project.xml.

Calling ((ClassLoader)Lookup.getDefault().lookup(ClassLoader.class)).loadClass("some.other.modules.Class") to use classes from a module you do not declare a dependency on is strongly discouraged - in some cases it will work, in others it will not. In general use of reflection between modules is a poor idea, and there is generally a cleaner (and simpler) solution. Do not be afraid to split your module into a general half, and a half which additionally depends on some other module and uses its API. If you need to communicate between the two halves, do not use reflection from the general half to call into the optional half - provide a registration interface in the generic half that the optional half can use to add its functionality. This could be a simple interface and a static registration method, or it could mean using lookup APIs for a more powerful solution.

Versioning of user data

As a rule, modules should be very careful to ensure that data stored by a user is not corrupted by newer versions of a module. Settings, as opposed to development data, are generally not expected to be preserved without errors when downgrading to older versions of a module.

Serialized settings

Modules storing any settings in serialized form should pay attention to compatibility of these settings. Use serialVersionUID for all serializable classes, and do not change it once set. Newer versions of a module must be able to read settings stored by older versions without user-visible errors, as a rule of thumb. If a class is no longer needed except for deserialization, remove any unnecessary methods, @deprecate it, and if applicable return null from readResolve so it will be ignored.

Remember, common serializable objects include: SystemOptions; ServiceTypes (now rarely used); TopComponents; Node.Handles (usually only a concern for creators of top-level nodes in their own windows); .Env environments from open and edit supports; and DataLoaders. There are some other serializable things but these are the ones you will commonly deal with.

Helpful mechanisms for making serialized forms of objects more robust include implementing Externalizable and writing state in a specific order, to reduce the amount of information written; keeping state in a hashtable rather than direct nontransient instance variables, which makes it easier to recover from missing fields, and handle new ones; and using versioned serialization replacers, each version of which reads its own format from settings and constructs the current in-memory representation.

Non-serialized settings

If you store settings in some other way - for example, XML files in the system folder - then you are responsible for maintaining compatibility of them however is appropriate. This may be easier than for serialized settings, since old and inapplicable settings objects can be simply ignored.

XML DTDs and Schemas

Many modules have a need to specify XML DTDs or XML Schemas to store various kinds of information - commonly objects provided by modules in XML layers, or stored as part of user settings. Basic rules for creating a schema:

1. Define your schema, and choose an initial version for it. Store the schema inside your module somewhere, e.g. org/netbeans/modules/foo/resources/foodata-1.xsd.
2. Choose a public ID for the DTD. This must mention the version number in it, mention NetBeans or somehow indicate what part of the world this applies to, and be more rather than less descriptive. For example:

   -//NetBeans//DTD Foo Widget Configuration 1.0//EN
   

   XML Schemas use URIs instead. For XML Schema, include the version number in the namespace, e.g. http://www.netbeans.org/ns/foodata/1.
3. DTDs may be registered in /xml/entities/ in your XML layer, for use in XML completion. XML Schemas currently cannot.
4. Decide on a public URL for the DTD, such as http://www.netbeans.org/dtds/foodata-1_0.dtd. This must mention the version number.

   For XML schema, perhaps just append .xsd to the URI, e.g. http://www.netbeans.org/ns/foodata/1.xsd.

5. Place a copy of the DTD/schema at this location (in source, www/www/dtds/ or www/www/ns/) so it will be accessible from the internet. Also modify the catalog file in this directory to mention it (for DTDs); or catalog.xml (for Schema).
6. It is a good idea to include inside the schema comments giving its public ID and public URL (for DTDs), as well as a brief description of what it is for.
7. All XML files based on a DTD should include an explicit <!DOCTYPE> tag, so that XML editing tools can reliably recognize and handle them. For XML Schema, it is only necessary to use the correct namespace; the schemaLocation attribute is not necessary.

To make changes to a schema:

1. Never change a schema (other than adding comments etc.) without changing the public ID / namespace!
2. Choose a public ID / namespace for the new version of the schema, say by incrementing the version number in the ID / namespace.
3. Add the new schema to your module's resources package. Leave the old one there.
4. Register the new schema in your module's layer, if applicable. Leave the old registration there.
5. Add the new schema to the netbeans.org schema publishing area. Leave the old schema there.
6. Make sure your module code is capable of reading and handling any version of the schema.

Development data

Development data should be handled much more carefully than settings. This means any data which the user has created which actually forms a part of the developed application, rather than configuration of the IDE. For example, *.form files used by the Form Editor. Certainly new versions of a module should be able to read data produced by any older version. It is also very desirable for older versions of the module to be able to read the format produced by the newer version of the module, ignore any parts it cannot understand, and faithfully preserve these parts as it read them when saving. This permits a user to experiment with an older version of the IDE without fear of losing work. A careful design for development data is necessary to ensure that optional and added capabilities are clearly delineated, so that the current implementation will be able to avoid damaging future data.

Modules with special file formats for development data should also use readable textual formats whenever possible, and give special consideration to avoiding unneeded formatting changes when saving, so that the data can be used in a textual version-control system comfortably.

Numbering Scheme for Updates

While developers have the responsibility to manage dependencies from their modules to both the Open APIs and other modules, and mark API changes of all sorts with changes in the module or API specification version, release engineers who publish modules also need to make version-number changes. Remember, it is never particularly harmful to increase the specification version (for example before cutting a release of a module), and either developers or release engineers may do so - such changes of course do not need any matching documentation as described above for API changes.

It is recommended that API and module specification versions in the trunk follow a two-digit scheme such as 1.5, where the next in sequence would be 1.6. On a release branch, three-digit schemes should be used, such as 1.5.1, 1.5.2, and so on. Post-release patches could have four digits, and so on.

If a number of API changes are made between releases, it may be annoying for the API specification version to be e.g. 1.133. Additionally, if specification versions of the APIs are to be used to distinguish the APIs available in each IDE release, they should be more mnemonic. So it may be useful to choose a new first digit after a release. For example, 1.20 may be branched for a release, forming 1.20.1 and so on, released as 1.20.4; meanwhile, the development builds become 2.1 rather than 1.21, so that everyone can remember that 1.x numbers mean one release, and 2.x numbers the next release.

It is important that every published release of a module have a different specification version. Otherwise automated updates cannot work correctly. Of course, if a "new version" of a module is being published solely because it was included in some bugfix build, and in fact did not contain any user-noticeable changes from the last released version, release engineers may prefer to either avoid increasing its specification version, or withhold it from the update center altogether, so as to prevent users of the previous similar version from unwittingly wasting time downloading it; but this is difficult to manage and no one currently does so.

Please remember that implementation versions of modules are not intended to be ordered. Implementation versions need not actually be numeric at all, and the IDE's Modules API intentionally prevents inter-module dependencies from using them except as exact string comparisons. Specification versions, by contrast, must be numeric, and the only permitted comparisons in dependencies are of the form "version x.y.z or anything greater".

As a practical policy for using implementation versions, it is helpful to make them integers if they are being used in implementation dependencies from other modules, and use the build property spec.version.base in both producers and consumers of implementation dependencies in place of a fixed specification version. This trick makes management of complex sets of modules with implementation dependencies much easier. From the NBM project GUI, just check the checkbox Append Implementation Versions Automatically in the Versioning panel.

Release engineers should assume that module manifests provide complete information about which versions of what module may be run on which version of the IDE, via their major release versions, specification versions, and dependencies. Of course these assumptions should also be tested before actually publishing something on a public update server; but if any inconsistencies are found, these are P1/P2 bugs for the developer and it is better to resolve them properly in the code, than to use tricks in the update server to force certain configurations of modules to be loaded.