API Development
Changes: available in
CVS
Status: issue
26785
Overview
The NetBeans project provides a rich set of APIs
that is being developed as part of work of various groups.
By distributing development of the APIs to multiple people we can increase
the amount
of designed, reviewed, implemented, tuned and maintained contracts but
as each module is developed by a different person, there is a challenge in
providing APIs of the same quality, designed by using the same patterns and
documented in a common way.
This document describes the practices that should be used during the API
development, so NetBeans can produce APIs that will
stand the test of time and preserve investments
made by its customers.
We understand the APIs as
every aspect
another piece of system can depend on and not just method signature. That is why
providing documentation in form of javadoc is not usually be enough. Instead
NetBeans projects use document generated from answers to
Architecture Questions
as main entry point. The questions provide guidance to the module owner
and help him investigate architecture of his own module. By answering them the
owner is supposed to realize and discover various aspects that others might
depend on and remove them or document them.
Based on the detailed answers (especially the
<api/> tag) we generate overview tables like the one shown below
that are incorporated into the Javadoc.
If you write a NetBeans module you may want to setup the right layout of files
first. The default infrastructure
(nbbuild/templates/common.xml
and
nbbuild/templates/projectized.xml)
let you do various tweaks, but usually it is easier to just use the
expected default layout (currently described in
README).
The documentation related files shall be organized as follows:
module_dir/src/ - directory with your sources
module_dir/src/**/package.html - description of each package
module_dir/src/**/doc-files/ - directory for special javadoc files
module_dir/arch.xml - answers to architecture questions (see bellow)
module_dir/apichanges.xml - description of the history of changes
module_dir/nbproject/project.xml - project file with dependencies and other informations
The locations of arch.xml and apichanges.xml moreover has to be specified in
nbproject/project.properties
as javadoc.arch=${basedir}/arch.xml and javadoc.apichanges=${basedir}/apichanges.xml.
Generate arch.xml - open your project in NetBeans and
select Generate Architecture Description
from a context menu in projects tab. An empty, skeleton file will be
generated. You can always reinvoke this target, if your answers are
old, unanswered questions will be generated to the end of the file
(the file shall stay well formated).
When editing the file you can use HTML tags.
Some of the answers may have autogenerated default answers (currently
arch-where and dep-nb) for cases where the information is already recorded
anywhere else (for example in project.xml file). They may or may not be accurate.
You can accept the generated
answer while surrounding it with your additional comments or you
can suppress it. Just include
<defaultanswer generate='here' /> or
<defaultanswer generate='none' /> in the answer
of for your question. If you do not use the
<defaultanswer/>
element at all the default answer is apended to your own answer.
If you generate the defaultanswer, the source code for it is
put into the comments in the html file, so if you are not satisified
with the defaults, you can easily copy the output modify it and
<defaultanswer generate='none' />.
Use <api/> tag - this tag is one of the most important
in the architecture file.
Each use of the <api> tags generates new item into a table of
API interfaces. This is the main entry point to the documentation, so
use the tag a lot.
Not just for a description of javadoc interfaces, but for everything.
Remember that an api is any feature that someone
else rely on. Describe DTDs, properties,
files or layers you read, formats or protocols
that you communicate, etc.
The <api/> tag syntax is described by
its DTD
and consists of:
- name - the name of the API, DTD or property
- group - the group that this API belongs. For example
property
,
java
,
dtd
, layer
and possibly others. As we are
writing in java the attribute can be omitted and the default value
is java
.
- type you can either use someone else API (
import
)
or offer someone else dependency on your behavior (export
).
- category shall contain a name from
the enumeration (
official, stable,
devel, third, standard,
friend, private, deprecated) in
the meaning described
here.
- url shall refer to a document describing the API, if
available otherwise one can insert additional comments into the body
between the <api> and </api> tags.
An example is available here:
<api name="identification" group="dtd" type="import or export" category="stable" url="where is the description" >
Possibly some additional description to the API which may be skipped.
</api>
The interfaces in the table are grouped by the group of the API
and marked in the HTML text as <a name="group-name" /> so a reference
to these tables can be made by using <a href="#group-name" />.
Use <usecase> tag - when answering arch-usecases
question, surround the paragraphs describing
the way to use your API with <usecase name="..." id="..."> and
</usecase>. That way your paragraph will get correct heading
in the
How to use certain NetBeans APIs page.
Answer arch-what
- the first sentence of your answer to
arch-what
is used as a short description
in the overview page
so write it meaningfully. The full answer is then used
in
the details section, so again, make it real and useful description of your module.
Link between documents - important part of documentation is
the description of context. It is not enough to say: find this interface
in lookup
. The reader may not know what lookup
is, so it is better
to hyperlink
to its definition. You can use regular <a href> tag to link to
other documents, for root of your javadoc use @TOP@. So link to
lookup would be @TOP@/org/openide/util/Lookup.html.
Link between classes - consider making the prose section
part of package.html file. Then you can use @{link classname}
to address any class of your module or from modules you depend on.
Link between Javadoc sets - the context is often split between
multiple modules. To allow links between them, the root of each
module javadoc can be referred to as @org-netbeans-the-module-code-base-name@.
So to link to lookup from another module one can use
@org-openide-util@/org/openide/util/Lookup.html
(the list of all currently known module name substitutions is available
at
nbbuild/javadoctools/replaces.xml).
Use Relative Links - please remove as much as possible of usages
of non-relative links like http://www.netbeans.org/download/dev/javadoc
and replace them with @TOP@,
@org-netbeans-module-name@ or @JDK@ root points.
The javadoc is being scanned for allowed and disallowed links
(defined in
nbbuild/javadoctools/disallowed-links.xml) and violations cause the
build run from IDE to fail. It is generally not recommended
to refer to NetBeans website as the documentation shall be self contained,
but if you find a URL that makes sence, feel free to add it to the
nbbuild/javadoctools/disallowed-links.xml yourself.
Btw. it
seems better to use such pseudo root point than directly relative link as
for example content of package.html is usually duplicated into more
directories.
Create apichanges - important part of any api is history of its
changes. That is why create and maintain the
apichanges.xml
as described in NetBeans versioning policy.
When you refer to a class that no longer exists inside an API change, you can
use <class ... link="no"/>.
Validate your documentation - make sure the documentation format
is correct (links point to valid places, XML files has valid syntax, etc.).
This can be checked by invoking Generate Javadoc
from the
context menu. This builds the Javadoc and
(in addition to invoking ant javadoc from command line)
also checks for broken links and fails if there any - so make sure
all Javadoc of modules you are referring to has already been generated.
All NetBeans project Javadoc sets are being daily regenerated and uploaded to
the central NetBeans API List.
When your module can successfully build javadoc as described in previous section,
it is time to consider adding it to the API list as well.
To add it, you have to modify
nbbuild/build.properties
and add own module into config.fixedmodules.javadoc property.
Use
ant -f nbbuild/build.xml check-module-configs
cvs -q diff nbbuild
to review your changes.
Then verify that everything works correctly by rebuilding all Javadoc:
ant -f nbbuild/build.xml build-javadoc
and if the build succeeds and really contains your module, prepare for committing your
changes into CVS (check in the new moduleconfigs.txt too). Please note that three files in nbbuild/javadoctools
shall be modified by addition of references to your module root. Verify that
the additions are sane (e.g. contain no local references and look like the other lines
in the files) and then commit the modified
nbbuild/build.properties,
nbbuild/javadoctools/replaces.xml,
nbbuild/javadoctools/links.xml and
nbbuild/javadoctools/properties.xml.
By default the basic overview page is generated based on content of your
arch.xml,
apichanges.xml
and project.xml.
To see an
example, check the
overview page of component palette api.
This page has following structure:
- Title and description is taken from the
arch.xml's answer to
question arch-overall.
- List of javadoc packages is added by the default javadoc doclet.
- what is new section lists five recent api changes listed in
apichanges.xml.
Always add at least one change as this document is used to generate what is
new for the whole
release.
- List of usecases is taken from the
arch.xml answer to arch-usecases
question. It shall contain the main introduction into the meaning and usage of the API. Links to
javadoc classes and methods are welcomed. Also notice that the answer contributed to
global
page with usecases for all NetBeans APIs.
- Implementation details close the summary page. The contain answer to
arch-where question,
which shall contain link to NetBeans WebCVS with the module sources like
http://www.netbeans.org/source/browse/~checkout~/java/project/ for the
java/project module. Also an answer to deploy-dependencies arch question is generated
so other modules know how to express dependency on this one.
XXX need to describe:
{@link ...}, what
package.html can and cannot do,
@inheritDoc, etc.
In order to ensure good enough quality of produced APIs there is a service
provided to module writers - they can ask for an API review.
It is required that every new API will be reviewed prior to integration into trunk.
The exception from this rule is a friend API that is used only by modules
within the same cluster (the module has to explicitly list its friends). In this case
the review is recommended but not required.
Why?
The short answer to question why you should be interested in an architecture
review is because it will be useful
. Useful to you, as you discover
new possible solutions to your problems or mistakes in your design,
that might appear later, when integrated
together with the whole system or even in later versions, when problems with
maintainability and extensibility can show up. It will be useful to the whole
system as it will be composed from more stable components integrated in better
ways. It will be useful to whole your project as it will get better.
Nobody knows everything, but there is a lot of knowledge spread around.
Architecture review is a way to get the people with pieces of knowledge
together and cooperate in preventing us from
repeating known mistakes and solving problems in the wrong way.
Ask for advice through architecture review. It cannot hurt and it is likely going to be useful.
What?
It is unlikely that the review team will do some coding for you. It is also
unlikely that the reviewers are going to become domain experts and help
you understand your users or your requirements. This is your task and you have
to prepare these materials for the reviewers, as it is very likely they will ask
you about these questions in order to verify that your way of solving problems
of your users is really the right one.
As a result of architecture review you can expect advices and help
in identification of
- apis
that someone else could depend on,
- design or implementation that might have performance problems,
- influences of your solution on existing products or
- influences of other products on your solution in future,
- solutions that are solving
something different than was the original goal and
- other projects or efforts
going around that might help you in solving your problems.
More or less expect just
a high level help.
When?
Whenever you need architecture advice or clarification and because the charter of
the team is mostly high level, it is reasonable to come for the initial opinion
as soon as the architecture is visible so it can be reviewed.
This usually means after answering the first (more general) set of the
architecture questions which should be done before the actual start of
implementation. At this point the high level advices are of some use, later it
is always hard to change implementation that has been written.
Of course things are likely change during implementation, but the high level
direction given during this inception stage are likely not going to be
questioned then and only the newly discovered facts and differences from the
original suggestions are going to be evaluated during the before-commit review.
How?
For details about the process see the Architecture
Review Steps document or check the list of all reviews.
Why?
If an API is supposed to stand the test of time it has to preserve the
functionality that others are using, it has to be backward compatible.
Some tests for compatibility are easy, some require more work, but the testing
is necessary otherwise nobody can guarantee quality when the API is evolving.
Signature tests are simple starting point,
unit tests are very good for verifying the
contract
between a public API and its clients. Some people claim
that unit tests are poorly named since they imply that they are QA's
responsibility, but the development engineer is the one that really
benefits with several advantages:
the tests provide an example of how the developer expects the API to
be used.
Another is that when you run code coverage against a unit test
suite, it shows surprising areas where there is code that isn't
necessary to support the API, so one can easily remove
those extra bits
Another interesting feature of unit tests is support of
arrogance (which is part of all good programmers). So
here's the best, most compelling reason for creating and relying on
unit tests: you can much more confidently tell another engineer how wrong he
is when he claims your code is breaking his!
Read more about possible test patterns that
we use and how they can contribute to improvements in quality of your module.
There is an automated verification task that is executed after every daily build that checks
signature of classes and their fields and methods and sends reports to
api-changes mailing list.
Its reports contain both incompatible and compatible changes. So one gets
notified not only when something is broken, but also in case of accidental
API change like addition of a method by forgetting to make it
private.
By default the tests check all classes in official packages. E.g.
org.openide.*, org.netbeans.api.* and
org.netbeans.spi.* and recently also org.netbeans.jmi
that are part of modules included in daily build
of standard IDE and also those that are daily uploaded to
Alpha Update Center. That is why in order to have these tests running
on own module one has to package the API into one of the official package (or
request his own package to be added into the test)
and make the module part alpha autoupdate configuration.
Any questions related to the sigtest framework can be either sent to
nbdev@ or to
Rudolf Balada who maintains the
daily build (including sigtest) infrastructure.
Very important
verification of quality of an API is an automated test suite.
Most of NetBeans modules uses our test harness called xtest which is based on JUnit and enhances it with a few additional
features (tests should inherit from NbTestCase) and
configuration framework.
The simplest way how to make your module testable is to copy the
test directory from a small modules that already provide some tests
and modify it (e.g. html).
The needed changes include classpath modifications for compilation and
execution in build.xml, correcting the list of tests in
cfg-unit.xml and of course the placing your own tests into
unit/src directory.
The last step is to include the suite in daily execution of unit tests. For
that it is enough to modify the
xtest/instance/master-config.xml to include your module in the
unit-nb test config. Verify that you have done everything correctly by
running
ant -f nbbuild/build.xml unit-validation
and checking that your tests were successfully executed. Since then
make sure that your module tests really run
and pass, as since then other people start to use these tests to verify validity of their
own commits. And you should not cause false alarms by problems in your code.
Also consider to subscribe to notification
framework to get email notifications about automatic failures, if you can reach the URL.
The important part in a life cycle of an API (as well as any other product)
is the feedback from the users. In order to get it one should
let your users know that there is an API and allow they to try it. For that
purpose NetBeans use its Alpha Update Center
. Curious users may enable it and
that way be informed about latest development achievements.
To get a module into the Alpha Update Center
one needs to make sure that the
module's build.xml file
has netbeans, clean, and nbm targets that work in the normal way -
normally this is accomplished trivially by making a projectized module and not overriding
any targets from the default build harness.
Then one can add entries
for the new module to nbbuild/build.properties in the list
config.modules.daily-alpha-nbms.
Use
ant -f nbbuild/build.xml check-module-configs
cvs diff ide/golden/moduleconfigs.txt
to review your changes (check in the new
moduleconfigs.txt too).
One should test the NBM building process on local disk by making sure you
have everything of interest checked out from CVS, opening nbbuild
as a project in the IDE, and selecting Build Daily Alpha NBMs from its context menu.
If something is messed up, mail gets sent to broken_builds@netbeans.org
so it can be corrected. It is a good idea to notify
aumasters@netbeans.org too.
Publishing New Versions
The content of Alpha Update Center
is refreshed every day. The new
version of module NBM is build from trunk and specification version in its
module manifest is compared to the specification version of already uploaded
module. If the new one is greater, the new version of the module replaces the
old one.
This means that one can consciously and automatically upload new versions of an
API from trunk to its users just by increasing the specification version in the
CVS manifest file.
Usually an attempt to produce an API requires longer development time and it is
useful to mark it as not being finished yet. NetBeans use a set of
stability categories for that.
The expected scenario is that a module with an API starts its development in
a CVS sandbox (contrib.netbeans.org)
or as a regular netbeans.org project (like xml.netbeans.org) but is not part of
the regular build. Then it is
offered on Alpha or Beta AutoUpdate (early access mode)
and one can work on its finalization.
As the module is not part of a stable release, it can be modified in
incompatible way. As soon as one thinks that the API is fine
and it satisfies
quality
criteria,
it can be put on Stable Update Center
or
even find its way into standard distribution.
There is however one restriction. In order to make it easy for API users to find
out what is stable API we have come with a simple description:
If a class is in org.netbeans.api.* or
org.netbeans.spi.* packages, and is part of a stable release,
then it is stable
. Such API is
then called NetBeans Official API.
This rule is not meant to block anyone in producing APIs. One can always
create an API in less prominent package (say
org.netbeans.modules.ant.api),
publish it on netbeans.org as
stable one and use all the
infrastructure for API development that is available. Moreover this approach
is not as strict and allows the API to be part of a release even if it has not
reached enough stability for unlimited amount of time.
There can be situations when strictly following the official namespace restriction
may cause a lot of troubles to early adopters and hurt NetBeans acceptance and
competitiveness. Sometimes one needs to provide an API quickly, cannot guarantee
that it is that stable, but is strongly
willing to stabilize it in close future. In such case it may be acceptable
to release the API in official packages, mark it as
under development
(by warning in javadoc and special name of the module)
and stabilize it in next release.
In such cases it may be possible to allow a temporary release of an API
under development in official namespace if
following is guaranteed:
- the API is intended to become stable
- making it stable has been
agreed to be the highest priority for next release
- the API is of
nearly stable
quality - documented, tested,
published and successfully reviewed
The purpose of this temporary release
is to encourage early adopters to
test the APIs in real world and provide feedback to make them better. The
publishers of the APIs in return promise to stabilize them soon and do that
carefully considering the user impact.
As a result NetBeans should be able to deliver important APIs as soon as
possible, offer them to early adopters and increase API quality by
incorporating the feedback. The adopters could be sure that the APIs will be
stabilized in a given time frame and that they will not need to do major changes
(like repackaging of all import statements, which was the current strategy)
when the API becomes stable. In
order to clearly communicate the exceptional state of the APIs, there should be
a visible and non-ignorable stamp that such APIs are
under development:
the javadoc documentation headers shall contain visible warnings
the module name shall indicate that it is not stable yet. This
shall be done using /0
in module name, for example
org.netbeans.api.projects.ant/0. As this string
has to be used by every module writer to specify module dependency it
forms appropriate warning. When stable version of the module is produced and
is incompatible with the /0
version, the
name should change to
org.netbeans.api.projects.ant/2. If the stable
version remains compatible the name of the module should be
org.netbeans.api.projects.ant/1 and appropriate
ModuleAutoDeps shall be provided to upgrade
dependencies of modules that used the /0
version.
These suggestions shall ensure that the right balance is achieved between
produces and consumers of any NetBeans API.
It should however be stated that nobody shall be restricted by
official namespace restriction
or bound by a time limit of the
temporary release
,
because if one does not want to create stable API, one does not have to.
Contributing to official API set
shall be a privilege as such effort requires more attention and increased amount
of work that only those who really care are willing to invest.
Comments or corrections to
nbdev@netbeans.org or
apireviews@netbeans.org.
