Sunday, 11 November 2012

JD-Eclipse Realign - Usability


In my previous post I talked about the problems caused by the original JD-Eclipse Realign approach of using a special editor to open decompiled source for a class. The key problem was that opening a class file is something which you only expect to involve the selected class but the resulting decompiled source is attached to the entire jar. Worse, the decompiled source attachment was something which could then be used by the standard Class File Viewer resulting in some weird behaviour.

I previously fixed this by switching to a model where decompiled source was enabled/disabled through a context menu that was intended to make the similarity with source attachments much more explicit.

This solution was technically sound but the usability was frankly awful. Any time you wanted to enable/disable decompiled source you were forced to use the Package/Project Explorer context menu which has loads of entries and is therefore always quite unpleasant to use.

Toolbar Buttons (V1.1.0)

The first step to improve this situation was to take the ability to attach/detach decompiled source and move it into the top level toolbar.

This pair of buttons expose two commands - Attach Decompiled Source, Detach Decompiled Source. The buttons are dynamically displayed in the toolbar when either a Class File Viewer editor is active or when at least one class, package in a jar or entire jar is selected in the Package/Project Explorer.

Pressing one of these buttons applies the action to all relevent selected objects in the active "Part". This means that if an editor is active the action applies to the class in that editor and if the Package/Project Explorer view is active the action applies to all selected classes, packages in jars or entire jars.

Decompiled By Default (V1.1.1)

Having added the toolbar buttons I realised that it was still not easy enough to jump straight into decompiled source. If you open a class without source you have to deal with a tiny moment of disappointment/frustration before reaching for the toolbar button to enabled decompiled source.

The solution to this problem is to restore the JD-Eclipse Realign editor (JDE-R Editor) with subtely different functionality.

The original decompiled source editor was styled with a different icon and always attached decompiled source when opening a class file. In the restored JDE-R Editor the behaviour has been changed so that decompiled source is only attached when there isn't already a normal source attachment. The icon of the JDE-R Editor has also been changed to be the same as the standard Class File Viewer.

The overall intention is that the JDE-R Editor should behave and look exactly the same as the built in Class File Viewer except that opening a class in a jar without a source attachment will attach decompiled source.

Once you have a class file opened with either normal or decompiled source in either the standard Clas File Viewer or JDE-R Editor you can use the toolbar buttons to quickly switch between normal and decompiled source.

File Associations

With the JDE-R Editor restored in V1.1.1 I have also restored the code which automatically attempts to set up the "correct" file associations on startup. These are as follows:

  • .class file - Class File Viewer [Decompiled]
  • .class file - Class File Viewer
  • .class file without source - Class File Viewer [Decompiled]
"Class File Viewer [Decompiled]" is the name I have given to the JD-Eclipse Realign editor.

Getting Started

To try out JD-Eclipse Realign you should use the update site listed on the JD-Eclipse Realign project page.

Tuesday, 30 October 2012

Eclipse Tips - Navigation

Eclipse is a really powerful tool but it's really easy to miss some of the most powerful features. I was therefore interested to come across the following link on /r/java: My Top 10 Tips on how to be more productive with the Eclipse IDE.

I liked these tips but I think there are some other important ones which are worth pointing out.

Navigation Tips

Eclipse has some fantastic functionality to enable fast navigation to find whatever you are looking for.

1) Open Type (Ctrl-Shift-T)

The Open Type window is the right way to open Java classes. If you find yourself looking for classes by manually looking around in the Package Explorer you are almost certainly wasting your own time unless you have no idea what the class you are looking for is called.

Open Type supports wildcards (e.g. H*Map matches HashMap) and camelcase (e.g. LHM matches LinkedHashMap).

2) Open Resource (Ctrl-Shift-R)

The Open Resource window is the right way to open any file which isn't a Java class e.g. a properties file or an xml file etc.

Open Resources works very similarly to Open Types with support for wildcards and camelcase. However there is also support for specifying Folder Prefixes (e.g. */ui/*.html matches all html files within a folder called ui).

3) Quick Open (Ctrl-O within a Java file)

Quick Open is the right way to jump to particular elements within a Java file. If you use the Outline View you are wasting your time as you are forced to visually search for what you are looking for.

Quick Open supports wildcards and camel case (as of Eclipse Juno).

Other Tips

There are three more tips which I think are particularly worth noting.

4) Quick Access (Ctrl-3)

Eclipse functionality is exposed through a large number of Views, the menus include a huge number of Commands and there are lots of Preferences which control all of this. Finding all of these elements is extremely tricky.

Quick Access supports wildcards and can be used to load any of these elements. This is by far the fastest way to open things like Views.

5) Local History

Have you ever made a series of changes to a file and then wished that you could go back to a previous version? It's easy if you are using a version control system and have checked in a previous version but what if you didn't?

Eclipse has got a feature to help you out. The Local History feature keeps track of a limited number of previous versions of every file which you save through Eclipse. To revert to one of these previous versions you just have to use the correct menu. Right click the file within Package Explorer and select Replace/Compare With -> Local History...

6) Eclipse Help

It is absolutely worth noting that the Eclipse Help is really excellent. For example, try looking at the “Open Type” or “Open Resource” articles for full details of the pattern matching which you can do.

Monday, 29 October 2012

Java Libraries Which Use Annotations

I recently added a command line user interface to my Java Thread Dump Analyser tool (Issue #2). As part of this work I tried out a new library that leveraged Java annotations in a nice way. This was the third library which I have encountered that uses annotations so I thought it might be useful to write a post about annotations and point out which Java libraries are putting them to good use.


What are annotations? Annotations are user defined markers which can be used in Java source code to encode any kind of metadata. Here is a contrived example:

    id       = 2868724,
    synopsis = "Enable time-travel",
    date     = "4/1/3007"
public static void travelThroughTime(Date destination) { ... }

More information is available in the excellent Java documentation. It is interesting to note that Annotations get a better constructor syntax than the rest of the Java language including named parameters and default values.



My first contact with a library which makes extensive use of annotations was the use of JDO within Google App Engine. Once again I'll dive straight into an example:

public class Employee {
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Key key;

    private String name;

What I like about this code is that the annotations provide a very concise way of marking which fields need to be stored and how they should be stored. The actual mechanics of storing and retrieving the data are then entirely generic and driven by the annotations which are used.



GSON is an excellent library for handling JSON serialization. By default it doesn't actually use Java annotations but the ExclusionStrategy API allows the user to define arbitrary logic for picking which fields to serialize. The docs include an example of how to use this API to select fields based on an annotation:



This command line argument parsing library uses annotations to simplify command line argument parsing. Here is a simple example:

public class JCommanderExample {
  @Parameter(names = { "-log", "-verbose" }, description = "Level of verbosity")
  private Integer verbose = 1;
  @Parameter(names = "-groups", description = "Comma-separated list of group names to be run")
  private String groups;

The online docs provide plenty of examples to get you started:


Thursday, 9 August 2012

Eclipse 4.2 (Juno) - Bugs

For the last couple of releases the Eclipse project has been working on two streams of development. 3.x uses the legacy workbench and has been the primary release train in past years. However, since Eclipse Helios (3.8) there has also been a second 4.x release train working on a new workbench framework. This has been delivering releases at the same time as 3.x (e.g. Helios (3.6/4.0), Indigo (3.7/4.1)).

This year Eclipse Juno (3.8/4.2) was released and for the first time the 4.x version was chosen to be the primary release and 3.8 is in theory the last 3.x release. Unfortunately my testing has quickly revealed some glaring bugs - basic elements of interacting with the workbench don't quite work correctly. I have raised five bugs and found one existing bug covering the key issues which I hit:
  • 362420 - Make "Quick access" optional and hidden by default
  • 386804 - The target area to DND Views and Editors into existing stacks is too small
  • 386803 - Quick Access mishandles Home and End
  • 386802 - DND of Views and Editors not working properly
  • 386806 - New Eclipse Icon not used by Eclipse 4.2 on Windows 7
  • 386817 - Customize Perspective Tool Bar Visibility appears broken in 4.2
Eclipse 4.2 also includes a new default theme (new and noteworthy, bug covering implementation) which I really dislike. A classic theme is included which makes Eclipse look much more like the previous 3.x releases but tediously you have to restart to fully apply a theme change despite Eclipse updating most elements dynamically and not warning that a restart is needed to fully apply the change (362522).
    Eclipse 3.8 workbench:

    Eclipse 4.2 workbench:

    As a result of all of these issues I will be sticking to Eclipse 3.8 this year but I will keep an eye on future 4.x releases. The Eclipse 4.2 downloads are easy to find but there is only one 3.8 package available:

    Wednesday, 8 August 2012

    JD-Eclipse Realign - Source Lifetime

    In my initial post about my fork of JD-Eclipse Realign I mentioned the difficulties I encountered in trying to allow a user to switch quickly between decompiled source and attached source. Since that post I have realised that my original fix was insufficient.

    Opening a class file in the JD-Eclipse Realign editor involves the following steps:
    1. Check whether a source mapper is already installed for the container jar.
    2. If one already exists use that to get the source for this class file.
    3. If one does not exist create a new one, register it with the container jar and then use it to get the source for this class file.
    The JD-Eclipse editor subclasses the built in Class File Viewer (which already implements steps 1. and 2.) and provides an implementation of 3. that always returns a source mapper that generates decompiled source code. This has the unfortunate side effect that once you have loaded a single class in the JD-Eclipse editor (which registers a decompiling source mapper with the jar) you can open the standard Class File Viewer and get decompiled source for any other class in the same jar.

    My original JD-Eclipse Realign editor changes fixed this by de-registering the source mapper when you close the editor. This still meant that while a single JD-Eclipse Realign editor was open you could still get decompiled source in the standard Class File Viewer. I tried to address this once and for all by removing the source mapper as soon as the JD-Eclipse Realign editor was opened. Unfortunately this had unforeseen consequences. It turns out that navigating to a particular element (e.g. from the Outline view) relies on a source mapper being available.

    In light of all of this I concluded that having a special JD-Eclipse editor was actually the wrong approach. A better solution is to treat enabling class decompilation the same way as attaching real source code. To enable this I reworked the Open Class With menu which I had already added to work slightly differently.
    • When there is no source attached and decompilation is disabled:

    • When there is no source attached and decompilation is enabled:

    • When there is source attached:

    In this last case the menu doesn't allow decompilation to be directly enabled for dull implementation reasons. In all cases when the source attachment/decompilation state changes all open class file viewers are updated.

    With the addition of these new menu options I have also removed the JD-Eclipse Realign editor. My code is packaged as a fragment that extends the base JD Eclipse plugin so I can't remove the vanilla JD Eclipse editor. However, on every startup my fragment will check whether the JD Eclipse editor has been associated with class files (or class files without source) and revert any such associations to use the standard Class File Viewer.

    An updated version of JD Eclipse Realign is available from my update site now:

    If you have any issues please do raise it here.

    Tuesday, 7 August 2012

    Java Packaging

    jar (

    The basic mechanism for packaging Java code is within a jar file. This can be achieved in ant using the jar task. The jar task is more powerful than it first appears as you can use a nested zipfileset to package multiple source jars into a single output jar.

    However, the danger with this mechanism is that you may end up packaging a version of a class which conflicts with the version required by another dependency.

    jarjar (

    jarjar extends the default jar ant task with a new "rule" child element which specifies mappings of package names. This rewrites all the classes within the output jar to move the classes within the specified packages and update all the references to these classes.

    For example could be moved to This is one way to avoid the classic jar hell.

    jarinjarloader (no site)

    The jarinjar classloader is an internal part of the Eclipse JDT project. However, you can get the classes by opening org.eclipse.jdt.ui_*version_number*.jar with an archiver and extracting the file

    With the aid of these classes it is possible to package a jar within another jar and still load the contained classes and resources. For an example of what this looks like in practice you can look at getSWTClassloader() in

    Another project which claims to do the same is One-Jar ( However, I haven't used this myself.

    swtjar (

    The standard approach to packaging SWT applications is to create a bundle per platform (32/64 bit and Win/Linux/OSX). This is because SWT requires the application to load the correct jar for the platform being used.

    However, with the help of the jarinjar loader classes it is possible to include multiple SWT jars within an application jar and then use standard platform detection code to pick the correct one to programmatically add to the class path. My swtjar project packages this solution as an ant task.

    jarbundler (

    OSX has its own package format to allow an application to provide an icon and control startup options. This is particularly useful as SWT on OSX requires an extra command line argument to work. The jarbundler project is an ant task that helps you to build OSX application bundles.

    For details about using this tool you should refer to this page:

    Thursday, 26 July 2012

    Reverse Literate Programming - Code Trails


    Literate Programming is a technique which was introduced by Donald Knuth in 1984:

    I believe that the time is ripe for significantly better documentation of programs, and that we can best achieve this by considering programs to be works of literature. Hence, my title: "Literate Programming."

    Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

    The practitioner of literate programming can be regarded as an essayist, whose main concern is with exposition and excellence of style. Such an author, with thesaurus in hand, chooses the names of variables carefully and explains what each variable means. He or she strives for a program that is comprehensible because its concepts have been introduced in an order that is best for human understanding, using a mixture of formal and informal methods that reinforce each other.

    The key idea here is that the programmer is tasked with writing documentation that compiles into code rather than writing the code and attempting to document it separately.

    At the other end of the spectrum is the imaginatively named Reverse Literate Programming. This tasks the programmer with writing code that is directly compiled into documentation. This is likely to sound quite familiar as systems like Javadoc enable API documentation to be produced by compiling program source code. However, API documentation alone is not enough as it is focused on the minutia of source code rather than any high level details. Package level documentation can be added but this isn't very accessible as it is stored in a separate file.

    Code Trails

    I think IDEs need to do a better job of making it easier to provide simple high level source code documentation.

    The use of markers within code comments allow key locations within code to be highlighted. The IDE collects together these markers into Trails that are displayed together. These Trails provide simple high level documentation and every marker can be double clicked to quickly examine the code at the location of the marker.

    What does this look like in Eclipse? Here is some example code:

    And the corresponding Trail view:


    If you are interested in trying out the Code Trails plugin you can install it from my update site:

    When you first open the Trails view you will get the following warning about needing to run a full workspace rebuild.

    Monday, 9 July 2012

    Realignment for JD-Eclipse

    Update: I have posted details about a new release of my plugin here.

    In Eclipse, when debugging a Java program or simply trying to explore some code, it is very frustrating when you hit a class without source.

    There are a small number of open source Java decompilers which promise to improve this by generating some source code based on the class file. The best one which I have tried is called JD-Eclipse. However, JD-Eclipse has a couple of key limitations:

    1. The generated source code doesn’t line up with the debug line numbers because the generated code can’t include the original comments.
    2. If you choose to associate the “*.class” file type within Eclipse with the “Class File Editor” then you always get decompiled output even if you have real source attached.
    I recently came across a project Realignment for JD-Eclipse which partially fixes (1) by properly lining up methods and lines within methods. I have forked this project and made some further enhancements to fix both (1) and (2). In particular, I have added the following:

    • Enhance the re-alignment process to handle enums and fields.
    • Automatically attempt to setup the “correct” file associations.
      • On first start:
        • On Eclipse Juno there is a new “class without source” file type, register to decompile this and restore the built in Class Viewer for classes with source so that attached source can be viewed.
        • On Eclipse Indigo and below register to handle the “class” file type. This can be undone in Window -> Preferences -> General -> Editors -> File Associations.
      • On every other start:
        • If the default “JD Eclipse” editor is registered to handle “class” or “class without source” update the mapping to use the realignment enabled JD Eclipse editor.
    • Add a context menu for choosing which editor to open a class with (see picture below).
    • Add entry to this context menu to allow the attached source to be changed.

    If you are interested in trying this out you can download both JD-Eclipse along with my fork of Realignment for JD-Eclipse from my update site:

    Obligatory warning: This code has had very little testing so you please do report any weird behavior to me or raise an issue (

    Implementation Wrinkle

    One of the trickiest parts of this project so far has been trying to make it possible to switch quickly between the built in Class Viewer (to view attached source) and the JD Eclipse Class Viewer (to view decompiled source). It turns out that there are two mechanisms which make this tricky when you open a Class in the JD Eclipse Class Viewer.

    Firstly the IClassFile is associated with the decompiled source in the BufferManager. With this mapping in place this means that even the built in Class Viewer will display decompiled source. However, the BufferManager is a 60 element LRU cache so if you open enough other files the decompiled source will drop out of the cache.

    Secondly the SourceMapper which is used to generate the decompiled source is registered with the IClassFile's parent PackageFragmentRoot. This means that even when the decompiled source is not cached in the BufferManager the built in Class Viewer will still get hold of the cached SourceMapper and generate and then display decompiled source!

    To solve both of these issues I added some code to the dispose() method of the Realigned JD Eclipse Class Viewer to both remove the entry from the BufferManager and remove the SourceMapper.

    Why does this matter? One of the frustrations with using a decompiler is that once you have one installed you can only view decompiled source even if you have real source attached. I added an "Open Class With" menu to allow a class file to be quickly opened in the regular class viewer to view attached source.

    Eclipse Juno makes this much better as it introduces a "class without source" file type so that you can associate your decompiler with only those classes for which you don't have real source.

    Wednesday, 2 May 2012

    Talk at JUGS

    Today I gave a talk at Java User Group Scotland. The talk covered the material in my last three posts along with the Java Live Thread Analyser. I demoed InTrace, InMemProfiler and JLTA and talked about how they worked.

    The slides from my talk are available here.


    The next instrumentation tool which I wrote (also started in 2010) was motivated by a session using the excellent VisualVM memory profiling to investigate why an application was performing profligate memory allocation. The results in VisualVM are presented as a tree of allocation stack traces.

    This was frustrating in some cases. For example a string constructed from several parts results in stack traces through StringBuilder.append() as well as through StringBuilder.toString(). In practice I would prefer to see both of these allocations presented together as the methods are called from the same site. I also wanted to get a summary of the lifetime of the objects being allocated grouped together into buckets. This is something which VisualVM doesn't offer at all - the closest thing available is an indication of the number of GCs which an object has survived.

    I managed both of these by using the JVM TI API to write an agent that added an InMemProfiler method call into the java.lang.Object constructor and after every array allocation (as these don't use the Object constructor). This instrumentation is done with the native JVM TI API.

    When profiling object lifetimes the profiler code creates a Java weak reference for each allocated object to detect when it has been collected. The tracking of collection could be done much more efficiently using the native JVM TI object tagging mechanism but I wanted to do as much as possible using Java.

    For further information you should refer to my GitHub page:


    In my last post I introduced the low level APIs available in the JVM.

    The first tool which I wrote to use the java.lang.instrument API was InTrace which I started in the summer of 2010. This tool was written to allow the execution of any Java program to be traced - method entry, exit, method args, thrown exceptions, caught exceptions and execution path within methods. All of this is made possible by inserting InTrace method calls into classes. This can be done at any time thanks to the JVM's support for redefining already loaded classes.

    There are two parts to InTrace. The InTrace Agent allows for the dynamic instrumentation of Java classes to add trace calls. The InTrace UI allows for the remote control of the InTrace Agent and collection of trace output.

    To get started with InTrace you can use the Agent and UI directly. However, I have also written an Eclipse plugin which automates the process of launching a program with the agent and connecting to the launched program with the UI.

    Java Instrumentation APIs

    Historically there were two low level APIs within the JVM:
    Both of these were deprecated in Java 5 and removed in Java 6. The replacement introduced in Java 6 was the JVM Tools Interface. This is a native C API but there is an equivalent available in two parts in pure Java. The Java Debug Interface and the java.lang.instrument package. One of the most noticeable changes made in the switch from JVMPI to JVMTI was the reliance on bytecode instrumentation for tracking method entry/exit and object allocation. The justification for this was to allow the profiling code to be optimised along with the code being profiled to minimise the overall performance impact.

    In my opinion the most interesting part of all of this is that the current JDI and java.lang.instrument APIs allow for interesting debugging and profiling tools to be written in pure Java. In other words, if you are an experienced Java developer, it is relatively easy for you to write great tools that interact with the internals of the JVM and the running of Java program's.

    Wednesday, 18 April 2012

    Exciting Java Tools

    I wanted to blog about some of the most exciting Java tools which I have come across. Sadly none of these tools are free but the technology which they implement is really impressive.

    Chronon records a complete trace of the execution of a Java program. There are a couple of reasons why this is really cool.
    Firstly, the actual logging mechanism is claimed to have a low enough impact that you could realistically run with Chronon all the time. The theory here is that when you hit a hard to reproduce bug you will always have enough information to investigate and fix the bug.
    Secondly, Chronon have produced some excellent UI to get the most out of the collected trace files. The time travelling debugger allows you to step forwards and backwards through a trace file as if you were stepping through a live program. Even cooler is the post execution logging which allows the developer to add logging statements and have them be executed as if they were present when the trace file was captured. 

    The Sun Hotspot JVM includes support for a cool feature called Hotswap. This allows the body of Java methods to be updated on the fly while the JVM is running. The feature is nice but limited as a lot of things are not allowed - methods can't be added or removed, method signatures can't be changed, fields can't be added or removed etc.
    ZeroTurnaround have managed to produce a system which runs within the standard Hotspot JVM and removes all of these limitations. This is really cool!
    This technology has been packaged in two ways. JRebel allows developers to work on applications and change the code without restarting the JVM. LiveRebel does the same but is aimed at live upgrade of webapps without downtime. 

    Azul Systems have created a new garbage collector (C4) that claims to put an end to stop the world garbage collections. The promise is that this allows a single JVM to use 10s of GB of heap without having to endure periodic stop the world GCs that last several seconds.
    The big side benefit of this work is that Azul have published lots of great information about the details of garbage collection.

    One more thing

    All the tools I have spoken about so far aren't free but there is one more tool which I should mention.
    Eclipse is both free and excellent and is definitely the tool that makes the most difference to me every day that I work on Java code.

    Tuesday, 10 April 2012

    Java Live Thread Analyser

    One of my most recent projects was motivated by a session using the excellent Visual VM tool. I was monitoring the startup of a Tomcat based application and watching the number of threads as a rough metric for how far along the startup was. The application I was monitoring has completely started when there are around 200 live threads out of around 300 started in total.

    I was somewhat taken aback at the number of live threads which were being used and also the number of short lived threads which had been started. Who allocated all these threads? What are they all doing? Unfortunately many of the threads had not been named. A thread dump gives some clues about what live threads are doing but many threads were blocked in generic methods (e.g. Timers). What I really want is to capture the stack trace of all Thread allocations and track the lifetime of all threads.

    This objective prompted me to write a tool which I've called the Java Live Thread Analyser (JLTA).

    There are two main parts to JLTA.
    1. A Java Agent allows me to collect the raw data and allow it to be downloaded over a TCP socket.
    2. A Java SWT GUI to connect to the agent, download the raw data and summarise the results in a useful way.
    The Agent collects data by making two transformations to the classes within the JVM. The Thread class is transformed to add a call to my tracking code from the end of all the Thread constructors. The Thread class and all subclasses are transformed to add a call to ky tracking code to the start and end of all run() methods. A Boolean thread local is used to make sure the run() method entry and exit are only handled once. This takes care of the case where a subclass calls into a superclass run() method. The tracking code attempts to detect when a Thread has been allocated by a Tomcat webapp by examining the context classloader of each Thread.

    The GUI filters the collected data by grouping thread allocations by stack trace. Threads can be filtered based on their thread state, context and whether they have been given a name. The number of stack frames being used for grouping threads together can be limited.
    I have published a static site describing this tool and linking to the appropriate files to download and use it.

    Monday, 9 April 2012


    For the last couple of years I have been writing software in my spare time to solve problems I encounter and because I frankly enjoy coding.  For the last few months I have been enjoying getting to grips with Arduino hardware and blogging about it. Given this experience I thought it was time to start this new blog about my coding projects. In this blog I plan to talk both about new projects and some of my old projects. I have an existing static site which documents some of my projects. However, I thought that starting this blog would give me the opportunity to go into more of the technical details behind my projects.