Reverse Literate Programming - Code Trails

Background

Literate Programming is a technique which was introduced by Donald Knuth in 1984:

I believe that the time is ripe for significantly better documentation of programs, and that we can best achieve this by considering programs to be works of literature. Hence, my title: "Literate Programming."

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

The practitioner of literate programming can be regarded as an essayist, whose main concern is with exposition and excellence of style. Such an author, with thesaurus in hand, chooses the names of variables carefully and explains what each variable means. He or she strives for a program that is comprehensible because its concepts have been introduced in an order that is best for human understanding, using a mixture of formal and informal methods that reinforce each other.

The key idea here is that the programmer is tasked with writing documentation that compiles into code rather than writing the code and attempting to document it separately.

At the other end of the spectrum is the imaginatively named Reverse Literate Programming. This tasks the programmer with writing code that is directly compiled into documentation. This is likely to sound quite familiar as systems like Javadoc enable API documentation to be produced by compiling program source code. However, API documentation alone is not enough as it is focused on the minutia of source code rather than any high level details. Package level documentation can be added but this isn't very accessible as it is stored in a separate file.

Code Trails

I think IDEs need to do a better job of making it easier to provide simple high level source code documentation.

The use of markers within code comments allow key locations within code to be highlighted. The IDE collects together these markers into Trails that are displayed together. These Trails provide simple high level documentation and every marker can be double clicked to quickly examine the code at the location of the marker.

What does this look like in Eclipse? Here is some example code:

And the corresponding Trail view:

Installation

If you are interested in trying out the Code Trails plugin you can install it from my update site:

http://mchr3k-eclipse.appspot.com/

When you first open the Trails view you will get the following warning about needing to run a full workspace rebuild.

Realignment for JD-Eclipse

Update: I have posted details about a new release of my plugin here.

In Eclipse, when debugging a Java program or simply trying to explore some code, it is very frustrating when you hit a class without source.

There are a small number of open source Java decompilers which promise to improve this by generating some source code based on the class file. The best one which I have tried is called JD-Eclipse. However, JD-Eclipse has a couple of key limitations:

1. The generated source code doesn’t line up with the debug line numbers because the generated code can’t include the original comments.
2. If you choose to associate the “*.class” file type within Eclipse with the “Class File Editor” then you always get decompiled output even if you have real source attached.

I recently came across a project Realignment for JD-Eclipse which partially fixes (1) by properly lining up methods and lines within methods. I have forked this project and made some further enhancements to fix both (1) and (2). In particular, I have added the following:

• Enhance the re-alignment process to handle enums and fields.
• Automatically attempt to setup the “correct” file associations.
• On first start:
• On Eclipse Juno there is a new “class without source” file type, register to decompile this and restore the built in Class Viewer for classes with source so that attached source can be viewed.
• On Eclipse Indigo and below register to handle the “class” file type. This can be undone in Window -> Preferences -> General -> Editors -> File Associations.
• On every other start:
• If the default “JD Eclipse” editor is registered to handle “class” or “class without source” update the mapping to use the realignment enabled JD Eclipse editor.
• Add a context menu for choosing which editor to open a class with (see picture below).
• Add entry to this context menu to allow the attached source to be changed.

If you are interested in trying this out you can download both JD-Eclipse along with my fork of Realignment for JD-Eclipse from my update site: http://mchr3k-eclipse.appspot.com/

Obligatory warning: This code has had very little testing so you please do report any weird behavior to me or raise an issue (https://github.com/mchr3k/jdeclipse-realign/issues).

Implementation Wrinkle

One of the trickiest parts of this project so far has been trying to make it possible to switch quickly between the built in Class Viewer (to view attached source) and the JD Eclipse Class Viewer (to view decompiled source). It turns out that there are two mechanisms which make this tricky when you open a Class in the JD Eclipse Class Viewer.

Firstly the IClassFile is associated with the decompiled source in the BufferManager. With this mapping in place this means that even the built in Class Viewer will display decompiled source. However, the BufferManager is a 60 element LRU cache so if you open enough other files the decompiled source will drop out of the cache.

Secondly the SourceMapper which is used to generate the decompiled source is registered with the IClassFile's parent PackageFragmentRoot. This means that even when the decompiled source is not cached in the BufferManager the built in Class Viewer will still get hold of the cached SourceMapper and generate and then display decompiled source!

To solve both of these issues I added some code to the dispose() method of the Realigned JD Eclipse Class Viewer to both remove the entry from the BufferManager and remove the SourceMapper.

Why does this matter? One of the frustrations with using a decompiler is that once you have one installed you can only view decompiled source even if you have real source attached. I added an "Open Class With" menu to allow a class file to be quickly opened in the regular class viewer to view attached source.

Eclipse Juno makes this much better as it introduces a "class without source" file type so that you can associate your decompiler with only those classes for which you don't have real source.