At the LJC openjdk hackday

One way to relax, after a major project release, or as near to it as to allow a deep breath, is to spend some time with others setting up, testing, and working on something at a Hackday. “What?” I hear someone exclaim, “you’ve just built software for a release; it’s exhausted you; and now you still want to work on software, though of another kind, and you find that relaxing? Come now, surely, you jest!”

Before responding to this, here’s an outline of this post:

• The openjdk hackday and what it consisted of. The hackday happens once every month.
• How to understand, or at least navigate, a complex codebase; and how this applies to complicated legacy codebases.
• What’s involved in building and testing the openjdk; and thoughts on keeping up with the different versions of Java.

Like I said, it’s one of the ways to relax. Secondly, not all activities exhaust you in the same way nor to the same degree: he who has spent the week in Spreadsheets might find it relaxing to work in PowerPoint; he who works indoors might relax outdoors, and vice-versa; he whose mind has been wrestling with Prose, in its rhythmic irregularities, soothes himself in rhythmic Poetry. In all these examples, and in many others that occur to you, effort is expended yet the activities vary; each exercises different mental faculties, and breaks off that weariness of the monotonous exertion of the same faculties. Hence the recommendation to so arrange our activities that, being neither idle nor worn out, we find a certain rest in them.

Before going to the Hackday, I read the instructions, followed numerous links: one link to get the software I needed, then attracted by another, and then another, and there I stayed tangled in the web. One effect was that feeling of being overwhelmed by the information I came across, and the proficiency that would be expected, and my impatience to start contributing; I had forgotten that we all have to start somewhere. Even on the day, I was hesitant: to go or not to go. But, having committed, I went. I also realised that it’s better to download all the reading materials and the required software in case there’s no Wi-Fi.

At the hackday were several people, all with different experiences, all with varying degrees of proficiency; yet, all eager to learn, willing to be taught, and willing to teach. To this, you can add, as an auxiliary, the venue. The hackday was at the Salesforce office, the 26th floor of new The Heron building.

After having breakfast, talking with the other attendees, connecting to the Wi-Fi, the host gathers us, explains the ground rules, outlines the main ideas on the agenda, and opens it up to the attendees to suggest more ideas that they’d like to work on. Most of us attending had a preference of what to work on: some wanted to work on openjdk, some on Scala, others on Play, some on Go, some on Clojure.

The agenda had two pre-planned items:

• Getting started on, and improving the openjdk. The openjdk is a Reference Implementation (RI) of the Java programming Language since version 7.
• Working on betterrev – a wrapper, written in Play, for the JDK build system. More details are here

The floor was then opened to the attendees to speak on what they would like to work on, and also what they’re working on.

• One of the attendees (Norbert Radyk ) was working on a Scala wrapper for the Apache Commons POI library (a Java API for Microsoft Documents). Those who had come for Scala had something to work on; Norbert had helpers; and I think that explaining his project to strangers and seeing them productive was a good preparation for his talk at the London Scala User Group (video)

• Another was working on Hugo, a static blog generator written in Go. I was interested, but I couldn’t be in two places at the same time: I’d like to do away with setting up WordPress and a database, and return to using Emacs orgmode for my notes.

Finding your group – Pairing

Now that we know who’s working on what, it becomes easy to find the group we’d like to work with. Having found the openjdk group, each of us explained where we were on the learning curve: some were having issues with the virtual machines, some with getting around Linux, some with understanding the relationships between the main JDK components. Some were further ahead – they had attended other sessions previously. We coalesced around Mani, around each other, asked for help, received help, and began experimenting. Having the prepackaged virtual machines helped much.

We followed two documents of instructions to get started:

• The beginner’s guide – google-doc-1.
• The intermediate guide – google-doc-2

One the attendees had already downloaded the 2 virtual machines that were created for developing openjdk. The links to the virtual machines are kept up-to-date in the beginner’s guide. I was working on the virtual machine that had the jdk sources in Eclipse, not the one that uses IntelliJ for its IDE. Retrieving the sources of jdk9 and building it consumed most of virtual machines space on the hard disk. Because of this, I skipped the section on building jdk9 and focused on those instructions which apply to jdk8 – I had to make the most of the day. Expanding the size of the virtual machine’s hard disk was something I could do later.

Level 1 directory hierarchy

Two questions occurred to me: the first is how do you grasp the different components that make up the JDK, and how quickly can you do that? How are they structured? What’s the biggest component? One way to see the structure of the jdk components is to the UNIX tree command. The one below shows the tree structure of the first level hierarchy of the openjdk.

But it’s much easier to visualise these components, their size and complexity using SonarQube. Most tools that are used to build and test the jdk predate Maven, Hudson/Jenkins, and SonarQube. Hence the reason why some jdk components will not appear in SonarQube’s reports.

For a codebase that we’re new to, using these tools is a quick way of understanding their structure and complexity. On any new project, I always ask for documentation of the application. Not getting it, or finding it non-existent, I would mentally add a percentage to the estimate I had in mind. Now with tools like SonarQube, if allowed to use them, would reduce the time to build up knowledge of an application.

The second was: the virtual machine we were working on had three versions of Java; we were installing a fourth – so, how are these different versions used in build and testing a new version?

Using several jdk versions, and keeping up

What I found interesting was how the different versions of Java are needed to build and test new versions. At one of the LJC sessions on the new version of the JMS API, much was said about how it uses the new features of Java 6 and 7, and I never understood why it was so or why it had to be. Why should new versions of the Java language necessitate a re-write, or at least the perception of one, of the components that make up the Java EE platform? Doesn’t this seem like a waste of time and effort?

Seeing how new versions of the Java language are built answered this question that has been lingering within me. You currently have Java 7; how would you build Java 8? Isn’t it by using Java 7:

• Using Java 7, write the new feature that you need for Java 8 – (I’m using Java as the umbrella term for both the jdk and the jre).
• Test that Java 7 compiles your new Java 8 feature. Also ensure that the test you’ve written passes in Java 7.
• Write a program to use the new Java 8 feature. Compile and run it using Java 8. Ensure that the test you’ve written in Java 8 for your program passes.
• Now you can go back to Java 7, and mark your new feature as complete.

This process is also what the jdk test suite (jtreg) allows us to test. There are more complex tests but that’s for another time.

Extend this process to Java EE: it stands to reason that as new versions of the Java language are being created, the relevant components in Java EE will also need to be written using the features of the new Java version.

What, then, does this mean for us who work with the different versions of both Java the language and Java EE? Can’t I just learn one version thoroughly and not trouble myself with these “new” features? The answer, as always, depends on what our work entails, and on what we can ignore, or pass by, without being culpable of incompetence in our work. I like how Cobbett uses the word ignorance, restricting it to the relation between all knowledge and that proportion of it that we’re expected to know:

Ignorance consists in a want [lack] of knowledge of those things which your calling or state of life naturally supposes you to understand. A ploughman is not an ignorant man because he does not know how to read; if he knows how to plough. [1]

Hammerton goes further. He shows how we’re qualified for our calling, or profession, partly by our knowledge and partly by our ignorance since “everything we learn affects the whole character of the mind.” I would re-phrase him thus: we call a small proportion of knowledge ignorance; whereas we call, as science, a larger proportion of it.” This larger quantity,” he continues, “is recommended as an unquestionable good, but the goodness of it is entirely dependent on the mental product that we want. [2]

Moreover, I think we’ll also save time, even in learning these new features if we know what problems these versions and platforms are attempting to solve. I think we know, or are learning, about them; and we should frequently bring them to mind again as new versions and technologies being released and weigh them. Consider Java EE – one central problem it attempts to solve is distributed computing: how do I, as a computer, communicate with another computer existing on some other network to get some work done? If that computer uses Java objects, how do I talk to it? Java EE provides Remote Method Invocations(RMI) or EJBs. What if that computer doesn’t use Java? If it uses HTTP, try Web Services over HTTP. But that requires me to wait for a response before I can continue with my work. I don’t want to wait; I have other things I could be doing while the other computer is processing my request. Ok, use Messaging which is in Java EE. As these initial solutions were improved, newer versions come out. At the same time, other problems were becoming apparent and were looked at, such as, how do we make it easier for the developers to use what we’re providing? how do we ensure consistency between those implementing these services?

Knowing the problem, and recognising it as a problem, gives us a standard which allows us to consider the extent to which the solution addresses the problem, to compare the different solutions, and to determine how just and solid are the arguments in their favour. Then, perhaps an order of these solutions is established in our minds, ordered by how they address the problem – either addressing part of the problem or addressing the whole problem. That standard also becomes the criteria we use to classify these solutions. And each subsequent version or new technology can find its place in our mental catalogue. How do we go about learning about these problems and solutions?

• Reading blogs – how many would you need to read, notwithstanding the weak reasoning in some, or the repeated and diluted information in others?
• How about writing sample programs – how many would you have to write to determine the correct behaviour for all scenarios? How would you know all the scenarios that are likely to occur?
• Reading about the different frameworks, and probably their source code – how many to go through?
• What about StackOverflow? How about mailing lists? Or even books?

Of course, each has its place. But for a comprehensive view which, like a map of a city, shows you the major towns, their relation to each other, in terms of distance and size, and leaves you free to decide how to get from one town to the other, there’s no better place to start than by reading the Specifications – those of the Java language itself and those of Java EE, the Java Specification Requests (JSR). What I quoted from Hammerton still applies: “the goodness of it is entirely dependent on the mental product that we want.” The best expression on comprehensive views that I could find was Cardinal Newman’s:

. . . we cannot gain real knowledge on a level ; we must generalize, we must reduce to method, we must have a grasp of principles, and group and shape our acquisitions by means of them. It matters not whether our field of operation be wide or limited ; in every case, to command it, is to mount above it. Who has not felt the irritation of mind and impatience created by a deep, rich country, visited for the first time, with winding lanes, and high hedges, and green steeps, and tangled woods, and every thing smiling indeed, but in a maze? The same feeling comes upon us in a strange city, when we have no map of its streets. Hence you hear of practised travellers, when they first come into a place, mounting some high hill or church tower, by way of reconnoitring its neighbourhood. In like manner, you must be above your knowledge, not under it, or it will oppress you ; and the more you have of it, the greater will be the load. . . Instances abound; there are authors who are as pointless as they are inexhaustible in their literary resources. They measure knowledge by bulk, without symmetry, without design. [3]

Here are some high hills from which to reconnoitre the Java EE neighbourhood: JEE 5 (JSR-244) ; JEE 6 (JSR-316); JEE 7 (JSR-342). Page 6 of each of those JSRs has a diagram showing what the new version contributes to the old one, and the context of the subsequent JSRs; just as the Executive Summary of Annual Reports describes the business and the environment in which it operates.

References

1. William Cobbett, (1829), Advice to Young Men, and (Incidentally) to Young Women, in the Middle and Higher Ranks of Life, p. 119. Online pdf.
2. Philip Gilbert Hammerton, (1873), The Intellectual Life, p. 52
3. John Henry Cardinal Newman, (1886), The Idea of a University, 6th ed., pp. 139-140. Online PDF.