beryx.org posts

How to create modular jars that target a Java release before 9

admin@beryx.org — Wed, 21 Nov 2018 17:54:07 +0100

If you’re a library author, you probably want to see your library used in a large number of applications. One way to increase the number of applications that can include your library is to make it compatible with older Java versions. At the same time, you may consider modularizing your library in order to make it attractive for applications that take full advantage of the Java Platform Module System (JPMS).

However, the JPMS is implemented only by Java 9 and newer versions. So, how can you build a library that is both modularized and compatible with Java versions before 9?

A modular jar is just a normal jar that includes a module-info.class file, which represents the module descriptor. This module descriptor is typically produced by compiling a corresponding module-info.java file. The module descriptor is ignored when running on Java 8 and older versions, because module-info is not a legal Java identifier. This means that a modular jar compatible with a pre-Java 9 release can be produced by creating a normal jar for the target version and inserting a module descriptor into it.

The standard way to do this implies invoking the java compiler twice:

compile all sources except module-info.java using the appropriate javac flags to provide compatibility with the target pre-Java 9 version.
compile module-info.java using a Java 9+ compiler.

Maven

Maven describes a build method along the lines of the above mentioned strategy..

An alternative solution is offered by ModiTect, a Maven plugin for working with the Java 9 module system. One of the goals provided by ModiTect is add-module-info, which allows you to add a module descriptor to the JAR produced by the current Maven project:

...
<plugin>
    <groupId>org.moditect</groupId>
    <artifactId>moditect-maven-plugin</artifactId>
    <version>1.0.0.Beta1</version>
    <executions>
        <execution>
            <id>add-module-infos</id>
            <phase>package</phase>
            <goals>
                <goal>add-module-info</goal>
            </goals>
            <configuration>
                <jvmVersion>11</jvmVersion>
                <module>
                    <moduleInfoFile>
                        ${basedir}/src/main/java/module-info.java
                    </moduleInfoFile>                
                </module>
            </configuration>
        </execution>
    </executions>
</plugin>
...

ModiTect uses a clever technique to create the module descriptor without the need of a Java compiler. It analyzes the module-info.java file with the JavaParser and uses the ASM bytecode manipulation framework to generate the corresponding module descriptor. This way, it allows you to create a modular jar even if you use a pre-Java 9 compiler.

Gradle

Taking inspiration from ModiTect’s approach, I implemented a Gradle plugin for creating modular jars that target Java 8 or older versions.

Using this plugin is straightforward: in build.gradle you just need to add the plugin and specify the Java version targeted by your library:

plugins {
    id 'java'
    id 'org.beryx.jar' version '1.0.0'
}
sourceCompatibility = 1.8
targetCompatibility = 1.8

After placing the module-info.java file in the src/main/java directory of your project, you can build the modular jar with:

./gradlew jar

You don’t need a Java 9+ compiler to build the modular jar, because the plugin uses the same technique as ModiTect. However, keep in mind that this technique cannot check the validity of your module descriptor. References to nonexistent packages, modules, services, or service implementations go undetected.

Therefore, it is strongly advised to validate your module descriptor by additionally building your project using a Java 9+ compiler with the sourceCompatibility and targetCompatibility set accordingly. The good news is that you don’t need to change your build script in order to do this. The plugin lets you override the sourceCompatibility and targetCompatibility values by setting the project property javaCompatibility:

./gradlew -PjavaCompatibility=9 jar

Note that this project property overrides both sourceCompatibility and targetCompatibility with the same value.

If the plugin detects that at least one of sourceCompatibility and targetCompatibility has an effective value >= 9, it automatically applies the Chainsaw plugin, which adds support for the JPMS. That’s why no changes are required to your build script when running in Java 9+ compatibility mode.

In the discussion above we considered that sourceCompatibility and targetCompatibility are configured with pre-Java 9 values in build.gradle and you override them with 9+ values using the javaCompatibility project property. You can also work the other way around: set sourceCompatibility and targetCompatibility with 9+ values in build.gradle and override them with a pre-Java 9 value (via javaCompatibility) in order to generate a modular jar targeted to Java releases before 9.

Both approaches have their pros and cons. Choose the one that suits you the most. The important thing is to test your library constantly on both the class-path and the module-path. The best way to do this is to configure your continuous integration server to execute the Gradle build twice: once with the javaCompatibility property set and once without it.

Typically, a modular jar includes the module-info.class file in its root directory. This may cause problems for some older tools, which incorrectly process the module descriptor as if it were a normal Java class. To provide the best backward compatibility, the plugin creates by default a modular multi-release jar with the module-info.class file located in META-INF/versions/9, as shown below.

If, for some reason, you don’t want to create a multi-release jar, set the multiRelease property to false in the jar section of your build script:

jar {
    ...
    multiRelease = false
    ...
}

Examples

To illustrate how to use this plugin, I implemented a Java library and a Kotlin library for solving the N-Queens problem.

Note how Travis is configured in these projects to test the library on both the module-path and the class-path:

./gradlew -PjavaCompatibility=11 --no-daemon -i -s build
./gradlew --no-daemon -i -s build

Add code snippets to your technical presentations

admin@beryx.org — Wed, 01 Aug 2018 00:14:42 +0200

Most of my technical presentations contain code snippets, but putting nice looking code into my slides was always a tedious task. After years of using PowerPoint, I started looking for alternatives. There are many good ones, and I was able to find a solution that matches both my personal preferences and my typical use cases.

My source code is usually Java, Groovy, or Kotlin. For each presentation, I set up a project containing the code examples I’m going to use in my slides. I want the presentation files to integrate seamlessly into this project. Below are a few aspects that I considered in order to tailor my solution.

Version control

Source code and presentation files should be managed using a version control system (VCS). I use git, but any decent VCS will do.

With a VCS, you can use branches to manage several variants of the same presentation, each one refined for a different audience.

To be able to compare different versions of the presentation files, they need to be written in a text format. I opted for AsciiDoc.

Build tool

Besides building the code examples, the build tool should also convert the presentation files into a slide deck. Asciidoctor provides a Gradle plugin, a Maven plugin, and an Ant task for this job.

There are also several presentation backends to choose from: deck.js, reveal.js, DZSlides, Bespoke.js. The build tool should also take care of downloading and configuring the chosen backend.

I settled on Gradle and deck.js.

Keeping the slide content in sync with the codebase

To ensure that the slides always contain valid and up-to-date code, the code snippets should be retrieved directly from your codebase. Luckily, AsciiDoc lets you include whole source code files or only portions of them in your presentation files.

The example below shows how to add a code snippet containing the main method to your slides.

src/main/java/org/example/Hello.java

package org.example;

public class Hello {
    // tag::main[]
    public static void main(String[] args) {
        System.out.println("Hello, world!");
    }
    // end::main[]
}

asciidoc/presentation.adoc

:sourcedir: ../src/main/java

[source,java]
----
include::{sourcedir}/org/example/Hello.java[tags=main,indent=0]
----

As you can see, the source code portion to be included (in our case, the main method) is marked with user-defined tags appearing as comments in the code. I would probably be annoyed if I saw these comments cluttering production code, but I’m totally fine with them in a codebase used for presentation purposes only.

Putting all together

To make your life easier, I created a BootHub template that sets up a customized presentation project for you. It lets you configure things like project name, presentation title, author, programming language, or the base package of your project.

BootHub will generate a presentation file in accordance with your settings. To give you a quick start, the content of this presentation file illustrates some of the AsciiDoc and deck.js features that allow you to create a great slide deck: tables, images, blockquotes, transitions, incremental reveal, source code inclusion.

To build the codebase and create the slides execute:

./gradlew build asciidoc

Then, open the index.html file found in build/asciidoc/html5 in your browser.

Click on the image below to see the slide deck produced by a generated presentation file.

Sample slide deck generated by BootHub

If you prefer another build tool or another presentation backend, I encourage you to adapt my template to your tastes and publish your version to BooHub. I’m sure a lot of people will thank you for that. See the documentation about writing BootHub templates.

And if you want to learn more about BootHub, i recommend you this article.

Create java.awt.Color from string representation

admin@beryx.org — Tue, 03 Jul 2018 21:37:26 +0200

When using the Text-IO library, you can configure the text terminals via properties files. Colors can be specified using standard HTML names (e.g. orange, chocolate, crimson), hex strings (e.g. #ee5533, #e53, 0xEE5533), rgb format strings (e.g. rgb(217,33,119), rgba(112,25%,50%,0.9)), or hsl format strings (e.g. hsl(240,90%,70%), hsla(270,0%,100%,0.7)).

How does Text-IO create colors from these string representations? Until recently, the implementation used the static method web(String colorString) provided by javafx.scene.Color. This method supports all the above mentioned formats and returns a javafx.scene.Color instance. Text-IO needs a java.awt.Color, but creating one from the returned javafx.scene.Color is straightforward.

However, I was not happy with this solution. It made my code depend on JavaFX, although Text-IO doesn’t actually use JavaFX. And starting with JDK 11, JavaFX is no longer part of the JDK.

So I decided to write a little library named AWT Color Factory for creating java.awt.Color objects from string representations. The simplest way to do this is to copy the code of the javafx.scene.Color.web(String colorString) method and adapt it to create java.awt.Color instances.

But, what about licensing? JavaFX is released under the GPL, so my library must use the same license. Text-IO is instead released under the Apache-2.0 license. Is it then allowed to use my library? Yes, it is. The license under which JavaFX is released is actually not GPL, but GPL 2 with Classpath exception. The same applies to my library. This is very important, because the Classpath exception clause gives you permission to include the AWT Color Factory library in your executable code, regardless of the license under which your software is distributed.

Example usage

Color c1 = ColorFactory.valueOf("firebrick");
Color c2 = ColorFactory.valueOf("#aa38e0");
Color c3 = ColorFactory.valueOf("0x40A8CC");
Color c4 = ColorFactory.valueOf("rgba(112,36,228,0.9)");
Color c5 = ColorFactory.web("forestgreen", 0.7);
Color c6 = ColorFactory.web("hsl(270,90%,70%)", 0.8);

AWT Color Factory is available in JCenter and Maven Central.

Maven

<dependency>
    <groupId>org.beryx</groupId>
    <artifactId>awt-color-factory</artifactId>
    <version>1.0.0</version>
</dependency>

Gradle

compile 'org.beryx:awt-color-factory:1.0.0'

The library requires Java 7 or newer. The jar artifact is modularized under the name org.beryx.awt.color.

Starting with version 3.2.0, Text-IO uses the AWT Color Factory library and no longer depends on JavaFX.

Partial mocks for Java 8 interfaces with default methods in Spock

admin@beryx.org — Fri, 05 May 2017 01:43:55 +0200

Spock 1.1 has been released today and includes a series of new features and improvements. One of them, which is my contribution, is the ability to create partial mocks for Java 8 interfaces that contain default methods.

Let’s see how it works:

By creating a spy for the ISquare interface and mocking the getLength() method, we are able to test the default implementation of the getArea() method.

Nothing surprising here. It’s probably exactly what you expected. But it didn’t work with Spock version 1.0 or earlier.

Java console applications for the Web

admin@beryx.org — Mon, 09 Jan 2017 22:07:44 +0100

A few weeks ago I released the first version of TextIO, which is a library for creating Java console applications that read interactive input from the user. I encourage you to read this DZone Article and the TextIO documentation in order to learn more about this library.

Today I released the version 1.7, which adds a new interesting feature: the ability to access your console application via a browser, by using a WebTextTerminal. See the image below for an example:

Unlike the other text terminals provided by TextIO, the WebTextTerminal works only in conjunction with a web server supporting the DataApi (such as the SparkDataServer) and a web page that contains code for accessing this API. This is typically accomplished via textterm.js, as shown in the code snippet below.

<div id="textterm">
    <p class="textterm-pair">
        <span class="textterm-prompt"> </span>
        <span contenteditable="true" class="textterm-input"> </span>
    </p>
</div>

<script src="textterm.js"></script>
<script>
    var textTerm = TextTerm.init(document.getElementById("textterm"));
</script>

Look at the source code of WebTextIoExecutor.java and web-demo.html for more usage details.

Currently, only WebKit-based browsers (such as Chrome, Opera or Safari) are able to mask input strings. Keep this in mind when working with sensitive data.

Shuffling very large sequences

admin@beryx.org — Sun, 16 Oct 2016 23:18:43 +0200

A few days ago I released the first version of the Streamplify library, which offers a series of useful streams (most of them related to combinatorics) and other utilities for building streams that allow efficient parallel processing.

I will not give an overview of the Streamplify library, because I already published an article on DZone about it. Instead, I will discuss the ability to shuffle the streams created with this library.

Shuffling a sequence is not a trivial task if the number of elements in the sequence is very large. Still, such large cardinalities occur frequently when working with combinatorics streams. Take for example the stream of all possible arrangements of the playing cards in a standard 52-card deck. There are 52! arrangements, which is a 68-digit number roughly equal to the number of atoms in our galaxy. Such huge numbers make impossible to store the elements of a stream. They have to be generated the moment they are needed.

The elements of a stream provided by Streamplify can be generated based on the value of the previous element in the stream or based on their rank. The rank of an element is given by its index in the stream. Therefore, shuffling the elements of a Streamplify stream implies defining a (pseudo-)random permutation of their indices. For a stream with N elements, this means to generate a permutation of the numbers 0, 1, ..., N-1. The problem is that N is often astronomically high.

An overview of the most common approaches for solving this problem is given in this blog post. Some of the methods described in the blog can only generate the permutation elements in a sequential way. This means, the K element of the permutation can be obtained only after generating the elements 0, 1, ..., K-1. Such methods are not appropriate for Streamplify, because the stream source is sometimes splitted in order to parallelize operations, and the value of the previous element in the stream is not available immediately after a split. The other methods are based on format-preserving encryption. Some of them are restricted to predefined block sizes, while others require encryption keys whose generation is computationally expensive.

I took a pragmatic approach and devised a shuffling algorithm that is fast, memory efficient and decently scatters the elements, although not in a uniformly distributed manner. For a given index, each byte in its representation is handled separately. First, it is XORed with its successor byte (if it exists). Then, a permutation is applied on the resulting byte. The permutations to be applied to each byte are pseudo-randomly generated during the initialization of the shuffler. Each byte-permutation has only 256 elements, therefore they are generated fast. The algorithm uses only 4 byte-permutations, applying them in circular order. Because the BigInteger representation of a number may have a number of bits such that the most significant ones do not fit exactly in a byte, the algorithm also generates so-called bit-permutations to be applied on byte fragments. During its initialization, the shuffler creates 7 bit-permutations, corresponding to byte fragments with 1, 2, …, 7 bits.

Let’s consider that the algorithm has generated the following permutations:

bytePerm[0]: [3C, B3, 2B, BD, 64, FF, A9, 3B, 2A, EC, 81, 57, 35, 0A, D7, E3, ...] (256 elements)
bytePerm[1]: [99, 41, F6, 8C, E9, C3, 6E, 3C, E0, 22, 89, 15, 42, BC, 4D, A3, ...] (256 elements)
bytePerm[2]: [C5, 14, 42, 8C, 6A, 9F, 85, 86, 7C, 0B, 7A, C7, 93, 3D, BA, 40, ...] (256 elements)
bytePerm[3]: [D9, 74, 6C, 92, E8, DE, 07, C6, DA, 56, A0, F3, BA, A3, CC, 87, ...] (256 elements)

bitPerm[1]: [01, 00]
bitPerm[2]: [03, 00, 01, 02]
bitPerm[3]: [02, 05, 06, 03, 04, 00, 07, 01]
bitPerm[4]: [0D, 08, 0E, 00, 05, 0A, 09, 0B, 07, 03, 01, 0F, 06, 0C, 04, 02]
bitPerm[5]: [1D, 04, 18, 13, 11, 02, 10, 0A, 1A, 03, 00, 0F, 17, 05, 06, 12, ...] (32 elements)
bitPerm[6]: [06, 29, 20, 26, 01, 15, 3E, 19, 36, 07, 3F, 2E, 12, 0A, 1B, 28, ...] (64 elements)
bitPerm[7]: [15, 65, 5F, 43, 19, 60, 61, 57, 51, 56, 22, 24, 23, 6F, 6E, 21, ...] (128 elements)

We will apply now the shuffling algorithm to the index 2345678765432 of a sequence with 7654321234567 elements.

count: 7654321234567 = [06 F6 29 19 22 87] (43 bits = 5 bytes + 3 bits )
index: 2345678765432 = [02 22 25 59 7D 78]
idx <-- byte[5]       = 78 ^ 00 = 78; shuffled[0] <-- bytePerm[0][78] = FB
idx <-- byte[4] ^ idx = 7D ^ 78 = 05; shuffled[1] <-- bytePerm[1][05] = C3
idx <-- byte[3] ^ idx = 59 ^ 05 = 5C; shuffled[2] <-- bytePerm[2][5C] = 71
idx <-- byte[2] ^ idx = 25 ^ 5C = 79; shuffled[3] <-- bytePerm[3][79] = F7
idx <-- byte[1] ^ idx = 22 ^ 79 = 5B; shuffled[4] <-- bytePerm[0][5B] = 86
idx <-- byte[0] ^ idx = 02 ^ 5B = 01; shuffled[5] <-- bitPerm[3][01] << 5 = 05 << 5 = A0

On each line in the listing above you can see the two operations performed on a byte: executing a XOR with its successor byte and applying a permutation. In the last line a 3-bit permutation is applied, because the representation of count requires 43 bits, thus containing a 3-bit fragment. Note that the obtained value is then shifted (8-3) = 5 positions to the left in order to join this 3-bit fragment to the other bytes.

Note also that the shuffled bytes are computed in reversed order. For example, shuffled[0] is computed based on the value of byte[5]. This way, any two consecutive indices will have corresponding shuffled values that differ in their most significant byte. This leads to a good scattering of the shuffled values.

The drawback is that whenever the distance between two indices is a multiple of 256 (which implies that their least significant bytes have identical values), the most significant bytes of the shuffled values of these indices will be identical. In order to fix this problem, the algorithm performs a XORing of neighboring bytes in the shuffled value, starting with the next-to-last right byte:

from right to left XOR with next byte: FB C3 71 F7 86 A0 --> 98 63 A0 D1 26 A0

Finally, the resulted shuffled value is shifted (8-3) = 5 positions to the right in order to obtain a number with the right magnitude:

shuffledIndex <-- BigInteger([98 63 A0 D1 26 A0]) >> 5 = 167553667245728 >> 5 = 5236052101429

The BigInteger representation of the shuffled value has at most the same number of bits as the representation of the maximum allowed index (that is, count-1). Still, it is possible that the shuffled value is greater than the maximum allowed index. In this case, the algorithm will be run again with the shuffled value as the input index. This process is repeated until a shuffled value in the admitted range is obtained.

As mentioned before, the values generated by this algorithm are not uniformly distributed. This can be demonstrated for example by applying the Chi-squared test. However, the shuffled values are nicely scattered, which means that the algorithm is good enough for most practical purposes.

I wanted to see if it’s possible to observe that the shuffled values are not exactly random without using statistics tests. Therefore I wrote a small program that displays shuffled values based on their binary representations. Each value is shown as a line with black pixels for 1s and white pixels for 0s. One drawback of the algorithm is that the permutations are applied on each byte separately, which makes that two indices with one identical byte in the same position will also have one identical byte used in the construction of their corresponding shuffled values. Thus, it is more probable to observe non-random patterns for indices at distances that are multiples of 256. The figure below has five columns, each corresponding to a sequence of 256 indices. In the first column, the indices have consecutive values. In the second, they are an arithmetic progression with the common difference 2⁸ = 256. In the third column, they are an arithmetic progression with the common difference 2¹⁶, and so on.

Everything looks pretty random in the above image, which means that the algorithm does a good job at hiding its flaws. But a simple modification will change this. Starting from left, let’s XOR each byte with its successor. The result is shown below:

Except for the first column, where indices have consecutive values, the other ones exhibit non-random patterns. The number of stripes grows with the exponent used for the common difference in the arithmetic progressions.

Should you worry about this? No, unless you intend to use the shuffling algorithm for hardcore scientific research. In practice, it is very unlikely that you will observe any non-random behavior, so use the shuffler along with all the other goodies provided by Streamplify.

Logic for logic-less templates

admin@beryx.org — Thu, 07 Jul 2016 20:36:12 +0200

There are many situations where content needs to be created automatically. The most common case is probably the dynamic generation of web pages, for which template engines are a popular choice. Some template engines take a logic-less approach, by denying or discouraging the use of logic in templates. Examples of logic-less template systems are Mustache and Handlebars, for which Java implementations are also available: Mustache.java and Handlebars.java.

Logic-less templates seem to polarize opinions. Proponents see them as the best way to achieve a clear separation of concerns in MVC architectures. Opponents argue that forcing the controller to do data-massaging before sending the data to the UI is actually a bad thing, because it forces the controller to be aware of the presentation logic.

I will not take a side in this debate, because I have very little experience in developing web applications. However, template engines are also useful for other purposes and I argue that there are cases, especially in non-MVC settings, where using logic in templates is legitimate. Nonetheless, putting too much logic in a template is still a bad idea.

I used recently Handlebars.java for generating source code and I needed to add some logic to my templates. Handlebars.java allows you to do this through the use of helpers, therefore I wrote the Handlebars.java Helpers library, which provides a series of helpers for common use cases. Most of them are basic helpers that can be used as subexpressions in the built-in block helpers of Handlebars.java. This lets you write the template logic in a fluent way.

Example

Given the following YAML model:

birthYear: 1997

and the following template:

{{def 'fifteenYear' (math birthYear '+' 15)}}
{{#ifb (or
    (and
        (compare (math fifteenYear '%' 4) '==' 0)
        (compare (math fifteenYear '%' 100) '!=' 0)
    )
    (compare (math fifteenYear '%' 400) '==' 0)
   )
}}
Your fifteenth anniversary was in a leap year!
{{else}}
Your fifteenth anniversary was in a non-leap year!
{{/ifb}}

The resulting text will be:

Your fifteenth anniversary was in a leap year!

If you use Handlebars.java, you may find this library useful, so read its documentation and give it a try!

In-memory URL content

admin@beryx.org — Thu, 02 Jun 2016 23:27:32 +0200

A few days ago I had to write a suite of integration tests, where each test case required a URL as input data. In my first implementation, I stored the URL content of each test case in a separate file and used the corresponding file URL as input data. However, I was not happy with this solution, because in order to understand what a test case does, one had to open the associated file containing the URL content. For my test cases, each URL content consisted of only a few lines of text, so I wanted to keep the URL content in the source code of the test case, as a string.

One idea was that each test case implementation will create a temporary file containing its associated content string. But this was boring, so I started thinking about providing a URL with a custom protocol, which will retrieve its content from memory. One way to do this is to escape the content and store it as the URL’s query part. However, many browsers and servers have a limit of about 2000 characters for the length of a URL. Because I did not want to restrict the number of characters allowed for the URL content, I decided to keep the URL contents as values in a static map with integer indexes. You can see the implementation below:

The above implementation is in Groovy, but it can be easily ported to Java.

Of course, this solution works only if the module that retrieves the URL content runs in the same JVM as the code that created the URL.

In my integration test suite, I used a relatively small number of URLs with a small content size, therefore the growing size of the inMemoryMap was of no concern. But if your use case involves a huge number of URLs and large content sizes, you should add housekeeping capabilities to the InMemoryProtocol.

JavaFX controls and skins

admin@beryx.org — Mon, 30 May 2016 23:48:26 +0200

JavaFX controls follow the MVC pattern, which allows multiple views for the same model. Therefore, multiple Skins can be provided for a Control. A good example is the Medusa library, which offers a gauge control with about 30 different skins.

I wanted to play a bit with JavaFX controls and skins, so I decided to write my own gauge control. My goal was to keep the code simple and easy to understand, underlining the separation between model (the gauge) and views (the skins). The result is the JFXGauge library.

JFXGauge provides only two skins. The first one is probably the most simple skin you can think of. It consists only of a Text component. However, using CSS styling you can customize even this skin in many interesting ways:

(click for animated gif)

The second skin is a thermometer skin and is more complex:

(click for animated gif)

For this skin I also defined a few styleable properties that allow configuring the thermometer’s aspect ratio and the position of the range and threshold markings. Implementing styleable properties involves writing boilerplate code, but Hendrik Ebbers’ CSS Helper reduces the amount needed.

JFXGauge is available in Maven Central and JCenter.

Try the demo application included in the latest release, read the documentation and use the gauge in your projects!