Managing Java Dependencies for Mendix Modules

If you have ever tried to implement a certain function as a reusable module in Mendix, there’s a good chance you’ve had to add some jar dependencies. There are multiple ways of achieving that goal. But there’s one method that I find usually works best, because it simultaneously addresses multiple issues.

But first, let me rank the game-breaking or simply annoying issues I have encountered when trying to implement a reusable module:

  1. If you want to use a jar file in your module, e.g. pdfbox v2.3, but another version of this jar e.g. v1.8 is already being used in a another module of your application, you are out of luck. As far as I know, this is simply not possible. The class loader will pick up only one version, leaving one of the two modules to deal with the wrong version, which usually results in the application hanging or crashing. To make matters worse, this only occurs when invoking a specific java action with a conflicting dependency at runtime.
  2. If you are maintaining a Mendix module and you need to update a jar file within it, you need to make sure everyone deletes the old jar on update. Otherwise, you end up having two jars with a different version in your classloader path, which again likely leads to the application not working. For more on this specific issue, check out a post on the idea forum.
  3. Mendix requires you to manage transitive dependencies manually. This usually means you have to run the code to see which classes are missing, then find the right jar file and add it to the project manually. Then, rinse and repeat until there are no more “NoClassFound” errors.
  4. Wasting time when exporting modules is my next pet peeve. There are too many jars to include/exclude, especially if you are using something like community commons or the rest module in your project. Of course, that could be solved simply by a select/deselect button in the export dependencies dialog, but that is not the point of this post. One way or another, time is lost.

One way to effectively deal with all of these issues is to employ a build tool to produce a so-called fat jar, a single jar file that contains all your dependencies. This on its own resolves multiple issues from the previous list, including: rom the fat jar file will be updated automatically when the module is updated, thus solving point 2; the build tool can take care of all the transitive dependencies, thus solving point 3, and finally the only dependency is a single one-jar file, eliminating point 4.

The first issue we mentioned can be resolved using a technique called shadowing. Shadowing replaces patterns in class names with a given string. For example, you can replace org.json with community.commons.org.json. This lets the Java classloader load two versions of the org.json library because they have different class names.

Case study: community commons

What better module to demonstrate the techniques described above than the community commons? In its current status, it has some 20 or so dependencies (check out that scroll bar).

In this instance, I will be using Gradle. But you can opt to do the same thing using other tools e.g. Maven or JarJarLinks.

dependencies

Adding dependencies to Gradle

First, I installed the Gradle Eclipse plugin after downloading it from the Eclipse marketplace. Next, I created a new Gradle project. The main file in every gradle project is the gradle build script, where many options can be specified, such as which Java version to use. The dependencies for a Gradle project are also defined in the build script. As you can see below, my first iteration of the build script is mostly standard stuff with the exception of the shadowing tool that I added:

buildscript {
repositories {
mavenCentral()//look for dependencies here
}
dependencies {
classpath "com.github.jengelman.gradle.plugins:shadow:2.0.0"
//this is the tool we will use to build the fat jar and shadow it
}
}

plugins {
id 'com.github.johnrengelman.shadow' version '2.0.0'
id 'java'
}

group 'com.mendix.community-commons'
version '1.0.0'

apply plugin: 'java'
apply plugin: 'maven'
apply plugin: 'eclipse'

task wrapper(type: Wrapper) {
gradleVersion = '3.0'
}

compileJava {
sourceCompatibility = 1.8
targetCompatibility = 1.8
}

repositories {
mavenLocal()
mavenCentral()
}

dependencies {
}

So far so good. Next, I started adding the dependencies from the community commons module:

dependencies {
// https://mvnrepository.com/artifact/org.owasp.antisamy/antisamy
compile group: 'org.owasp.antisamy', name: 'antisamy', version: '1.5.3'

    // https://mvnrepository.com/artifact/com.google.guava/guava
compile group: 'com.google.guava', name: 'guava', version: '14.0.1'

    // https://mvnrepository.com/artifact/commons-codec/commons-codec
compile group: 'commons-codec', name: 'commons-codec', version: '1.10'

    // https://mvnrepository.com/artifact/org.apache.pdfbox/jempbox
compile group: 'org.apache.pdfbox', name: 'jempbox', version: '1.8.5'

    // https://mvnrepository.com/artifact/joda-time/joda-time
compile group: 'joda-time', name: 'joda-time', version: '2.9.6'

    // https://mvnrepository.com/artifact/commons-fileupload/commons-fileupload
compile group: 'commons-fileupload', name: 'commons-fileupload', version: '1.2.1'

    // https://mvnrepository.com/artifact/commons-io/commons-io
compile group: 'commons-io', name: 'commons-io', version: '2.3'

    // https://mvnrepository.com/artifact/org.apache.commons/commons-lang3
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.0'

    compile group: 'org.apache.servicemix.bundles', name: 'org.apache.servicemix.bundles.batik', version: '1.8_1'

    // https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox
compile group: 'org.apache.pdfbox', name: 'pdfbox', version: '2.0.3'

    // https://mvnrepository.com/artifact/xerces/xercesImpl
compile group: 'xerces', name: 'xercesImpl', version: '2.8.1'
}

I noticed that some of the dependencies are not listed in Maven Central (or at least I could not find them). No problem—I have the jar files from the community commons project on GitHub. I created a folder libs in my Gradle project, and then added the com.springsource.org.apache.batik.css-1.7.0.jar and nekohtml.jar to it. Then, I added the following line to my dependencies, which, as you might expect, adds all jar files from the libs folder to the gradle project.

compile fileTree(dir: 'libs', include: '*.jar')

Doing this, we have now resolved all Java dependencies.

Dealing with Mendix classes

My basic premise is to use the Java action to only call a corresponding function from the fat jar that I create. This is already the case for most Java actions in community commons, i.e. the actual implementation code is not inside the java action class. Instead, execution is delegatad to another class. I copied the classes from the communitycommons package where the logic is implemented, specifically ConversationLog, DateTime, ORM, StringUtils, etc., to my Gradle project. I built the project and refreshed the build path in Eclipse. Success ... at least somewhat. The dependencies are loaded and recognized by Eclipse, but I can see some missing classes. Most of the missing classes are from the Mendix API, which is used extensively in the community commons in the form of IContext, IMendixObject, Core, etc. I added them to the dependencies as well.

compile files('C:/Program Files/Mendix/7.2.0/runtime/bundles/com.mendix.public-api.jar')
compile files('C:/Program Files/Mendix/7.2.0/runtime/bundles/com.mendix.logging-api.jar')

I need to include these classes when developing in Eclipse because the compiler has to recognize them. Otherwise, the project will not compile. However, I do not want the Mendix API classes to go into my fat jar, so I excluded them.

shadowJar {
dependencies {
exclude 'com/mendix/**'
}
}

Rebuild and refresh, a lot of unrecognized classes are now ok. Here is a screenshot of my gradle project at this point:

stringutiles

But I can still see a few imports which are unrecognized. These were enumerations and proxy classes from the Mendix Modeler, such as system.proxies.FileDocument. I could not think of a way to add them to my list of dependencies, so I decided to wrap them. I declared an interface IFileDocument with the following code:

public interface IFileDocument {
public IMendixObject getMendixObject();
public boolean getHasContents();
}

I will only work with this interface within the fat jar. Then, I added a method to convert a system.proxies.FileDocument to an org.community-commons.main.IFileDocument. Every time a Java action needs to call some method from my fat jar file that works with files, I do a conversion, and then pass the interface.

misc-java

That just leaves the enumerations and the system.proxies.Language. For the enumerations e.g. communitycommons.proxies.LogLevel, I used a similar wrapping method to the one I employed for the FileDocument. For the Language I decided to just keep the whole code (all four lines of it) in the Java action. It has no other external dependencies, so why bother? If needed, though, it can be wrapped in a similar way. I did another rebuild and added the fat jar to the project. After everything compiles, these are the dependencies that remain in the modeler.

dependencies-2-1Let me make a small digression here to talk about wrapping Mendix proxy classes. This solution is clearly not what you would normally want to do. But on the other hand, it is mostly mechanical work. The way I see things, this means there is some way to automate it. A tool can be developed that generates code for the wrapper classes. You could even go a step further and have the tool replace all usages of proxy classes with the corresponding wrapper classes.

Adding resources to a fat jar

As you can see in the screenshot, we managed to get rid of most files, with the exception of the antisamy XML files. I could just leave them as they are and everything would work fine, but they are really bugging me. For the sake of being thorough, I will demonstrate how any resource files can be included in the fat jar. But this is clearly optional.

folder-structure-1As you probably already know, a jar file is just a zip file, which means we can include any file we want in it. In fact, Gradle automatically considers any files that are in a folder named resources as resource files and adds them to the jar. I created such a folder under src/main and copied all the XML files there. To test that the resources are really added, I did a rebuild and then extracted the jar file. I could see that all XML files are really included in the fat jar.

Next, I had to change the way the xml files are read. Previously they were loaded from the resources folder just like regular files. However, this is not possible once they are packaged in a jar file. Instead, you should use the classloader to load them as resources. I changed the following lines in StringUtils.XSSSanitize (which is part of the community commons):

String filename = Core.getConfiguration().getResourcesPath() + File.separator
+ "communitycommons" + File.separator + "antisamy"
+ File.separator + "antisamy-" + policyString + "-1.4.4.xml";
Policy p = Policy.getInstance(filename);

Instead of using a filename, I use the classloader to get the file contents as a stream:

code-3

Finally, we are left with only three dependencies:

dependencies-3-1

  • java that contains classes to interface with the fat jar,
  • the fat jar file, and
  • txt - a license file.

Because I do not see a way to reduce the number of dependencies any further, we can now move on to the next topic:

Shadowing

To keep it simple, I will just prepend the cc_ prefix to all external classes with the following code:

folder-structure-1-1

shadowJar {
relocate('org.apache', 'cc_org.apache')
relocate('org.cyberneko', 'cc_org.cyberneko')
relocate('org.joda', 'cc_org.joda')
relocate('org.owasp', 'cc_org.owasp')
relocate('org.w3c', 'cc_org.w3c')
relocate('org.xml', 'cc_org.xml')
relocate('javax', 'cc_javax')
relocate('java_cup', 'cc_java_cup')
relocate('com.google', 'cc_com.google')
dependencies {
exclude 'com/mendix/**'
}
}

Gradle will automatically replace the matching class names both in the files where these classes are defined and in the files where these classes are used.

Notice how I left out the org.community_commons package. These classes do not involve any third-party dependencies, so there is no danger of conflicts arising.

By opening the jar archive, I can confirm that all the classes that come from an external dependency are, in fact, renamed. That is all that's needed. Now, developers who use community commons in their project can use libraries that are also used by the community commons module without worrying about conflicts.

code-2-1For example, let us assume we want to use the pdfbox library from Apache that is used by community commons also because there is a newer version with some feature we like, or there is component that has a dependency on an earlier version of pdfbox. That is now possible. We can add the pdfbox.jar to the userlib folder and use it in our code without running into any dependency-related issues.

At this point, I would like to stress that although it is technically possible to use shadowed classes in your Mendix project, as shown in this screenshot, this really should be avoided at all cost. There is no guarantee that the shadowed class will not be removed/renamed, and thereby no longer available when the module is updated.

Final thoughts

Packaging and shadowing dependencies correctly goes a long way to prevent headaches when using Mendix modules. My hope is these examples will motivate developers of app store modules to start managing their internal dependencies in a way that makes everybody’s life a bit easier. Perhaps if there is enough interest and support from the community, the process of shadowing jar file dependencies also will become part of some future Mendix best practices document for module—or, better still, we will get an integrated build/shadowing tool inside the Mendix modeler.

I hope you found this post interesting. If you have any remarks or ideas on how to improve this post or the code herein, please reach out to me. You can also check out the complete project on Github.

Happy coding!

-Andrej Gajduk

Andrej Gajduk

OUR LATEST TWEETS