There are many great resources on the topic so just a short explanation:
Continuous integration refers to the practice of creating a build as new code is
committed and running automated tests against that build to verify that
everything works correctly. Continuous delivery is the ability to automatically
release any new code, usually with a single click.
In a way continuous integration is the basis for continuous delivery by 1B
requiring that the application can be automatically deployed in order to run
test which in turn 2B provide evidence that new code does not break the
application. The primary focus of CI/CD is automation.
How does ATS fit in CI/CD?
The Application Test Suite (ATS) is a low-code tool for automated testing on
GUI level, specialized for Mendix applications. As such it can provide great
value to development teams by frequently running a so called regression test
suite. ATS has two options for setting this up: 1B schedules and 2B the ATS API.
Schedules is a light-weight alternative for CI/CD that is meant for small
companies. It requires minimal set up and no external infrastructure. The
scheduling and configuration is all done on the ATS platform itself and alerts
can be sent when the tests finish. On the other hand, the ATS API offers a
much more versatile approach for automating the testing process. As such it is
geared towards large companies that have multiple apps and require more
control and a centralized place to manage and monitor the whole CI/CD
Smart Digital Factory
To help companies that want to incorporate CI/CD in the development process
Mansystems has launched the concept of the Smart Digital Factory which
comes with a set of tools specifically geared to Mendix development such as
ACR, ATS, AMS and APD. The toolkit includes a CI/CD pipeline solution that
comes with out of the box support for many Mendix related task such as
creating deployment packages, creating and restoring backups and running
automated tests with ATS. The toolkit is great for companies that are
committed to the Mendix platform and do not want to invest resources to set
up and maintain their own infrastructure.
ATS Jenkins plugin
That all said, as part of the dev team for ATS, I am aware that many users of
ATS already have existing infrastructure for CI/CD. So in this blog post I want
to show how to integrate ATS into your existing pipeline, with specific focus
on Jenkins. I choose Jenkins because of its ubiquity.
Although Jenkins is very popular it can be quite overwhelming and not easy to
extend especially for people with no coding experience. In order to enable
everyone to use ATS in combination with Jenkins I decided to write a custom
Jenkins plugin. On Github you can find both the source code and the plugin
In order to run an ATS test case or test suite with this plugin you only need to
add a single build step to Jenkins and specify the application id, an API token
and a job template ID. No additional configuration and no explicit HTTP
requests or XML parsing.
Optionally, failed tests can be automatically rerun. Forcing tests to fail at least
twice is a good strategy to deal with flakiness. To understand why, I recommend reading this great post. When executing an ATS build step, if the tests fail the Jenkins build will also be marked as failed. Additionally, ATS will log information to the console, which relates the build to jobs in ATS.
This plugin is installed manually. To do this, go to your Jenkins dashboard and
Under Advanced you should see the option to upload a file. Click Choose file
and then Upload.
Now you can add the build step Run ATS tests to your Jenkins Build.
The motivation for this blog post was twofold. First, I wanted to develop a
Jenkins plugin that is simple to use and requires no coding skills. The plugin
package is free to use for everyone. Second, by making the plugin open source I hope to provide a starting point and guide for people looking to integrate ATS into other CI/CD tools.
Thanks for taking the time to read this blog post, and I hope it helps you
develop great apps!