While Drupal moved its code hosting to GitLab five years ago, moving testing from our home-grown DrupalCI to GitLab CI has been a gradual process. We are announcing several important changes and future plans in this post that affect all developers working on code on Drupal.org.
Contributed projects should now use Gitlab CI
As of July 2023, contributed projects are fully equipped to adopt GitLab CI, and should do so.
The DrupalCI system will be retired once its functionality for Drupal core and the Security Team has been fully replaced by GitLab CI. While we don't have a specific date for that yet, it will be retired within a year.
Testing core is now five times faster
We've been adapting our GitLab CI configuration for better core testing. Core testing on GitLab CI is more modern and significantly faster than Drupal CI. Drupal CI core runs take just over 50 minutes, but GitLab CI can run the same test suite in under ten minutes!
How did we do it?
You don't need to know the internal details of how this was achieved to benefit from it, but here is a high level summary, if you are interested:
- Mounted MySQL and the filesystem on ramdisks for faster access.
- Introduced different jobs for different linting types and test suites so they start at the same time.
- Specified explicit job dependencies with the 'needs' keyword so that jobs can start sooner.
- Made tests run concurrently on each runner via
run-tests.sh. - Used parallel test runners for functional tests to allow over 100 tests to be executed simultaneously.
- Identified the slowest tests and tagged them with
@group #slow. Using this, we then distribute the slowest jobs at the beginning of each run and between runners. This means they finish as early as possible, instead of continuing to run when all the shorter jobs have finished. - Refactored the slowest tests that take more than six minutes for a single test class into multiple classes so those can run in parallel.
Core on-commit testing and scheduled testing now moved to GitLab CI
Due to the performance improvements, we moved several testing setups from DrupalCI to GitLab CI. Drupal core is automatically tested on GitLab CI following each commit and on a set schedule to give early warning of possible regressions.
GitLab CI test matrix results shown for a commit
Take advantage of fast test runs and early feedback in core merge requests today!
Merge requests filed in GitLab for core issues will automatically benefit from all these improvements. When coding standards are not met or test failures occur, the GitLab UI makes it easier to see which parts of the pipeline have failed and why, and will report back on GitLab as soon as they're done. No more waiting for the full pipeline to complete!
If you have an existing merge request from before GitLab CI support was added to core, rebase it on the 11.x branch to automatically gain support.
Test-only runs are now automated in GitLab CI
To prove that added test coverage for a bugfix is working as intended, we often require a patch or merge request that has only the test coverage additions (without the code for the bug fix). This has meant that either two patches or two merge requests for each issue were needed. Now, we have an automated solution to run only the changed tests. So, in most cases a separate merge request will no longer be needed! If an issue using patches is missing a test-only patch, it will be marked "Needs work" and must be converted to a merge request (so that the test-only CI job can run without the need to manually create a test-only patch).
GitLab CI child pipeline showing automated test only option
Code quality checks are now integrated in the merge request user interface
Another major developer experience improvement is that code quality checks are now integrated into the merge request user interface, with reports displayed separately from test results and even displayed inline in the code review interface.
GitLab integrated code quality check report
GitLab CI jobs are re-run when failures are encountered outside tests
Another improvement over DrupalCI is that GitLab CI jobs will re-run automatically (up to 2 retries) if there were failures outside of tests, such as timeouts, scheduling issues, etc.
Thank you!
All of these improvements are thanks to tremendous work by Moshe Weitzman, Lee Rowlands, Kirill Roskolii, Chris Burgess, Nick Schuch, Fran Garcia-Linares, Luhur Abdi Rizal, Michael Strelan, Alex Pott, Nathaniel Catchpole, Dave Long, Björn Brala, Neil Drumm, Wim Leers, Juraj Nemec, and the Drupal Association for prioritizing developer tools improvements. This will be an incredible benefit for developer productivity!
Future milestones
Automated performance testing is coming
OpenTelemetry Application Performance Monitoring capability was recently added to Drupal core. More infrastructure is needed to collect that data including GitLab CI configuration updates to run performance testing on a schedule.
Internal Security Team testing will move to a private GitLab.com mirror
DrupalCI is still used for testing core patches privately for security releases. We plan to move this to a private GitLab.com mirror with issues privately submitted on GitLab.com.
Drupal 7 core and contributed projects will move to GitLab CI
We are currently preparing Drupal 7 core for Gitlab CI and preparing CI templates for Drupal 7 contributed projects. Drupal 7 test run times on GitLab CI are down to three minutes and include PHP compatibility checks, which were previously not run on Drupal 7 patches. Once those are finalized, only GitLab CI will be used for testing Drupal 7.