Like many folks, I've been fascinated by the incredible evolution of AI tools in 2023. At the same time, I've been working to figure out exactly where AI fits into my various roles as a Drupal developer, trainer, and business owner.
In this blog post, I detail a recent exploration I made into using AI (GitHub Copilot, to be precise) to generate a PhpUnit test in a Drupal contrib module that I maintain.
tl;dr I was impressed.
Prerequisites
For this exercise, I used Visual Studio Code with the GitHub Copilot extension installed. I am using the Copilot Business plan ($19/month, but there is a free trial available).
The task
The Markdown Easy contrib module includes a very simple service class with no dependencies that implements a single method. Normally, Drupal service classes would require a kernel test (due to dependencies,) but in this case a simple unit test will do the job.
While using Drush's generate command has generally been my preferred method for scaffolding a test class, I found that using Copilot provides a much more advanced starting point. But, like anything else generated via AI, knowledge of the right way to perform a task is not optional. Code generated by AI might be correct, but blind confidence in what the AI provides will surely get you into trouble sooner, rather than later.
The getFilterWeights() method that we tested is a relatively simple method that returns an array of filter weights for three filters related to the configuration of the Markdown Easy text filter. The method takes a single parameter: an array of filters assigned to a text format. This method ensures that the configured order of filters related to Markdown Easy provides a secure experience.
Therefore, it makes sense that the unit test should pass in several sets of filters to the getFilterWeights() method and ensure that the returned array is correct - a fairly simple test that utilizes a data provider. To be honest, if I wasn't experimenting with using Copilot to generate tests, I probably wouldn't have this test, as it is almost trivial.
Regardless, adding test coverage to custom or contrib modules is a fantastic way of building modules where sustainability is a top priority. This is one of the reasons why writing PhpUnit tests is a prominent aspect of the Professional Module Development course that I teach.
Using the Copilot contextual menu - attempt 1
In this example, I placed my cursor at the end of the method name and then navigated the contextual menu to Copilot | Generate tests
By default, this will attempt to create the unit test class in markdown_easy/src/MarkdownUtilityTests.php, which is not the best-practice for location for tests in Drupal modules. Luckily, we can modify the location via the Create as option.
This results in a Save as dialog box in which we can specify the proper location for the test:
The test file is created in the proper place, but (at least in my test) it oddly didn't include an opening <?php tag, nor a namespace. Luckily, Copilot didn't make me work too hard on the namespace, as it correctly suggested the proper namespace via its autocomplete feature:
Once I manually added these (as well as declare(strict_types = 1);), the resulting test was:
<?php
declare(strict_types = 1);
namespace Drupal\Tests\markdown_easy\Unit;
use Drupal\markdown_easy\MarkdownUtility;
use Drupal\Tests\UnitTestCase;
/**
* Tests the MarkdownUtility class.
*
* @group markdown_easy
*/
class MarkdownUtilityTest extends UnitTestCase {
/**
* Tests the getFilterWeights method.
*/
public function testGetFilterWeights() {
$markdownUtility = new MarkdownUtility();
// Test case 1: Empty filters array.
$filters = [];
$expectedWeights = [];
$this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters));
// Test case 2: Non-empty filters array.
$filters = [
'filter1' => 10,
'filter2' => 5,
'filter3' => 15,
];
$expectedWeights = [
'filter1' => 10,
'filter2' => 5,
'filter3' => 15,
];
$this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters));
// Test case 3: Filters array with duplicate keys.
$filters = [
'filter1' => 10,
'filter2' => 5,
'filter1' => 15,
];
$expectedWeights = [
'filter1' => 15,
'filter2' => 5,
];
$this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters));
}
}As-is, this test does not pass, as Copilot missed the fact that the method will only return the weights of specific filters, so using filter1, filter2, etc.. doesn't help much...
Using the Copilot contextual menu - attempt 2
In this attempt, I highlighted the entire getFilterWeights() method and then selected Generate tests from the contextual menu. The result (without any changes on my part) was much better:
<?php
namespace Drupal\Tests\markdown_easy\Unit;
use Drupal\Tests\UnitTestCase;
use Drupal\markdown_easy\MarkdownUtility;
/**
* Unit Test coverage for MarkdownUtility.
*
* @coversDefaultClass \Drupal\markdown_easy\MarkdownUtility
*
* @group markdown_easy
*/
class MarkdownUtilityTest extends UnitTestCase {
/**
* Tests the getFilterWeights method.
*
* @covers ::getFilterWeights
*/
public function testGetFilterWeights(): void {
$markdownUtility = new MarkdownUtility();
// Create a sample array of filters.
$filters = [
'filter_html' => [
'status' => true,
'weight' => 10,
],
'filter_autop' => [
'status' => true,
'weight' => 20,
],
'markdown_easy' => [
'status' => true,
'weight' => 30,
],
];
// Call the getFilterWeights method.
$result = $markdownUtility->getFilterWeights($filters);
// Assert that the weights are set correctly.
$this->assertEquals(10, $result['filter_html']);
$this->assertEquals(20, $result['filter_autop']);
$this->assertEquals(30, $result['markdown_easy']);
}
}Not only did Copilot get the opening PHP tag and the namespace correct, but the quality of the test was much better, and passed without any changes. However; this test was generated with only a single set of test data, so I decided to try again…
Using the contextual menu - attempt 3
In this attempt, I again highlighted the entire getFilterWeights() method and then proceeded to select Generate tests again from the contextual menu. The result was much better (but again oddly missing the opening PHP tag):
namespace Drupal\Tests\markdown_easy\Unit;
use Drupal\Tests\UnitTestCase;
use Drupal\markdown_easy\MarkdownUtility;
/**
* Tests the MarkdownUtility class.
*
* @group markdown_easy
*/
class MarkdownUtilityTest extends UnitTestCase {
/**
* Tests the getFilterWeights method.
*/
public function testGetFilterWeights() {
$markdownUtility = new MarkdownUtility();
// Test case 1: All filters are disabled.
$filters = [
'filter_html' => ['status' => FALSE],
'filter_autop' => ['status' => FALSE],
'markdown_easy' => ['status' => FALSE],
];
$expectedWeights = [
'filter_html' => NULL,
'filter_autop' => NULL,
'markdown_easy' => NULL,
];
$this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters));
// Test case 2: Only filter_html is enabled.
$filters = [
'filter_html' => ['status' => TRUE, 'weight' => 10],
'filter_autop' => ['status' => FALSE],
'markdown_easy' => ['status' => FALSE],
];
$expectedWeights = [
'filter_html' => 10,
'filter_autop' => NULL,
'markdown_easy' => NULL,
];
$this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters));
// Test case 3: All filters are enabled with different weights.
$filters = [
'filter_html' => ['status' => TRUE, 'weight' => 10],
'filter_autop' => ['status' => TRUE, 'weight' => 20],
'markdown_easy' => ['status' => TRUE, 'weight' => 30],
];
$expectedWeights = [
'filter_html' => 10,
'filter_autop' => 20,
'markdown_easy' => 30,
];
$this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters));
}
}In this attempt, the test was created with three sets of data, and passed without any changes (other than adding the opening PHP tag). But, can we use Copilot to make it better? Ideally, the three sets of input data should be provided as part of a PhpUnit data provider method.
Refactoring the test method to use a PhpUnit data provider
To achieve this, I placed my cursor inside the test method, and selected Copilot | Start inline chat, and entered the prompt: "Can we refactor this method to use a data provider?" The result displayed the proposed result in a Visual Studio Code diff (similar to what is used when resolving Git code conflicts,) with the option for me to either Accept, Discard, or Regenerate (represented by a circle-arrow icon) the proposed solution. I accepted, and the resulting code took a little bit of massaging to make correct. While Copilot did properly create the data provider method and updated the test method, it left a bunch of the original test method code at the bottom of the class, outside of any method. Once I (easily) removed this artifact, the test passed as expected.
The resulting code was:
<?php
declare(strict_types = 1);
namespace Drupal\Tests\markdown_easy\Unit;
use Drupal\markdown_easy\MarkdownUtility;
use Drupal\Tests\UnitTestCase;
/**
* Tests the MarkdownUtility class.
*
* @group markdown_easy
*/
class MarkdownUtilityTest extends UnitTestCase {
/**
* Data provider for testing the getFilterWeights method.
*
* @return array
* An array of test cases.
*/
public function filterWeightsDataProvider(): array {
return [
[
[
'filter_html' => ['status' => FALSE],
'filter_autop' => ['status' => FALSE],
'markdown_easy' => ['status' => FALSE],
],
[
'filter_html' => NULL,
'filter_autop' => NULL,
'markdown_easy' => NULL,
],
],
[
[
'filter_html' => ['status' => TRUE, 'weight' => 10],
'filter_autop' => ['status' => FALSE],
'markdown_easy' => ['status' => FALSE],
],
[
'filter_html' => 10,
'filter_autop' => NULL,
'markdown_easy' => NULL,
],
],
[
[
'filter_html' => ['status' => TRUE, 'weight' => 10],
'filter_autop' => ['status' => TRUE, 'weight' => 20],
'markdown_easy' => ['status' => TRUE, 'weight' => 30],
],
[
'filter_html' => 10,
'filter_autop' => 20,
'markdown_easy' => 30,
],
],
];
}
/**
* Tests the getFilterWeights method.
*
* @dataProvider filterWeightsDataProvider
*/
public function testGetFilterWeights(array $filters, array $expectedWeights) {
$markdownUtility = new MarkdownUtility();
$this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters));
}
}So, that's not too bad for a few minutes of work! But, as a stickler for clean code, there was still some work ahead for me to get an acceptable PhpStan report.
Code quality changes
Overall, the quality of the code that was provided by Copilot was really good. But, this comes with the caveat that I utilize the PHP Sniffer & Beautifier Visual Studio Code extension (configured to use Drupal coding standards,) so it could be that code generated by Copilot is automatically formatted as it is generated (I really have no idea.). The bottom line is that I had zero phpcs issues in the code generated by Copilot.
For PhpStan, I normally try to achieve level 6 compliance - this can be especially tricky when it comes to "no value type specified in iterable type" issues. Without getting into the details of solving issues of this type, I decided to let Copilot have a go at updating the docblock for the filterWeightsDataProvider() method - and much to my surprise, it was able to provide a reasonable fix:
The process to update the docblock for the testGetFilterWeights() method wasn't as simple, as it was missing the parameter information, so I added that manually and then used Copilot in a similar manner to solve the PhpStan issue.
There was an additional, minor, PhpStan issue that I solved manually as well. With that, I had achieved zero PhpStan level 6 issues, a clean phpcs report, and a passing test! 🎉
This new test has been committed to the module.
Lesson learned
- Context matters (a lot) when generating tests (any code, really) using Copilot. In my (limited) experience, the files open in Visual Studio Code, where the cursor is, and what is highlighted makes a significant difference in the results provided.
Do not be afraid to use the Regenerate button (the circle-arrow icon) when generating any code, including tests. I have found that if I don't like the initial result, regenerating often results in a much better option the second or third time around.
- The Start Inline Chat option in the contextual menu is rapidly becoming my new best coding friend. Do not be afraid to experiment with it and use it more than you think you should. I find it very useful for making code more concise, suggesting documentation descriptions, and giving me a headstart in scaffolding entire methods.
- This should go without saying, but don't trust anything that is generated by Copilot. This is a tool that you should look at to save time, but not solve problems for you.
Header image generated using ChatGPT4.
Thanks to Andy Giles and Cindy Garcia for reviewing this blog post. Cindy is a graduate of both Drupal Career Online and Professional Module Development. Andy is the owner of Blue Oak Interactive.