Everything you need to know about Node.js's new test runner

Like I mentioned in this other article, Node.js 18 came packed with new stuff, among them the global availability of the fetch command and the start of adopting the node: prefix for importing system modules, which we’re going to need to talk about another addition: the system’s native test runner.

What’s a test runner#

Before I start, I want to give a quick introduction to what a test runner is and why it’s so necessary in pretty much any development environment.

Any code can be tested automatically, which means writing another chunk of code, ironically one that isn’t tested itself, that contains a call to the original function and stores the result of that call to be compared against an expected success or error output, depending on the case being tested.

The libraries for doing the assertion (testing whether a result is what you expect) already ship natively with Node.js’s assert module, so we could have a file like this:

1
const add = (a, b) => a + b
2
export { add }

And test that simple function using the assert module:

1
import { add } from './function.mjs'
2
import assert from 'node:assert'
3

4
let result = add(1, 2)
5
assert.equal(result, 3, 'add(1, 2) should return 3')
6

7
result = add(1, '2')
8
assert.equal(result, 3, 'add(1, "2") should not return 3')

Running it is as simple as node addTest.mjs. But what would happen if we had hundreds or thousands of tests? Would we keep running the same file? Split it into several? How would we handle growing and automating the test base?

And that’s where test runners come in. Their job is to orchestrate test runs so they’re as efficient as possible while also being informative, providing data like code coverage and internal errors.

Why a test runner?#

Tools like Mocha, Jest, Jasmine and Ava are already super well known in the market for existing since… well… since forever, so why would Node’s test runner make any difference? We already have great tools out there.

The answer is simple: standardization. One of the biggest problems, at least in my opinion, is that all these tools behave differently and have different APIs (otherwise we wouldn’t have different tools in the first place), and that keeps shrinking the number of people who run automated tests on their code.

Not writing tests leads to a larger amount of untested systems that are susceptible not only to security failures (in the worst case), but also to critical system failures, and a lot of critical systems don’t have tests.

With native ecosystem tools instead of third-party ones, we lower the barrier for devs to start writing tests natively, and we also standardize the API so other tools can be swapped in and out of each other.

node:test#

The test module is the solution to the problem I just mentioned. It’s been available since Node.js version 18, although you need version 18.1.0 to actually run the tool successfully from the command line (don’t ask me why).

Even though it’s present in the LTS version, the testing API’s status is still described as experimental, meaning the API is close to final compatibility with the rest of the system, but it’s possible future versions will change things around or even drop some commands, so it’s still not advisable for production environments.

Using node:test#

Starting with the import, we already see a big difference: we need to import the module with the node: prefix. If the test module isn’t imported with the prefix, Node will try to load a local module called test.

Right now only this module has that requirement, but I believe it’s part of a long-term plan to move every native module under the node: prefix.

The most common lines will be:

1
import test from 'node:test'

The module exports a function called test (which we could very well call whatever we want, the most common being describe). The function has the following signature:

1
type Options = {
2
  concurrency: number,
3
  only: boolean,
4
  skip: boolean | string,
5
  todo: boolean | string
6
}
7

8
type test = (name: string, options?: Options | Function, fn: Function) => Promise<any>

• name: the test’s name, this is where you’ll describe what the test is testing
• options: an optional options object. If it’s not passed, the second argument will be the test function to run
  • concurrency: the number of tests that can run at the same time within this scope. If not specified, subtests will inherit from the closest parent
  • only: if true, when the CLI runs in --only mode this test will run, otherwise it’ll be skipped
  • skip: defaults to false. If it’s true or a string, it skips the test (with the string being the reason)
  • todo: the same as skip, but the test is marked as a to-do instead
• fn: the function to run as the test, it’s only the third parameter if there’s an options object. It can be a sync or async function.

A test can be one of 3 types:

• Synchronous: a sync function that fails the test if it throws

1
test('sync test passing', (context) => {
2
  // No exceptions thrown, so the test passes
3
  assert.strictEqual(1, 1);
4
});
5

6
test('sync test failing', (context) => {
7
  // Throws an exception and produces a failure
8
  assert.strictEqual(1, 2);
9
});

• Async with Promises: an async function in the shape of a Promise that fails if the promise is rejected

1
test('async passing', async (context) => {
2
  // No exceptions, the Promise resolves, success!
3
  assert.strictEqual(1, 1);
4
});
5

6
test('async failing', async (context) => {
7
  // Any exception makes the promise reject, so: error
8
  assert.strictEqual(1, 2);
9
});
10

11
test('failing manually', (context) => {
12
  return new Promise((resolve, reject) => {
13
    setImmediate(() => {
14
      reject(new Error('we can reject the promise directly too'));
15
    });
16
  });
17
});

• Async with callbacks: the same as the previous one, but the test function gets a second callback parameter (usually called done) which, if called with no parameters, makes the test pass. Otherwise, the first parameter will be the error.

1
test('callback passing', (context, done) => {
2
  // Done() is the callback function, no parameters, it passes!
3
  setImmediate(done);
4
});
5

6
test('callback failing', (context, done) => {
7
  // Done is invoked with an error parameter
8
  setImmediate(() => {
9
    done(new Error('Test error message'));
10
  });
11
});

To keep things closer to what we already use today, like I mentioned at the start, we can call the test function as describe:

1
import describe from 'node:test'
2

3
describe('My test here', (context) => {})

Subtests#

Just like the most famous testing frameworks, the Node test runner also has the ability to run subtests.

By default the test function accepts a second parameter, as you probably noticed in the previous examples, which is a function that takes two parameters, a context and, if passed, a callback called done.

The context object is a class of type TextContext and has the following properties:

• context.diagnostic(message: string): you can use this function to write text output to the TAP protocol, which we’ll cover further ahead. Think of it as a debug output. Instead of a console.log, you can use diagnostic to get that information in the final test report.
• context.runOnly(shouldRunOnlyTests: boolean): a programmatic way of running the test runner with the --test-only flag. If the function’s parameter is true, this context will only run tests that have the only option set. If you run Node with --test-only, this function has no effect.
• context.skip([message: string]) and context.todo([message: string]): the same as passing the skip and todo parameters to the function
• context.test([name][, options][, fn]): it’s recursively the same function, so tests can keep being nested this way

To create a subtest, just call context.test inside a higher-level test:

1
test('top level', async (context) => {
2
  await context.test('subtest 1', (context) => {
3
      assert.strictEqual(1,1)
4
  })
5

6
  await context.test('subtest 2', (context) => {
7
      assert.strictEqual(1,1)
8
  })
9
})

It’s important to note that subtests need to be async, otherwise the functions won’t run.

Skip, only and todo#

Tests can receive special flags as parameters. Right now there are 3 flags:

• skip will be skipped if the skip option resolves to true, meaning a string or any other truthy value. If it’s a string, like I mentioned before, the message will show up in the test output at the end:

1
// Skip with no message
2
test('skip', { skip: true }, (t) => {
3
  // Never runs
4
});
5

6
// Skip with a message
7
test('skip with message', { skip: 'this is skipped' }, (t) => {
8
  // Never runs
9
});
10

11
test('skip()', (t) => {
12
  // Always try to return the function call
13
  return t.skip();
14
});
15

16
test('skip() with message', (t) => {
17
  // Always try to return the function call
18
  return t.skip('this is skipped');
19
});

• only is a flag used when the test runner runs with the --test-only flag on the command line. When that flag is passed, only tests with the only property set to true will run. It’s a pretty dynamic way to skip or run only specific tests.

1
// Let's assume we run the node command with the --test-only flag
2
test('this one will run', { only: true }, async (t) => {
3
  // All the subtests inside this test will run
4
  await t.test('will run');
5

6
  // We can update the context to stop running
7
  // in the middle of the function
8
  t.runOnly(true);
9
  await t.test('the subtest will be skipped');
10
  await t.test('this will run', { only: true });
11

12
  // Going back to the previous state
13
  // where we run every test
14
  t.runOnly(false);
15
  await t.test('now this one will run too');
16

17
  // Explicitly not running any of these
18
  await t.test('skipped 3', { only: false });
19
  await t.test('skipped 4', { skip: true });
20
});
21

22
// The `only` option isn't set so the test won't run
23
test('not executed', () => {
24
  // Will never run
25
  throw new Error('fail');
26
});

• todo is simply a message that marks the test as “to do” instead of running or skipping it. It works exactly like the other flags and can also be set in the options object.

Running from the command line#

To run it, we can simply run the node command followed by the --test flag. If we want to run specific files, we just pass them to the command as the last parameter:

1
$ node --test file.js other.cjs other.mjs directory/

If we don’t pass any parameters, the runner follows these steps to figure out which files are the test files to run:

1. With no path passed, the cwd, or working directory, will be the current directory, which gets searched recursively for the following:
   1. The directory is not node_modules (unless specified)
   2. If a directory called test is found, every file inside it is treated as a test file
   3. For every other directory, any file with the extension .js, .cjs or .mjs is treated as a test if:
      • It’s called test following the ^test$ regex, like test.js
      • It’s a file starting with test- following the ^test-.+ regex, like test-example.cjs
      • It’s a file with .test, -test or _test at the end of its basename (without the extension), following the .+[\.\-\_]test$ regex, like example.test.js or other.test.mjs

Each test runs in its own child process using child_process. If the process finishes with exit code 0 (no error), it’s counted as passing, otherwise it’s a failure.

The most interesting part is that any kind of test that emits TAP output can be run by Node’s test runner, even if it doesn’t use node:test internally.

Using TAP for a more readable output#

The test runner uses a pretty famous protocol called TAP (Test Anything Protocol). It’s great, but it’s extremely ugly and hard to read when you run it from the command line. On top of that, the default output doesn’t have some of the analysis you’d want, like code coverage.

For that, there are packages like node-tap, which parse that protocol to show a much friendlier output to the user. To use it, just install it locally or globally:

1
$ npm i [-g] tap

tap accepts any input from stdin, so we just pipe it in when running the tests with: node --test | tap, and then we get a much easier output both for errors:

And for successes:

Conclusion#

Node’s test runner is going to be one of the tools that can impact code flows the most across pretty much every application, which means it’s possible other packages and other systems will start using these assumptions to define the testing standard across every JavaScript environment.

And remember, the package’s documentation is already up on Node’s site!