Everything you need to know about Node.js's new test runner

As I’ve mentioned in another article, Node.js 18 came with lots of new features, including global availability of the fetch command and the start of adopting the node: prefix for importing system modules, which, by the way, we’ll need to use to discuss another addition: the native test runner.

What is a test runner#

Before I start, I want to give a brief introduction to what a test runner is and why it’s so necessary in practically any development environment.

Any code can be tested automatically, which means creating another portion of code (which ironically is not tested) that contains a call to the original function and stores the result of that call to be compared with a success or error output depending on the test case being tested.

The libraries for making assertions (testing if a result is what’s expected) are already native with Node.js’s assert module, so we could have a file like this:

1
const add = (a, b) => a + b
2
export { add }

And test this simple function using the assert module:

1
import { add } from './function.mjs'
2
import assert from 'node:assert'
3

4
let result = add(1, 2)
5
assert.equal(result, 3, 'add(1, 2) should return 3')
6

7
result = add(1, '2')
8
assert.equal(result, 3, 'add(1, "2") should not return 3')

To execute it’s as simple as node addTest.mjs, however what would happen if we had hundreds or thousands of tests? Would we keep executing the same file? Split it into several? How would we handle the growth and automation of the codebase?

And that’s where test runners come in. Their job is to orchestrate test executions so they’re as efficient as possible and, at the same time, informative. Providing data such as code coverage and internal errors.

Why a test runner?#

Tools like Mocha, Jest, Jasmine, and Ava are already very well-known in the market, having existed for, well, forever, so why would Node’s test runner make any difference? We already have great tools out there.

The answer is simple: standardization. One of the biggest problems, at least in my opinion, is that all these tools behave in different ways and have different APIs (if they didn’t, we wouldn’t have different tools), and this increasingly reduces the number of people who run automated tests in their code.

Not writing tests leads to a greater number of untested systems that are susceptible both to security failures (in the worst case) and to critical system failures, and many critical systems lack tests.

With native ecosystem tools instead of third-party tools, we lower the barrier to entry for developers writing tests natively and standardize the API so other tools can be interchangeable.

The node:test#

The test module is the solution to the problem I just mentioned, it’s available starting from Node.js version 18, although you need to install version 18.1.0 to run the tool successfully from the command line (don’t ask me why).

Despite being present in the LTS version, the state of the test API is still described as experimental, meaning the API has near-final compatibility with the rest of the system, but it’s possible that future versions will undergo some changes or even have commands removed, so it’s still not recommended for production environments.

Using node:test#

Starting with the import, we’ll already see a big difference. We need to import the module with the node: prefix. If the test module is not imported following the prefix, Node will try to load a local module called test.

At the moment only this module has this requirement, but I believe this is part of a long-term plan to convert all native modules to the node: prefix.

The most common lines will be:

1
import test from 'node:test'

The module will export a function called test (which we could very well call whatever we wanted, the most common being describe). The function has the following signature:

1
type Options = {
2
  concurrency: number,
3
  only: boolean,
4
  skip: boolean | string,
5
  todo: boolean | string
6
}
7

8
type test = (name: string, options?: Options | Function, fn: Function) => Promise<any>

• name: the name of the test, this is where you’ll describe what the test is testing
• options: An optional options object; if not passed, the second argument will be the test function to be executed
  • concurrency: The number of tests that can be executed at the same time within this scope; if not specified, subtests will inherit from the closest parent
  • only: If true, when the CLI is run in --only mode this test will be executed, otherwise it will be skipped
  • skip: By default it’s false; if true or a string, the test will be skipped (with the string being the reason)
  • todo: Same as skip but the test is marked as to-do, or to be done.
• fn: The function to be executed as a test; it’s only the third parameter if there’s an options object. It can be a synchronous or asynchronous function.

A test can have 3 types:

• Synchronous: a synchronous function that will fail the test if there’s a throw

1
test('teste síncrono passando', (context) => {
2
  // Não lança exceções, portanto o teste passa
3
  assert.strictEqual(1, 1);
4
});
5

6
test('teste síncrono falhando', (context) => {
7
  // Lança uma exceção e gera uma falha
8
  assert.strictEqual(1, 2);
9
});

• Asynchronous with Promises: An asynchronous function in the form of a Promise that will fail if the promise is rejected

1
test('assíncrono passando', async (context) => {
2
  // Sem exceções, a Promise resolve, sucesso!
3
  assert.strictEqual(1, 1);
4
});
5

6
test('assíncrono falhando', async (context) => {
7
  // Qualquer exceção faz a promise rejeitar, portanto: erro
8
  assert.strictEqual(1, 2);
9
});
10

11
test('falhando manualmente', (context) => {
12
  return new Promise((resolve, reject) => {
13
    setImmediate(() => {
14
      reject(new Error('podemos falhar a promise diretamente também'));
15
    });
16
  });
17
});

• Asynchronous with Callbacks: Same as above, but the test function receives a second callback parameter (usually called done) that, if executed with no parameters, makes the test succeed; otherwise the first parameter is the error.

1
test('callback passando', (context, done) => {
2
  // Done() é a função de callback, sem parâmetros, ela passa!
3
  setImmediate(done);
4
});
5

6
test('callback falhando', (context, done) => {
7
  // Done é invocado com um parâmetro de erro
8
  setImmediate(() => {
9
    done(new Error('Mensagem de erro do teste'));
10
  });
11
});

To get closer to what we already use today, as I mentioned earlier, we can call the test function as describe:

1
import describe from 'node:test'
2

3
describe('Meu teste aqui', (context) => {})

Subtests#

Like the most famous test frameworks, Node’s test runner also has the ability to do subtests.

By default the test function will accept a second parameter, as you should have noticed in the previous examples, which is a function that takes two parameters: a context and an optional callback called done.

The context object is of type TextContext and will have the following properties:

• context.diagnostic(message: string): You can use this function to write text output to the TAP protocol, which we’ll discuss later. Think of it as a debug output; instead of using console.log, you can use diagnostic to receive information at the end of the test report.
• context.runOnly(shouldRunOnlyTests: boolean): A programmatic way to run the test runner with the --test-only flag; if the function parameter is true, this context will only run tests that have the only option set. If you run Node with --test-only, this function is not executed.
• context.skip([message: string]) and context.todo([message: string]): The same as passing the skip and todo parameters to the function
• context.test([name][, options][, fn]): The same function recursively, so they can continue to be nested

To create a subtest, just call context.test inside a higher-level test:

1
test('top level', async (context) => {
2
  await context.test('subtest 1', (context) => {
3
      assert.strictEqual(1,1)
4
  })
5

6
  await context.test('subtest 2', (context) => {
7
      assert.strictEqual(1,1)
8
  })
9
})

It’s important to note that subtests must be asynchronous, otherwise the functions won’t be executed.

Skip, only, and todo#

Tests can receive special flags as parameters; currently there are 3 available flags:

• skip: the test will be skipped if the skip option is truthy (a string or any value). If it’s a string, as mentioned before, the message will be shown in the test output at the end:

1
// Skip without message
2
test('skip', { skip: true }, (t) => {
3
  // Nunca executado
4
});
5

6
// Skip with message
7
test('skip com mensagem', { skip: 'this is skipped' }, (t) => {
8
  // Nunca executado
9
});
10

11
test('skip()', (t) => {
12
  // Tente sempre retornar a chamada da função
13
  return t.skip();
14
});
15

16
test('skip() com mensagem', (t) => {
17
  // Tente sempre retornar a chamada de função
18
  return t.skip('this is skipped');
19
});

• only is a flag used when the test runner is run with the --test-only flag from the command line. When this flag is passed, only tests with the only property set to true will be executed. It’s a dynamic way to skip or run only specific tests.

1
// Vamos assumir que rodamos o comando node com a flag --test-only
2
test('esse vai ser executado', { only: true }, async (t) => {
3
  // Todos os subtestes dentro desse teste vão rodar
4
  await t.test('vai ser executado');
5

6
  // Podemos atualizar o contexto para parar de executar
7
  // No meio da função
8
  t.runOnly(true);
9
  await t.test('o subteste vai ser pulado');
10
  await t.test('esse vai ser executado', { only: true });
11

12
  // Voltando para o estado anterior
13
  // onde executamos todos os testes
14
  t.runOnly(false);
15
  await t.test('agora este também vai rodar');
16

17
  // Explicitamente não executando nenhum destes testes
18
  await t.test('skipped 3', { only: false });
19
  await t.test('skipped 4', { skip: true });
20
});
21

22
// A opção `only` não é setada então o teste não vai ser executado
23
test('não executado', () => {
24
  // Nunca vai rodar
25
  throw new Error('fail');
26
});

• todo is a simple message that will mark the test as “to do” instead of executing or skipping the test. It works exactly like all other flags and can also be set in the options object.

Running from the command line#

To run it, we can simply use the node command followed by the --test flag. To run specific files, just pass them to the command as the last parameter:

1
$ node --test arquivo.js outro.cjs outro.mjs diretorio/

If we don’t pass any parameters, the runner will follow these steps to determine which test files to execute:

1. Without passing any path, the cwd (current working directory) is searched recursively in the current directory using the following criteria:
   1. The directory is not node_modules (unless specified)
   2. If a directory called test is found, all files within this directory will be treated as test files
   3. For all other directories, any file with the extension .js, .cjs or .mjs is treated as a test if:
      • Named test following the regex ^test$, like test.js
      • Start with test- following the regex ^test-.+, like test-exemplo.cjs
      • Have .test, -test or _test at the end of their base names (without the extension), following the regex .+[\.\-\_]test$, such as exemplo.test.js or outro.test.mjs

Each test is executed in its own child process using child_process. If the process exits with code 0 (no error), it’s considered correct, otherwise it’s a failure.

The most interesting thing is that any type of test that emits TAP output can be run by Node’s test runner, even if it doesn’t use node:test internally.

Using TAP for more readable output#

The test runner uses a fairly well-known protocol called TAP (Test Anything Protocol). It’s great, but the output is extremely hard to read when run from the command line. Plus, the default output lacks some analyses like code coverage.

For this, there are packages like node-tap that parse this protocol to display output in a much more user-friendly way. To use it, just install it locally or globally:

1
$ npm i [-g] tap

Tap accepts any input from stdin, so we just need to pipe to it when running tests with node --test | tap, and we get much cleaner output for both errors:

And for successful tests:

Conclusion#

Node’s test runner will be one of the tools that can most impact code workflows in practically all applications, and this means it’s possible that other packages and systems will begin to use these premises to set the testing standard in all JavaScript environments.

Remember that the package documentation is available on the Node website!