Alcotezt: An Alcotest Compatibility Wrapper for Tezt
====================================================

Alcotezt is a compatibility wrapper for tests originally written in :ref:`Alcotest <alcotest_section>` (now deprecated), enabling running them with :doc:`Tezt <tezt>`.

Alcotezts (i.e. tests wrapped with Alcotezt) are programmed against an API that is compatible with
Alcotest. But, under the hood, they are registered with Tezt. This
means that they are executed in the same way as all Tezt tests are
(see :ref:`running-alcotezts` below for more details).

Alcotezts are typically declared through the ``tezt`` manifest
function. This will:

 - Create a *test library* that registers the tests in the modules
   given as argument to ``tezt``.
 - Create a *test executable* ``main.exe`` in the test folder. The
   test executable links with the test library and consists of a single call to
   Tezt's `Test.run
   <https://ocaml.org/p/tezt/latest/doc/Tezt/Test/index.html#val-run>`__.
 - Finally, link the test library with Tezt's main entrypoint at
   ``tezt/tests/main.exe`` so that it registers all Alcotezts.

.. _running-alcotezts:

Running Alcotezts
-----------------

For a given folder ``$TEST_DIR``, the Alcotezts contained therein can be invoked in three ways:

 1. By executing the ``main.exe`` runner binary generated by
    Manifest. Through ``dune``, this is done with ``dune exec
    $TEST_DIR/main.exe``.
    This will execute the tests in ``$TEST_DIR`` (but not in its subdirectories).
 2. Through Dune, by building the ``runtest`` alias in the test target
    folder. That is, executing ``dune build @$TEST_DIR/runtest``.
    This will execute the tests in ``$TEST_DIR`` and its subdirectories.
 3. By executing the full Tezt test suite, via the main entrypoint at
    ``tezt/tests/main.exe``. Through Dune, this is done with ``dune
    exec tezt/tests/main.exe``.

Execution through the test executable
+++++++++++++++++++++++++++++++++++++

The advantage of passing through the Manifest-generated runner is that
it only depends on the tests themselves and the tested modules (and
their recursive dependencies), and so is faster to
compile. Furthermore, you can give arguments to Tezt to control
execution, e.g. passing ``--list`` to list the tests or ``--verbose``
to see debug output.

For example, to run the tests of :src:`src/lib_clic/test`::

   dune exec src/lib_clic/test/main.exe

This will execute the Alcotezts in :src:`src/lib_clic/test`. To pass
arguments to Tezt, append ``-- <TEZT_ARGS>`` to the above command::

   dune exec src/lib_clic/test/main.exe -- --list

Execution through the Dune ``runtest`` alias
++++++++++++++++++++++++++++++++++++++++++++

The ``runtest`` alias can be used to execute all tests, including
Alcotezts, in a folder and its recursive sub-folders. For example, to
run all tests of protocol Alpha, run::

  dune build @src/proto_alpha/runtest

On the other hand, there is no convenient way to pass arguments to the
underlying tests with this method.

The Alcotezts can be dynamically disabled in the ``runtest`` alias by
setting the environment variable ``RUNTEZTALIAS`` to ``false``. For
instance, to run all the unit tests in protocol Alpha exception the
Alcotezts, you can run::

  RUNTEZTALIAS=false dune build @src/proto_alpha/runtest

Execution through the Tezt main entrypoint
++++++++++++++++++++++++++++++++++++++++++

Finally, Alcotezts can be executed through the Tezt main entrypoint
at ``tezt/tests/main.exe``. All Alcotezts are linked with this binary,
so that all system, integration and unit tests registered with either Tezt
or Alcotezt can be executed by invoking::

  dune exec tezt/tests/main.exe

This is used to run all tests in the CI in a unified
and load-balanced manner. This is less convenient for local use
though, as there is currently no way of executing the subset of
tests that corresponds to a specific package, file or folder. A
forthcoming release of Tezt will ameliorate this.

Notable Differences Between Alcotest and Alcotezt
-------------------------------------------------

Test naming
+++++++++++

First of all, some definitions:

 - In Alcotest, a "test suite" is a sequence of "tests" that are
   each composed of a sequence of "test cases".
 - In Tezt, there are only "tests", no "test suites".

With Alcotezt, each Alcotest "test case" becomes a Tezt "test". The
Alcotest suite, test and case name are used to form the title of
the Tezt test with following format: ``SUITE_NAME: TEST_NAME
(TEST_CASE_NAME)``.

Given an Alcotest consisting of a suite ``Suite A`` that contains a
test ``Test a`` that itself contains the test cases ``Test case a1``
and ``Test case a2``. It also contains a test ``Test b`` with the test
case ``Test case b1``:

.. code:: ocaml

  let () =
    Alcotest.run
      "Suite A"
      [
        ( "Test a",
          [
            ("Test case a1", `Quick, fun () -> ...);
            ("Test case a2", `Quick, fun () -> ...);
          ] );
        ( "Test b",
          [
            ("Test case b1", `Quick, fun () -> ...);
          ] );
      ]

Running it in with Alcotest produces::

  Testing `Suite A'.
  This run has ID `3F91T9S2'.
  
    [OK]          Test a          0   Test case a1.
    [OK]          Test a          1   Test case a2.
    [OK]          Test b          0   Test case b1.
  
  Full test results in `/home/tezos/_build/_tests/Suite A'.
  Test Successful in 0.000s. 2 tests run.

And running it with Alcotezt produces::

  [17:07:42.289] [SUCCESS] (1/2) Suite A: Test a (Test case a1)
  [17:07:42.289] [SUCCESS] (2/3) Suite A: Test a (Test case a2)
  [17:07:42.290] [SUCCESS] (3/3) Suite A: Test b (Test case b1)

Test Output
+++++++++++

Alcotezt redirects ``Format``'s output to Tezt's `Log.debug
<https://ocaml.org/p/tezt/latest/doc/Tezt_core/Log/index.html#val-debug>`__.
To see the debug output of an Alcotezt, pass the ``--verbose`` flag to
Tezt. See the section :ref:`running-alcotezts` above for more
information on how to pass flags to Tezt when executing Alcotezts.

There is no way to redirect the output of ``Printf``. Consequently,
the output of Alcotezts that call this module directly cannot be
hidden.

Integration with the ``runtest`` aliases
++++++++++++++++++++++++++++++++++++++++

Alcotezts are registered as a dependency on the ``runtest``
alias. However, they are not executed through this alias in
the CI. Instead, they run through the Tezt main runner to enable load
balancing.