Revision 0e375dc971e6f51fd34932d4a60784578ca55389 authored by Marge Bot on 07 February 2024, 15:09:29 UTC, committed by Marge Bot on 07 February 2024, 15:09:29 UTC
Co-authored-by: Gabriel Moise <gabriel.moise@trili.tech> Approved-by: vbot <vincent.botbol@nomadic-labs.com> Approved-by: Victor Allombert <victor.allombert@nomadic-labs.com> See merge request https://gitlab.com/tezos/tezos/-/merge_requests/11784
README.md
# Test suite for the Tezos blockchain using the Tezt framework
This directory contains a test suite for Octez (an implementation of the Tezos blockchain) based on Tezt. Tezt is a
generic test framework that is used for writing tests in OCaml.
The directory includes the sources of the Tezt framework, its extension with specific support for Tezos (called Tezt-Tezos), and a whole test suite.
## API Documentation
The user manual of the testsuite is available online at <https://tezos.gitlab.io/developer/tezt.html>.
The APIs of the `tezt` and `tezt-tezos` libraries are documented in the
interface files (suffix `mli`). Both APIs are also available online at
<https://tezos.gitlab.io/api/api-inline.html>.
## Overview
This directory contains the following subdirectories:
- `lib` defines an OCaml library named `tezt` which contains the test
framework. It is independent from Octez and from the Tezos protocols.
- `lib_tezos` defines another OCaml library named `tezt-tezos` which
is specific to Octez, complementing the above with Octez-specific
features.
- `lib_performance_regression` defines another OCaml library named
`tezt-performance-regression` which implements the so-called
Long Test Framework (LTF). It provides ways to register tests that are
to be run on dedicated "executors" with stable performances.
It features InfluxDB and Grafana integration for measures,
and Slack integration for alerts.
- `tests` defines an OCaml executable which gathers automatic tests
written for Tezos.
- `manual_tests` defines an OCaml executable which gathers manual
tests written for Tezos. A manual test is a test which requires
extra configuration and cannot be set up easily in the CI.
- `long_tests` defines an OCaml executable which gathers tests that
are run on the LTF (see `lib_performance_regression` above).
- `remote_tests` defines an OCaml executable with some Octez tests that serve
as examples for the ability of Tezt to run external executables through SSH.
- `vesting_contract_test` defines an OCaml executable which gathers
tests for the vesting contract. Those tests are too long to run
in the CI, the goal is to run them manually. (They could be moved to
`manual_tests`.)
- `self_tests` defines an OCaml executable with some tests that test
other libraries in this directory (in particular `tezt` itself).
- `snoop` defines an OCaml executable that performs benchmarks for gas.
- `_regressions` defines a set of output expected from a call to an
RPC or from some OCaml value serialized to JSON. If a file
is added into this repository, it should be also added in the
regression test which can be either `tests/RPC_test.ml` or
`tests/encoding.ml`.
- `records` contains records of the time taken by tests, and a script that updates them.
Records are used by the CI to distribute tests in jobs of roughly equal duration.
To update it, get a recent CI pipeline ID and run (from the root of the Tezos repository):
dune exec scripts/ci/update_records/update.exe -- -i --from PIPELINE_ID
git add tezt/records
git commit -m 'Tezt: update records'
## Implementation Details
Tezt is implemented in the [OCaml
language](https://ocaml.org). Currently, the project ensures that
libraries `tezt` and `tezt-tezos` do not depend on any library of
`Tezos`. The motto of the library is to provide an **extensive** but
**simple** test framework for Tezos.
An important feature of `tezt-tezos` is the possibility to catch `events`
emitted by the node. This allows fine-grained tests. In particular, as
much as possible, a test in this testsuite should avoid the use of timeouts
and instead, should rely on `events` if possible.
The implementation of the `tezt` library is detailed in its own documentation in the `lib/README.md` file.
## Declaring test ownership with `TESTOWNERS.json`
`TESTOWNERS.json` is a JSON file mapping 'products' to tests. It's
used to declare test ownership by attaching tests to products.
Through this file, developers and automated tooling can know what
product team is responsible for the maintenance of which test.
In pseudo-TypeScript, the schema of `TESTOWNERS.json` is:
```typescript
type TESTOWNERS = [PRODUCT_NAME: string]: PRODUCT_SPEC;
type PRODUCT_SPEC = {
tags?: string[];
path_patterns?: string[];
};
```
where the top-level element of `TESTOWNERS.json` has the type `TESTOWNERS`.
Semantically, each `PRODUCT_NAME` - `PRODUCT_SPEC` association in
`TESTOWNERS` corresponds to a product and a set of Tezt test tags and
path patterns to associated with the product. Path patterns are
interpreted as Perl-style regular expressions such that a pattern `P`
matches all tests with a `~__FILE__` that matches `P`. Tags match
all the tests with the given tag. The full set of tests associated to
a product is the union of all tests matched by any of the tags or the
path patterns associated to the test. Each of the fields `tags` and
`path_patterns` are optional, and their absence is interpreted as the
empty list.
Note that `path_patterns` should be acceptable by the `--match` option
or the `=~` TSL operator in Tezt. For instance, a valid
`TESTOWNERS.json` is:
```json
{
"layer1": {
"tags": ["michelson"],
"path_patterns": [
"^src/proto_.*/lib_protocol/"
]
}
}
```
This example defines a product called `layer1` and declares that all tezts
tagged `michelson` or which are registered with `~__FILE__` in
`src/proto_.*/lib_protocol/` (that is,
`/src/proto_alpha/lib_protocol` but also similar tests for any other
supported Tezos protocol) belong to the Layer 1 product.
In other words, this TEZTOWNERS entry declares the Layer 1 product to
be the owner of the union of the results returned by:
```
tezt 'michelson || file =~ "^src/proto_.*\/lib_protocol/"'
```
To see all the tests associated with a product, run:
```shell
scripts/tests_of_product.sh PRODUCT
```
To see the products associated with a set of tests, run:
```shell
scripts/tests_of_product.sh --reverse TSL_EXPRESSION
```
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...