![gallery](pics/gallery-3.jpg)

# Pothole-Mix Segmentation

Code repository for training neural networks on the Pothole-Mix dataset

## The dataset

[Pothole Mix](https://data.mendeley.com/datasets/kfth5g2xk3/2) is a dataset for the semantic segmentation of potholes and cracks on the road surface.

The main dataset is composed of 4340 (image,mask) pairs at different resolutions divided into training/validation/test sets with a proportion of 3340/496/504 images equal to 77/11/12 percent. This is the dataset used in the SHREC2022 competition and it's the dataset that allowed us to train the neural networks for semantic segmentation capable of obtaining the nice images and videos like this one.

<!-- blank line - ->
<figure class="video_container">
  <video controls="true" allowfullscreen="true" poster="pics/VID_20211031_162912.mp4-inference.jpg">
    <source src="http://deeplearning.ge.imati.cnr.it/genova-5G/video/VID_20211031_162912.mp4-inference.mp4" type="video/mp4">
  </video>
</figure>
< ! - - blank line -->

[![video session](pics/VID_20211031_162912.mp4-inference.jpg)](http://deeplearning.ge.imati.cnr.it/genova-5G/video/VID_20211031_162912.mp4-inference.mp4)

This repository serves to provide all the code that was used during the SHREC 2022 Competition, both as a baseline by the authors, and as an approach/solution by the participants to train their neural networks.

## Submissions

We received two submission and we also provide a baseline model+training notebook to start using the dataset. This notebook uses Fast.ai/Pytorch frameworks, you can have a look at it [here](https://htmlpreview.github.io/?https://gitlab.com/4ndr3aR/pothole-mix-segmentation/-/raw/main/shrec-22/evaluation/shrec-2022-evaluation.html?inline=true).

Submission 1 is pretty self-explanatory, you can just look at the notebooks and load the Pytorch pretrained models. Conversely, please check Submission 2 [README.md](shrec-22/submission-2/team2/README.md) for more info about their solutions and how to correctly install their framework[^1].

[^1]: Please also note that we had to do some minor modification in the inference module of Submission 2 to make overlay images (RGB+predicted mask) brighter. Code of the original submission is also provided for reference.

## Evaluation

To run the [evaluation notebook](https://htmlpreview.github.io/?https://gitlab.com/4ndr3aR/pothole-mix-segmentation/-/raw/main/shrec-22/evaluation/shrec-2022-evaluation.html?inline=true), clone this repo:

`git clone --recursive https://gitlab.com/4ndr3aR/pothole-mix-segmentation`

then:

`cd pothole-mix-segmentation ; git lfs pull`

to also download all the GIT-LFS objects. Then create a new virtualenv with:

`python3.8 -m venv /tmp/pothole-mix-segmentation-venv`

source the newly created virtual env with:

`source /tmp/pothole-mix-segmentation-venv/bin/activate`

install the `wheel` package:

`pip install wheel`

change directory to the root directory of this repo and then run:

`pip install -r requirements.txt # (the one in the root folder of this repo)`

When all the packages are installed, always in the root folder of this repo, run:

`jupyter notebook --port 53333 # (change the port if needed, a new tab should open in your browser showing the Jupyter Notebook main page. Use the --ip <addr> flag if you want to work remotely)`

Navigate to `shrec-22/evaluation` and click on `shrec-2022-evaluation.ipynb` (it will open a new tab with the running notebook)

Download/uncompress the [Pothole-Mix dataset](https://data.mendeley.com/datasets/kfth5g2xk3/2) and change the paths in the first cell according to your filesystem layout.

After that, you can "safely" press `Kernel->Restart & Run All` and wait for all the `do_validate(learn)` blocks to produce their output. The numbers that come out of these blocks are the metrics of the different models tested against validation and test sets. They should be totally identical to the ones of the [HTML evaluation notebook](https://htmlpreview.github.io/?https://gitlab.com/4ndr3aR/pothole-mix-segmentation/-/raw/main/shrec-22/evaluation/shrec-2022-evaluation.html?inline=true) (our original run) and they should match the ones in Table 1 and 2 in [the paper](https://arxiv.org/pdf/2205.13326.pdf).

Note: a pc with an Intel Core i9-9900K, 32 Gb of RAM and an Nvidia RTX 2070 with 8 Gb of VRAM has been used to produce the evaluation notebook. Lower amounts of CPU RAM or VRAM may cause the ipython kernel to throw OOM.

### Figure 6 script

To recreate Fig. 6 of the paper, it's sufficient to clone the repo (don't forget the --recursive flag and the subsequent `git lfs pull`!) and then run:

`./recreate-fig-6.sh`

The script is standalone and _should_ take care of building its own virtualenv, installing all the needed packages, invoking neural net inference scripts, composite the generated images and copy the result in the root folder of this repo. Good luck.

### Notes

This code has been tested under Ubuntu 20.04 and Python 3.8. The distribution shouldn't matter a lot, as long as a 3.8 virtual environment is created. As of today, the size of this repository is ~~6.5~~ 8.4 Gb on disk, another 2.6 Gb are required for running the evaluation notebook and other 2.6 Gb are required to run the "recreate-fig-6.sh" standalone script.

This repository uses Git LFS, if you haven't already, install it with:

`sudo apt install git-lfs`

Please also note that this repository has two submodules ([segmentation-inference](https://gitlab.com/4ndr3aR/segmentation-inference) and [test-set-videos](https://gitlab.com/4ndr3aR/shrec-2022-pothole-detection-test-set-videos.git)) so remember to clone the repository with `--recursive` param followed by a `git lfs pull`.

`git clone --recursive https://gitlab.com/4ndr3aR/pothole-mix-segmentation`

`cd pothole-mix-segmentation ; git lfs pull`