This short text indicates how it is possible to use one of the grid5000
clusters for testing for performance regressions in a range of commits.

The idea is tu run a significant special-q range, because timing jitter
can be very large. It is also important to run the bench several times,
because jitter can take various forms (machine transiently busy during
bench, for instance).


I used one uniform grid5000 cluster, and ran several multi-nodes jobs,
all doing the same bench within one job. The submission command is:

fnancy ~/cado $ git rev-list --abbrev-commit master^..descent-new2 | (cd .. ; while read x ; do oarsub -p "cluster='griffon'" -l /nodes=15 "~/bench-many.sh  $x" ; done)

where ~/bench-many.sh is the following script:

----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----
#!/bin/bash

if [ "$1" = client ] ; then
    TMP=$(mktemp -d /tmp/cado-bench.XXXXXXXXXXXX)
    cd $TMP; c="$2"; h=`hostname --short` 
    (
        echo "Building $c on $h" >&2
        git clone ~/cado/.git . ; cp $HOME/cado/local.sh .
        git co "$c" ; make cmake ; make -j8 las
    ) >/dev/null 2>&1
    eval `make show` ; echo "Running $c on $h" >&2
    $build_tree/sieve/las   -poly $HOME/rsa704.poly -fb $HOME/rsa704.roots -I 15 -q0 1000000000 -q1 1000000800 > ~/cado/bench-$c-$h-`date +%s`.txt
    /bin/rm -rf $TMP ; exit 0
fi
for x in $(uniq $OAR_NODEFILE) ; do
    oarsh -q -n $x $0 client "$@" & :
done
wait
----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----

(Note the ``uniq'' call, which forces only one core per node to be used,
to avoid cores to perturbate eachother while competing for caches etc.)


Next, the timings obtained can be collected. The average stands very
little chance of being meaningful, but the median is. The following perl
script extracts all result files, and prints for each commit the median,
the extremal values, as well as the 1/4 and 3/4 values.


fnancy ~/cado $ for f in $(git rev-list --abbrev-commit master^..descent-new2 ) ; do echo -n "$f -> " ;  if [ "`ls bench-$f-*.txt 2>/dev/null`" != "" ] ; then tail -n 2  bench-$f-*.txt | grep time | perl -ne '/time ([\d\.]+)s/ && push @x, $1; END { @x = sort { $a <=> $b } @x ; $n=@x; $i=int($n/2); $j=int($n/4); $k=int(3*$n/4); print "$x[0] .. $x[$j] .. $x[$i] .. $x[$k] .. $x[$#x]\n"; }' ; else echo "nothing" ;  fi ; done

The output indicates here a significant regression at commit 178dfda
(expected, for that case), but nothing significant in the commit range
f06d293..178dfda. (One should also check that no relation is lost in the
bench, or course. This holds here.)

756d912 -> 1165.0 .. 1171.0 .. 1172.2 .. 1172.7 .. 1181.9   (20)
4f9ee94 -> 1167.9 .. 1170.9 .. 1171.6 .. 1173.2 .. 1181.1   (20)
d1b3ea4 -> 1165.6 .. 1166.7 .. 1167.9 .. 1169.1 .. 1174.3   (20)
1cc0739 -> 1165.5 .. 1166.5 .. 1168.0 .. 1172.6 .. 1180.4   (20)
1c739a0 -> 1164.8 .. 1166.7 .. 1167.3 .. 1169.1 .. 1172.4   (20)
c20cc32 -> 1164.8 .. 1166.6 .. 1167.1 .. 1168.6 .. 1181.8   (20)
178dfda -> 1149.9 .. 1154.4 .. 1156.4 .. 1157.4 .. 1165.1   (20)
9d1b524 -> 1108.9 .. 1111.1 .. 1111.8 .. 1114.9 .. 1120.0   (20)
2d7e915 -> 1107.3 .. 1109.5 .. 1110.5 .. 1113.1 .. 1116.3   (20)
86fbf13 -> 1114.3 .. 1116.1 .. 1117.2 .. 1119.3 .. 1122.7   (20)
6b21c42 -> 1114.9 .. 1116.9 .. 1118.1 .. 1119.8 .. 1127.3   (20)
f0f17d6 -> 1113.7 .. 1114.3 .. 1115.4 .. 1115.9 .. 1117.8   (20)
9893d40 -> 1109.2 .. 1111.5 .. 1111.8 .. 1112.6 .. 1115.2   (20)
472d519 -> 1055.2 .. 1055.8 .. 1112.9 .. 1114.6 .. 1117.5   (30)
051b5ec -> 1112.9 .. 1114.5 .. 1115.1 .. 1115.8 .. 1119.8   (20)
5a9df13 -> 1112.9 .. 1114.3 .. 1115.2 .. 1116.2 .. 1121.4   (20)
73b73fb -> 1112.8 .. 1114.7 .. 1115.4 .. 1116.3 .. 1120.0   (20)
29c14a1 -> 1109.9 .. 1110.6 .. 1111.4 .. 1112.3 .. 1116.4   (20)
6991a94 -> 1113.0 .. 1114.3 .. 1115.4 .. 1116.3 .. 1119.7   (20)
a60498f -> 1110.7 .. 1112.5 .. 1113.5 .. 1114.7 .. 1117.2   (20)
7bf38ea -> 1110.3 .. 1112.0 .. 1113.2 .. 1113.9 .. 1116.4   (20)
d07cb9e -> 1110.6 .. 1112.2 .. 1112.4 .. 1114.0 .. 1115.1   (20)
b04db37 -> 1109.9 .. 1112.1 .. 1112.9 .. 1114.5 .. 1116.4   (20)
ba9d394 -> 1111.2 .. 1112.3 .. 1112.8 .. 1113.4 .. 1115.0   (20)
0f477ea -> 1106.7 .. 1107.6 .. 1108.5 .. 1109.0 .. 1113.2   (20)
f06d293 -> 1110.1 .. 1111.5 .. 1111.8 .. 1113.2 .. 1118.5   (20)