\documentclass[a4paper,11pt]{article}
\usepackage[english]{babel}
\usepackage{hyperref}{}
\usepackage{titling}
\usepackage{titlesec}
\usepackage{amssymb}
\usepackage{natbib}
\usepackage[margin=3cm]{geometry}
\usepackage{parskip}
\usepackage{calc} % Needed for Appendix table - rdv
\usepackage{xcolor}
\usepackage{enumerate}

\newcommand{\subtitle}[1]{%
  \posttitle{%
    \par\end{center}
    \begin{center}\large#1\end{center}
    \vskip0.5em}%
}
\renewcommand{\familydefault}{\sfdefault}


% customize section
\titleformat{\section}%
{\Large\bfseries}% format
{\llap{% label
    \thesection\hskip 9pt}}%
{0pt}% horizontal sep
{}% before

% customize subsection
\titleformat{\subsection}%
{\bfseries}% format
{\llap{% label
    \thesubsection\hskip 9pt}}%
{0pt}% horizontal sep
{}% before

\newenvironment{notice}{
\begin{center}
    \begin{tabular}[h!]{|p{0.8\textwidth}|}
    \hline
    {\bf IMPORTANT NOTICE!}\\\hline}
{   \\\hline
    \end{tabular}
\end{center}}

\newenvironment{TODO}{
\begin{center}
    \begin{tabular}[h!]{|p{0.8\textwidth}|}
    \hline
    {\bf TODO}\\\hline}
{   \\\hline
    \end{tabular}
\end{center}}

\newcommand{\criterion}[2]{\subsubsection*{\underline{#1 - #2}}\label{id:#1}}

\newcommand\CheckTable{%
  \begin{tabular}{ccccc}
    No & Minimal & Adequate & Good & Perfect \\
    0 & 1 & 2 & 3 & 4 \\
    \hline
    $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
  \end{tabular}%
}

\newcommand{\refcrit}[1]{%
 \framebox[1.1\width]{\hyperref[id:#1]{#1}}
}



\begin{document}

\title{Guidelines for Software Quality}
\subtitle{CLARIAH Task 54.100} 
\author{Maarten van Gompel \\ 
\emph{Centre for Language \& Speech Technology} \\ \emph{Radboud University, Nijmegen} \\  
Jauco Noordzij \\	
\emph{Huygens ING} \\
Reinier de Valk \& Andrea Scharnhorst \\ 
\emph{Data Archiving and Networked Services (DANS)} \\ \emph{Royal Netherlands Academy of Arts and Sciences}}


\maketitle

\begin{center}
version 1.1
\end{center}

%\begin{notice} 
%This is a \textbf{draft document}, everything in it is to be considered a
%\textbf{proposal only}! The guidelines have not been reviewed nor adopted yet!
%
%Please see also the \textbf{Request for Comment} notice at the end of this document.
%\end{notice}

\tableofcontents

\section{Motivation}

CLARIAH aims to deliver a digital research infrastructure made explicitly
accessible for researchers from the social sciences and humanities (SSH). This makes the development of
advanced ICT tools a core activity within CLARIAH. To be able to assess the
quality of the research infrastructure as a whole, we need to be able to assess
the quality of its individual software parts, and their function inside of the
research infrastructure including data components. If we can establish a common
set of software guidelines, we may more readily identify weaker components of
the software infrastructure and work on their improvement. We may also be able
to better direct software production processes towards issues of
interoperability and sustainability.  A digital research infrastructure
operates at the intersection of supporting ICT and research practices. For the SSH those are far from being homogeneous. The
practices served by the infrastructure differ concerning the object of
analysis, the processes of analysis, and the required usability level. 

The need for increased attention to software quality and sustainability in an
academic context is stressed by
\cite{RESEARCHSOFTWARE}.
They recognize the fundamental role software plays in modern research and
observe that good development practice is less followed in academia than in the
commercial sector, which leads to problems in maintenance and adoption. They
thus set out to promote good software development practice, good models of
sustainability and dissemination of the best software across application
areas/disciplines. We fully agree with these goals, and the guidelines
presented here are intended to respond, on a practical level, to these with
specific software quality criteria, explicitly encompassing sustainability as
well.

\subsection{Structure of the Document}\label{sec:struc}

After a discussion of the context of questions of software quality in social
sciences and humanities research infrastructures, we start the main part of
this document with the essentials: definitions and the state-of-art of
discussions around software quality criteria.  Following that, the
guidelines---a number of actionable steps, intended to make software more
usable and discoverable for users and other developers---are described. Two
categories of guidelines are discerned: 

\begin{itemize}
	\item developer guidelines
	\item full assessment criteria 
\end{itemize}

While the former are minimal requirements that software should adhere to, the
latter constitute more comprehensive guidelines that can help to achieve software of
a higher quality. The developer guidelines are specified in a form that leaves little
room for discussion. They are but one way of meeting the more generic
assessment criteria. It is encouraged to follow them as-is; doing so will make their
implementation easier. It will also make consuming the results easier, as there
will be a firm structure that can be relied on.  The full body of assessment
criteria does not just consist of boxes to tick, but offers ways to look at quality in software and tips to make software
better and more useful. 

The assessment criteria are presented as questions that can be answered on a
5-point scale to indicate the level of compliance. In addition to these
answers, there is room for open-ended comments here.  In order to explain, for
example, any trade-offs made while developing. 

It should be noted that throughout this document, both categories are captured
under the umbrella term \textit{guidelines}.  

\section{Context}

Software quality is an issue for software engineering. In the same way that
experimental research depends on the quality of its instruments and the
processes to use them properly, the quality of software influences the outcome
of any computational processes: its accuracy (it should deliver what it
was built for) as well as its efficiency (it should do this efficiently in
time and with respect to use of resources). At a first glance, software
quality concerns the quality of software code as a product (sometimes
called its \emph{functional quality}). However, software products do not exist in
isolation, but they often form part of an architecture. So the quality of software
concerns not only the level of source code, of a unit, but also its interplay
with technology and systems (called \emph{structural quality}).
\footnote{\url{https://en.wikipedia.org/wiki/Software\_quality}}

\textit{Software quality} is a term that emerged within software
engineering in the 1950's. The term \textit{software sustainability}, however, is far less widespread. One
could also say that with a more mature level of relying on software
technologies, the aspect of their sustainability becomes more prevalent. The
more software (and hardware) become seamless, invisible and move to the
background, the more important becomes their quality also for use in the long
term. The latter is one aspect of sustainability.

For infrastructures that aim to be reliable and supportive, the
reliability and sustainability of elements they are built of is a must.
% picture on how software and infrastructure-architecture are related
For research infrastructures, an additional element that enhances the
complexity emerges. Research practices by their very nature are changing in time. While
we might rely on consolidated methods and theories to some extent---at least
for the phases of normal science outside of the Kuhnian scientific revolutions---the research questions are supposed to change all the time, securely targeted
towards the \textit{unknown}. As a consequence, the application of methods and
theories---their recombination (to use an evolutionary term)---is at the heart of
research. For research based on the application of digital methods on digital
material this implies a kind of constant revolution in software tools---those
digital instruments.  For science history that is actually not new. If we look
into the history of instrumentation we will see outsourcing of %TODO: find reference
instrument-making to industries and re-appropriating them for lab research and
the tailor-made development of instruments both occurring at the same time.  For an
evaluation of software as part of a research infrastructure the
differentiation between tools which are \textit{explorative} and tools
which are \textit{stable and reproducible} is crucial. It makes sense to require
quality assessment for both, but not all levels of the
checklists presented in these guidelines might be of equal importance for all tools developed. To
differentiate between them is one add-on task in developing these guidelines. 

\section{Introduction - glossary of terms}

Assessing software (or data) quality is not a trivial matter. 

Whenever we refer to \emph{software}, we intend the term in a broad sense and
encompassing all of the following aspects:
\begin{itemize}
    \item source code
    \item binary executables
    \item user interfaces (including application programming interfaces (APIs) and web APIs)
    \item associated essential data
% * <reinierdevalk@gmail.com> 2016-03-30T15:01:35.204Z:
%
% > associated essential data
%
% What exactly would this be? In the light of a clear distinction between sustainability for data on the one hand and sustainability of software on the other, this might be confusing.
%
% ^.
% <proycon@anaproy.nl> 2016-04-06: Agreed, I kept this rather vague but we will indeed to
% need to work out a clear distinction. With ``essential'' data I was
% considering mandatory data without which the software would not work. Which
% can be simply stuff such as associated imagery and configuration files; but also more
% comprehensive models.
    \item documentation (including tutorials, screencasts)
    \item support infrastructure (version control, build systems, issue trackers)
\end{itemize}

The assesment criteria for software quality are grouped into various assessment
dimensions, such as ``Documentation'' and ``Installability''. Each dimension in turn
consists of assessment criteria, mostly formulated as a series of questions. This
makes them directly applicable as an instrument for software quality
assessment. 

% TODO-rdv: issue #5. consumer vs developer. Ik weet niet precies hoe ik dit moet integereren. Volgens de comment in issue 5 is de tekst geschreven vanuit een consumer perspectief, maar hieronder staat dat we ook ontwikkelaars etc. willen bedoenen. Ik heb nu even voor de simpelste oplossing gekozen door "and, primarily," toe te voegen aan de eerste zin...  
These guidelines target developers, managers, and, primarily, users of software.
Developers will be more aware of the targets to meet, and able to identify and
remedy weak areas. Managers and users will be able to assess whether software
is of sufficient quality for their purposes. Although we intend to
formulate the questions as plainly as possible, a certain degree of technical
expertise is demanded of all of these in order to attain a successful
assessment.

Each criterion can be answered on a 5-point scale to indicate compliance. In
addition, a criterion may also be deemed \emph{not applicable}. The response scale is as follows:

\begin{enumerate}
    \setcounter{enumi}{-1} %count from 0
    \item No -- \emph{The criterion is not met}
    \item Minimal -- \emph{The criterion is met only minimally. This is not a good score and better compliance is recommended}
    \item Adequate -- \emph{The criterion is met adequately. This is not a great score but it is sufficient}
    \item Good -- \emph{The criterion is complied to well}
    \item Perfect -- \emph{The criterion is complied to completely}
\end{enumerate}

Evaluators will need to determine for themselves what compliance level, per
dimension, constitutes an acceptable passing threshold.

The criteria we yield are  to a large extent adopted from the
\emph{Criteria-based Software Evaluation Guide} \citep{SSIGUIDE} by the Software
Sustainability Institute (SSI). Their work, in turn, is modelled after
\emph{ISO/IEC 9126-1 Software Engineering - Product Quality} \citep{ISO9126}.
% * <reinierdevalk@gmail.com> 2016-03-30T15:24:10.327Z:
%
% > ISO/IEC 9126-1 Software Engineering - Product Quality
%
% Because I was unfamiliar with this standard, I googled it - and then found that it has been revised by this one: http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=35733 . Maybe also good to mention this (more recent) report from the SSI: http://knowledge-exchange.info/event/software-sustainability 
%
% ^.
% <proycon@anaproy.nl> 2016-04-06: I came accross the newer standard as well,
% but the SSI guidelines document is still modelled after the older one.. The
% newer SSI report would indeed to good to mention and incorporate into our
% work.

The Software Sustainability Institute\citep{SSIGENERAL} is an academic institute
explicitly geared towards researchers and software developers in science. The
criteria they drafted are therefore deemed to form a very relevant basis for
research projects such as CLARIAH. Software quality assessment will have a
large generic component regardless of the context of the larger project. A
large portion of these guidelines is therefore applicable in the broader context
of software quality in academics, software quality for open source projects, and
software quality in general.

The dimensions used by the SSI are shown in Table~\ref{tab:criteria}.

\begin{table}[hbt]
\begin{tabular}{l|l}
\hline
\multicolumn{2}{c}{\textbf{Usability}} \\
\hline
Understandability & Is the software easily understood? \\ 
Documentation & Comprehensive well-structured documentation? \\
Buildability  & Straightforward to build from source on a supported system? \\
Installability  & Straightforward to install and deploy on a supported system? \\
Learnability & Easy/intuitive to learn how to use its functions? \\
\hline
\multicolumn{2}{c}{\textbf{Sustainability \& Manageability}} \\
\hline
Identity & Project/software identity is clear and unique? \\
Copyright & Easy to see who owns the project/software? \\
Licencing & Adoption of appropriate license? \\
Governance & Easy to understand how the project is run and the development managed? \\
Community & Evidence of current/future community? \\
Accessibility & Evidence of good facilities to obtain versions of the software? \\
Testability & Easy to verify if the software functions correctly? \\
Portability & Usable on multiple platforms? \\
Supportability & Evidence of current/future developer support? \\
Analysability & Easy to understand at the source-code level? \\
Changeability & Easy to modify and contribute changes? \\
Evolvability & Evidence of current/future development? \\
Interoperability & Interoperable with other required/related software? \\
\hline
\end{tabular}
\caption{Software quality dimensions according to the SSI \citep{SSIGUIDE} (with some minor paraphrasing).}
\label{tab:criteria}
\end{table}

We adopt this scheme with the following changes:

\begin{itemize}
\item Copyright \& Licensing are merged.
\item Evolvability was merged into Changeability.
\item Security \& Privacy dimension is added to Sustainability \& Manageability
\item Performance dimension was added to Usability.
\item Reusability dimension was added to Sustainability \& Manageability: To what extend is the software reusable?
% * <reinierdevalk@gmail.com> 2016-03-30T15:28:10.134Z:
%
% > Reusability
%
% What exactly is the difference between reusability and sustainability?
%
% ^.
% <proycon@anaproy.nl> 2016-04-06: As I see it, reusability is more a property
% of the way the software is set-up, with respect to reuse of its components,
% whilst sustainability is a broader concept.
\item The Interoperability dimension is split into a generic section and a
    CLARIAH-specific section. The latter provides criteria to assess
whether the software complies to requirements for integration into the CLARIAH infrastructure.
\end{itemize}

%The next sections
Sections \ref{sec:usab} and \ref{sec:sustAndMaint} lay out all dimensions with the individual assessment
criteria. The individual criteria are inspired by the SSI guidelines but
are not directly copied. Our criteria are generally more condensed and may
group issues that were expressed as multiple criteria in the SSI guide.
Various criteria have also been added that did not exist in the SSI guide,
and there were a few we found too specific to use.

Each criterion carries an ID code for easy reference. Cross-references will be
made whenever there is a relationship between criteria.

Before the description of the full assessment criteria, a number of strict,
minimal, requirements that software should adhere to (see Section
\ref{sec:struc}), are described in the now following Section \ref{sec:req}.

\section{Developer Guidelines: Minimal Requirements}\label{sec:req}

This section describes a set of guidelines for developers to adhere to, they
are very straightforward to implement, follow common practice where possible,
and take a firm stand where interoperability benefits when a choice must be
made. However, depending on the context, more (or less) may be expected from
peers or collaborators.

It should be noted that there are two sets of opposing scenarios: the first
concerns end user software versus experimental software, and the second
software receiving active support versus unsupported software.  

\begin{itemize}
	\item Not all software is intended exclusively for \textbf{end users}: some software is \textbf{experimental}, and intended only for people who can modify the code. This makes such software less usable---but when the goal of a project is to verify if something is at all possible, this might be a valid trade-off.
	\item While everyone wants the software they use to receive \textbf{active support}, in practice this is not always feasible. It is acceptable to release \textbf{unsupported} software---provided that users are notified clearly and in advance.
\end{itemize} 

Which of the following requirements apply depends on which configuration of
these two sets of opposing scenarios is applicable.  Below are written out all four configurations: in each case, non-applicable guidelines are grayed-out. 
The guidelines in these configurations may summarise one or more of the full assessment criteria, as indicated by the codes.

%BEGINREQUIREMENTS
\newcommand{\one}{The software must be stored in a \textbf{version control
    system} (VCS). \refcrit{AC1} \refcrit{CH3}
    The source code must be published. \refcrit{AC2}
To lower the barrier of entry the VCS must be either in a Git, Mercurial or SVN repository. 
As much as possible should be stored alongside the code 
in the VCS. CLARIAH is registered as an organization on
Github (\url{https://github.com/CLARIAH/}), it is recommended to host the VCS here. }
%TODO-rdv write an explanation what should go into the VCS and what shouldn't (docs: yes, issues-tracker: no, data: depends)
\newcommand{\two}{There must be \textbf{one command that installs all
dependencies} on the supported operating systems (OSs). \refcrit{IS2} }
\newcommand{\three}{The software must be \textbf{built with one command}. This
command must not write to the file system outside of the source code directory
and the temp directories (installation actions should be part of a separate
script). Of course, the code will not be buildable on all OSs, and the command
needs not to be platform independent---but there must be one OS on which it
works without further configuration necessary. The command must call the
multiple build steps if necessary. If there are tests, the command should run
them as part of the build. \refcrit{B1} \refcrit{IS9} \refcrit{TS4}
\refcrit{PB1} }
\newcommand{\four}{The software must either result in a directory with
artefacts that can be moved to an arbitrary location on the system and still
work, or it must contain \textbf{one command that will perform the
installation}. \refcrit{IS1}}
%^-- production
\newcommand{\five}{While a normal process can be easily daemonized, a daemonized process cannot 
be easily kept in the foreground. The software must be therefore able to \textbf{run in the foreground}
after installation.}
%^-- production
\newcommand{\six}{All \textbf{runtime dependencies} must be available after
installation. \refcrit{IS2}}
%^-- production
\newcommand{\seven}{If the software can be installed it must also have
\textbf{one uninstall script}. This script must provide a switch to also remove
config files and leave them alone by default. \refcrit{IS8}}
%^-- production
\newcommand{\eight}{The above requirements regarding installation can be omitted if the software can be run from the source directory and during running will not write outside of the source directory or temporary directories.}
%^-- experimental
\newcommand{\nine}{The root folder of the software project contains a file
    called \textbf{README} (fully capitalised) \refcrit{D1} \refcrit{IS7},
    which must be a readable plain text file. It may link to images, but the
    reader must be able to follow along without them. A \textbf{readable plain
    text} file is a file that can be read by a human using both wordpad.exe and
    vi. Formats such as markdown and ReST are readable as plain text, but LaTeX
    and HTML are not. The text should be encoded using UTF-8.} %TODO-rdv add examples of README and related files. Show how the install script requirements can easily be fulfilled with Docker.
	\newcommand{\nineA}{The readme must start by explicitly mentioning the
    experimental nature of the software.. \refcrit{U5}}
    %^-- experimental
	\newcommand{\nineB}{Accurate support contact information must be provided
    at the top of the readme, and an end date, until which this information can
be assumed to be valid, must also be added here. A suitable end date is the
current end time of the project. If no clear end date exists, it should be set
no further than a year away, and it should be reset regularly. \refcrit{SP1}
\refcrit{U6}}
    %^-- supported
	\newcommand{\nineC}{It must be mentioned explicitly that the software is
    unsupported. \refcrit{U6} }
    %^-- unsupported
	\newcommand{\nineD}{The readme must contain an explanation of the
    intended audience of the software, where the different user groups should
be clearly named and described. \refcrit{U2}}
	\newcommand{\nineE}{The readme must describe the problems that these
    users face, which resulted in the software. \refcrit{U4}}
	\newcommand{\nineF}{The readme must describe what the software does to
    help solving these problems. \refcrit{U1} \refcrit{U4} \refcrit{D6}
\refcrit{D7}}
	\newcommand{\nineG}{The readme must contain a list of similar or
    related software, and a description how the current software fits in.
\refcrit{U4}}
	\newcommand{\nineH}{The readme must specify the commands used to build
    and install from point 2-6 above. Also, it should specifiy the OSes the
commands will work on. Alternatively, this may be described in a separate
\textbf{INSTALL} file, but the README must then clearly state this.}
	\newcommand{\nineI}{The readme must contain either a quick start,
    explaining how the first task can be performed with the system, or, in the
    more extensive documentation, a link to the quick start. \refcrit{U3}
\refcrit{L1}}
    %^-- production
	\newcommand{\nineJ}{The readme must specify system requirements, i.e.\ observed
    memory/file/processing usage while doing the happy path and a moderate load
path. \refcrit{PF2} \refcrit{PF3} It should also clearly specify the platforms it is
designed to work on. \refcrit{IS9}}
    %^-- production
	\newcommand{\nineK}{The readme must contain a description of how the
    software scales (the load can be shared over multiple instances, explicitly
mentioning if this is not possible). \refcrit{PF5}}
    %^-- production
	\newcommand{\nineL}{The readme must contain a non-exhaustive list of
    places where the software is running. \refcrit{CM1}}
    %^-- production
	\newcommand{\nineM}{The readme must specify until when the project
    is currently funded, as well as a list of the funders. \refcrit{CP3}}
\newcommand{\ten}{Any \textbf{documentation must be stored alongside the code}.
Long-form texts should live in the folder \texttt{docs/} in the root of the
project. \refcrit{D11}}
\newcommand{\eleven}{All \textbf{documentation must be also stored in an
archivable format}. Such documentation can of course be generated from another
source format, but the archivable version must also be stored in the VCS. An
\textbf{archivable format} is a file format that will, with great probability,
still be readable in ten years' time. Such formats are readable plain text (see
above), HTML that does not depend on external stylesheets or scripts, and PDF/A. In the case of
PDF/A, the source format must also be included---though it may be a
non-archivable format such as a Word file, a LaTeX file, etc. \refcrit{D2}}
\newcommand{\twelve}{The project must have a \textbf{website}, containing, at the very least:	 \refcrit{ID2} }
\newcommand{\twelveA}{a typeset version of the readme}
\newcommand{\twelveB}{screenshots/screencasts of the application, if applicable \refcrit{D8}}
\newcommand{\twelveC}{a link to the source code, i.e. the version controlled
repository \refcrit{AC2}, as well as a link to where the latest stable release
can be obtained}
\newcommand{\twelveD}{contact information, or a link to the bug tracker \refcrit{SP2}}
%^-- supported
\newcommand{\twelveE}{a link to the documentation: either a direct link to the pdf or html, if available, or a browseable list of the text files}
\newcommand{\twelveF}{the website must be registered on Google, and a link to the project must
be provided to the CLARIAH Dissemination \& Education workpackage}
\newcommand{\twelveNB}{note that the readme as typeset by GitHub fulfils almost all these criteria.} 

\newcommand{\thirteen}{The \textbf{user interface of the application should
point to the website}. If this user interface is a website, where possible, a
link should be visible on every page. If the user interface is a command line
tool, a man page should ideally be provided that mentions the website. The help output
should mention it as well. \refcrit{L3}}
%^-- production
\newcommand{\fourteen}{The application's \textbf{user interface should point to the contact information} or to a place to report bugs.}
%^-- supported
\newcommand{\fifteen}{Almost every application provides an API. This
\textbf{API must be discoverable}. API documentation must be written as
specifically crafted comments in the source code itself, from which API documentation
can be automatically extracted using tools like
doxygen, javadoc or sphinx. The result must be published. Use descriptive names. What must be provided is a list of
function names, their parameter names, and the constraints placed on their
values. It would be helpful to know under what condition which function is
expected to be called. If the sofware provides a webservice, it's API should be
clearly documented as well, usually in a separate document. \refcrit{L4} \refcrit{AN3}}
%^-- production
\newcommand{\sixteen}{If the software depends on configuration files, then a
    template or sample must be provided in the repository for each file, \textbf{containing all
available options} and set to sane defaults. Comments in the configuration file
should give clear descriptions what the options do. \refcrit{L5}}
%^-- production
\newcommand{\seventeen}{The software must be \textbf{distributed under an OSI
approved licence} in a LICENSE file in root. \refcrit{CP1}. The copyright
holder should be clearly mentioned in the header of each source file \refcrit{CP2}} 
\newcommand{\eighteen}{The software must have a \textbf{CONTRIBUTING file}
    \refcrit{CH2}. } 
%^-- supported
% weggehaald omdat ik hem niet op een manier kan formuleren zodat hij non-trivially implementable is. \newcommand{\nineteen}{The CLARIAH project uses RDF as the format for the exchange of structured data. Structured data, such as a json object from a web-api must therefore\refcrit{IP1}}
\newcommand{\twenty}{The software must \textbf{not store usernames and
passwords} \refcrit{SC2}; instead, it must use the federated authentication and
group management facilities provided by CLARIAH.} 
\newcommand{\twentyOne}{Experimental software must be provided to end users in a way that makes it impossible for them to mistake the experimental nature. For example: they should be required to build it from source themselves, or it should refuse to store their data for a longer period of time..}
\newcommand{\twentyTwo}{This list of requirements must be \textbf{distributed
alongside the code}. If a software quality self-assessment is made using the
full criteria described in this document, then include these results alongside
the code as well. All of this can be achieved by simply filling our web-based
survey at \url{http://softwarequality.clariah.nl}, and committing the resulting markdown document to your VCS. }
%ENDREQUIREMENTS

\subsection{Configuration 1: Actively Supported End User Software} 

\begin{enumerate}
    \item \one 
    \item \two 
    \item \three
    \item \four
    \item \five
    \item \six
    \item \seven 
    \color{lightgray} \item \eight \color{black}
    \item \nine
    \begin{enumerate}
       \color{lightgray} \item \nineA \color{black}
        \item \nineB 
        \color{lightgray}\item \nineC \color{black}
        \item \nineD 
        \item \nineE 
        \item \nineF 
        \item \nineG 
        \item \nineH
        \item \nineI 
        \item \nineJ 
        \item \nineK 
        \item \nineL 
        \item \nineM
    \end{enumerate}
    \item \ten
    \item \eleven
    \item \twelve
	\begin{enumerate}
		\item \twelveA 
		\item \twelveB 
		\item \twelveC 
		\item \twelveD
		\item \twelveE 
		\item \twelveF
	\end{enumerate}
    \item \thirteen
    \item \fourteen
    \item \fifteen
    % weer toegevoegd-rdv --->
    \item \sixteen 
    \item \seventeen 
    \item \eighteen 
    %\item \nineteen 
    \item \twenty 
    \color{lightgray} \item \twentyOne \color{black} 
    \item \twentyTwo
    % <--- 
    
\end{enumerate}

\subsection{Configuration 2: Unsupported End User Software} 
\begin{enumerate} 
	\item \one 
	\item \two 
	\item \three 
	\item \four 
	\item \five 
	\item \six 
	\item \seven 
	\color{lightgray}\item \eight \color{black} 
	\item \nine 
	\begin{enumerate}
		\color{lightgray} \item \nineA  
		\item \nineB \color{black}
		\item \nineC  
		\item \nineD 
		\item \nineE 
		\item \nineF 
		\item \nineG 
		\item \nineH 
		\item \nineI 
		\item \nineJ 
		\item \nineK 
		\item \nineL 
		\item \nineM
	\end{enumerate} 
	\item \ten 
	\item \eleven 
	\item \twelve 
	\begin{enumerate}
		\item \twelveA 
		\item \twelveB 
		\item \twelveC 
		\color{lightgray} \item \twelveD \color{black} 
		\item \twelveE 
		\item \twelveF
	\end{enumerate}
	\twelveNB
	\item \thirteen 
	\color{lightgray} \item \fourteen \color{black}
	\item \fifteen 
	\item \sixteen 
	\item \seventeen 
	\color{lightgray} \item \eighteen \color{black} 
	%\item \nineteen 
	\item \twenty 
	\color{lightgray} \item \twentyOne \color{black} 
	\item \twentyTwo 
\end{enumerate}

\subsection{Configuration 3: Actively Supported Experimental Software}
\begin{enumerate} 
	\item \one 
	\item \two 
	\item \three 
	\color{lightgray}\item \four 
	\item \five 
	\item \six 
	\item \seven \color{black}  
	\item \eight 
	\item \nine 
	\begin{enumerate}
		\item \nineA 
		\item \nineB 
		\color{lightgray} \item \nineC \color{black} 
		\item \nineD 
		\item \nineE 
		\item \nineF 
		\item \nineG 
		\item \nineH 
		\color{lightgray} \item \nineI 
		\item \nineJ 
		\item \nineK 
		\item \nineL \color{black}  
		\item \nineM
	\end{enumerate} 
	\item \ten 
	\item \eleven 
	\color{lightgray} \item \twelve 
	\begin{enumerate}
		\item \twelveA 
		\item \twelveB 
		\item \twelveC 
		\item \twelveD 
		\item \twelveE 
		\item \twelveF
	\end{enumerate}
	\twelveNB
	\item \thirteen \color{black} 
	\item \fourteen
	\color{lightgray} \item \fifteen 
	\item \sixteen \color{black}  
	\item \seventeen 
	\item \eighteen 
	\color{lightgray} %\item \nineteen 
	\item \twenty \color{black} 
	\item \twentyOne 
	\item \twentyTwo 
\end{enumerate} 

\subsection{Configuration 4: Unsupported Experimental Software} 
\begin{enumerate} 
	\item \one 
	\item \two 
	\item \three 
	\color{lightgray}\item \four 
	\item \five 
	\item \six 
	\item \seven \color{black}  
	\item \eight 
	\item \nine 
	\begin{enumerate}
		\item \nineA 
		\color{lightgray} \item \nineB \color{black}
		\item \nineC 
		\item \nineD 
		\item \nineE 
		\item \nineF 
		\item \nineG 
		\item \nineH 
		\color{lightgray} \item \nineI 
		\item \nineJ 
		\item \nineK 
		\item \nineL \color{black}  
		\item \nineM
	\end{enumerate} 
	\item \ten 
	\item \eleven 
	\color{lightgray} \item \twelve 
	\begin{enumerate}
		\item \twelveA 
		\item \twelveB 
		\item \twelveC 
		\item \twelveD 
		\item \twelveE 
		\item \twelveF
	\end{enumerate}
	\twelveNB
	\item \thirteen
	\item \fourteen
	\item \fifteen 
	\item \sixteen \color{black}  
	\item \seventeen 
	\color{lightgray} \item \eighteen 
	%\item \nineteen 
	\item \twenty \color{black} 
	\item \twentyOne 
	\item \twentyTwo 
\end{enumerate} 

\section{Quality Assessment Criteria - Usability}\label{sec:usab}

%BEGINCRITERIA

\subsection{Understandability}\label{sec:und}

\newcommand{\uOneID}{U1}
\newcommand{\uOneText}{Is it clear what the software does?}
\criterion{\uOneID}{\uOneText}

Software must be accompanied be a clear and concise high-level description,
describing what exactly it does. Both the \texttt{README} file that ships with
the software (see \refcrit{IS7}) as well as the project website (see
\refcrit{ID2}) should contain
this information. 

%\CheckTable

\newcommand{\uTwoID}{U2}
\newcommand{\uTwoText}{Is it clear for whom the software is intended?}
\criterion{\uTwoID}{\uTwoText}

It should be clear who are the intended users for the software. Software is
usually not appropriate for all audiences. Gearing software at multiple
audiences however, through for instance offering multiple interfaces (command
line interface (CLI) , graphical user interface (GUI), web-user interface
(WUI), web service) is good practice. References to projects already using the
software are recommended (see also \refcrit{CM1}).

%\CheckTable

\newcommand{\uThreeID}{U3}
\newcommand{\uThreeText}{Is it clear how the software works?}
\criterion{\uThreeID}{\uThreeText}

There should be a high-level description explaining how the software
accomplishes its task. Links to publications are recommended. Also, a schema
offering an architectural overview is suggested where appropriate. 

%\CheckTable

\newcommand{\uFourID}{U4}
\newcommand{\uFourText}{Is the software motivated?}
\criterion{\uFourID}{\uFourText}

There should be a written motivation for why the software does things the way
it does and why it was designed in the first place. It should be clear what
problems are solved by it. Links to publications and comparisons to similar
software are strongly recommended.

\newcommand{\uFiveID}{U5}
\newcommand{\uFiveText}{Is the development status of the software clear?}
\criterion{\uFiveID}{\uFiveText}

It should be clear in what stage of development the software is. Is it ready for
production use or is it experimental software and in a alpha or beta stage? If
the software is not ready for production use, the README/documentation should clearly state
so.

\newcommand{\uSixID}{U6}
\newcommand{\uSixText}{Is the support status of the software clear?}
\criterion{\uSixID}{\uSixText}

It should be clear whether the software is actively supported, and if so until
when. Actively supported software should have clear support structures in
place (\refcrit{SP1}, \refcrit{SP2}).

Unsupported supported should be clearly marked as such in the
README/documentation. If the software likely ceases to be supported after project
funding ends, this should be indicated as well.

%\CheckTable

\subsection{Documentation}\label{sec:doc}

When we refer to documentation, we refer to the set of all documentation
available for the software. This may consists of different types of
documentation for different audiences, and may include published papers.

\newcommand{\dOneID}{D1}
\newcommand{\dOneText}{Is the software documented?}
\criterion{\dOneID}{\dOneText}

All software should be properly documented. Software without any documentation
is as good as useless. At the very least, there should be some documentation at
a minimum level, targeted at the intended audience. A README (see \refcrit{IS7}) that
attempts to meet some of guidelines in this section (such as
\refcrit{D6},\refcrit{D8},\refcrit{D9}) can be
considered a bare minimum level of documentation.

%\CheckTable

% * <reinierdevalk@gmail.com> 2016-03-30T15:35:50.608Z:
%
% > documentation at a minimum level
%
% What would this minimum level be? Source code level?
%
% ^.
% <proycon@anaproy.nl> 2016-04-06: No, I intended to mean a minimum level
% targetted for the appropriate audience. I added to the text 

\newcommand{\dTwoID}{D2}
\newcommand{\dTwoText}{Is the documentation accessible?}
\criterion{\dTwoID}{\dTwoText}

Documentation must be publicly accessible and in an acceptable standard format
such as HTML or PDF.

%\CheckTable

\newcommand{\dThreeID}{D3}
\newcommand{\dThreeText}{Is the documentation clear?}
\criterion{\dThreeID}{\dThreeText}

Documentation should be written in clear language, use proper spelling, and
clearly describe the software. Step-by-step and task-oriented instructions are
recommended.

See also \refcrit{D7}.

%\CheckTable

\newcommand{\dFourID}{D4}
\newcommand{\dFourText}{Is the documentation complete?}
\criterion{\dFourID}{\dFourText}

Documentation should cover the entire software, including advanced features.
Features that are not documented in any way are generally useless. If the
software consists of multiple tools, are all documented? Are there no tasks the
software can perform that are not properly explained?

% * <reinierdevalk@gmail.com> 2016-03-30T15:37:41.996Z:
%
% > cover the entire software
%
% From my own experience I know that you can get carried away here. Is there some objective way of determining this? Also: does D4 not conflict with D6? 
%
% ^.
% <proycon@anaproy.nl> 2016-04-06: I think it is indeed pretty hard to objectively
% measure, but I do think that features that are not documented in any way, are
% generally useless. (added that to the text now)
% Regarding D4 vs D6: No, I mean that the documentation should at
% least also offer a high-level view before it goes into the finer details, not that it
% should go beyond a high-level view (it should). I rephrased D6.

%\CheckTable

\newcommand{\dFiveID}{D5}
\newcommand{\dFiveText}{Is the documentation accurate?}
\criterion{\dFiveID}{\dFiveText}

Documentation should describe the advertised version and not be out-of-date
with the latest release. Examples should be in line with how the tool looks and
behaves.

%\CheckTable

\newcommand{\dSixID}{D6}
\newcommand{\dSixText}{Does the documentation provide a high-level overview of the software?}
\criterion{\dSixID}{\dSixText}

A high-level overview of the software should be an integral part of the
documentation, alongside more detailed instruction where appropriate.
Documentation should not immediately dive into the details. It is important to
first provide users with the necessary high-level insights so they can understand how
these details form a part of the larger picture.

% <proycon@anaproy.nl> 2016-04-06: I rephrased this after Reinier's comment for D4.

%\CheckTable

\newcommand{\dSevenID}{D7}
\newcommand{\dSevenText}{Are all the necessary audiences addressed, at their appropriate levels?}
\criterion{\dSevenID}{\dSevenText}

Different groups of users require different documentation. Developers require
APIs if the software is a library (see \refcrit{L4}), end-users require a walkthrough of the GUI
if the software has one. A different level of expertise may be expected of
different user groups, the documentation should assume the appropriate level.

%\CheckTable

\newcommand{\dEightID}{D8}
\newcommand{\dEightText}{Does the documentation make use of adequate examples?}
\criterion{\dEightID}{\dEightText}

Documentation should contain examples appropriate for the interface that is
described. Command-line interfaces should see examples of invocation and input
and output. Graphical user interfaces should be illustrated through screenshots or
screencasts. API references should contain source code examples of usage.

%\CheckTable

\newcommand{\dNineID}{D9}
\newcommand{\dNineText}{Is there troubleshooting information?}
\criterion{\dNineID}{\dNineText}

Documentation should include information on troubleshooting, i.e.\ a
specification of possible error messages and explanation for resolution. A
frequently asked questions (FAQ) section is appropriate to cover questions
that are repeatedly asked by the user base.

%\CheckTable

\newcommand{\dTenID}{D10}
\newcommand{\dTenText}{Is the documentation available from the project website?}
\criterion{\dTenID}{\dTenText}

Documentation must be clearly linked from the project website.

%\CheckTable

\newcommand{\dElevenID}{D11}
\newcommand{\dElevenText}{Is the documentation under version control?}
\criterion{\dElevenID}{\dElevenText}

The sources for documentation must be under version control like the source
code, preferably alongside the code itself.

\newcommand{\dTwelveID}{D12}
\newcommand{\dTwelveText}{Does the documentation describe the latest version?}
\criterion{\dTwelveID}{\dTwelveText}

Documentation should be up to date and describe the latest version of the
software, rather than lag several version behind.

To facilitate this in case of API documentation or other documentation that can
be automatically generated from sources; consider using services such as
\url{http://readthedocs.io} that can automatically build and serve
documentation upon each commit to for example github.

%\CheckTable

\subsection{Learnability}\label{sec:lea}

This category partially overlaps with the documentation criterion, but explicitly
focusses on how straightforward it is to learn to use the software.

\newcommand{\lOneID}{L1}
\newcommand{\lOneText}{Is there a ``Getting started'' guide?}
\criterion{\lOneID}{\lOneText}

A \textit{Getting started} guide, or similar, outlines how to quickly get
started with the software using a basic yet practical example and is usually task-oriented.
It helps people to quickly get their hands dirty prior to diving in the
remainder of the documentation. 

%\CheckTable

\newcommand{\lTwoID}{L2}
\newcommand{\lTwoText}{Are there instructions for basic use cases?}
\criterion{\lTwoID}{\lTwoText}

Instructions should be provided for at least basic use cases, and ideally for
all possible use cases if this is feasible.

% * <reinierdevalk@gmail.com> 2016-03-30T15:43:20.362Z:
%
% > for
% > all possible use cases
%
% I'm not sure whether this is always possible...
%
% ^.
% <proycon@anaproy.nl> 2016-04-06: Yeah, I don't think so either.. I used the
% adverb ``ideally'' for that reason, I added another qualifier ``if this is
% feasible''.

%\CheckTable

\newcommand{\lThreeID}{L3}
\newcommand{\lThreeText}{Does the interface provide a help reference?}
\criterion{\lThreeID}{\lThreeText}

Help options should be provided by the interface. Command line interfaces
\emph{must} have a \texttt{-h}/\texttt{--help} pair describing usage and all
options, and ideally also a man page. GUIs should use tooltips/hints to clarify
their widgets (or through whatever convention is customary for the
platform/ecosystem). Alternatively, they can provide a help option referring to
the documentation.

The project website should be mentioned from the help reference.

%\CheckTable

\newcommand{\lFourID}{L4}
\newcommand{\lFourText}{Is there API documentation for developers?}
\criterion{\lFourID}{\lFourText}

If the software is a programming library, API documentation must be provided.
If the software is a web service, a specification of the web API must be provided. 

API documentation should be auto-generated from comments in the source code,
see \refcrit{AN3}.

%\CheckTable

\newcommand{\lFiveID}{L5}
\newcommand{\lFiveText}{If the software is configurable, are the configuration options clearly explained?}
\criterion{\lFiveID}{\lFiveText}

% * <reinierdevalk@gmail.com> 2016-03-30T15:46:17.721Z:
%
% No further explanation of L5?
%
% ^.
%<proycon@anaproy.nl> 2016-04-06:  Added one now

If the software is configurable, through for instance an external configuration
files, a preferences window in a GUI, or by any other means, then the
configuration options and their effect should be clearly documented.

Moreover, default values for configuration parameters, and their effect, should be
made clear.

\subsection{Buildability}\label{sec:bui}

Buildability applies to all software written in languages that compile to
either native machine code or any intermediate byte code to be interpreted by a VM.
This is contrasted to software that is interpreted at run-time from source
code. This section is therefore applicable only to languages such as C, C++,
Java, Pascal, Haskell, Scala, Rust, Cython but not to scripting languages such
as Python, Ruby, Perl.  Note that buildability does not include
installation, packaging, or deployment.

%\CheckTable

\newcommand{\bOneID}{B1}
\newcommand{\bOneText}{Are there good instructions for building/compiling the software?}
\criterion{\bOneID}{\bOneText}

Build/compilation instruction should be available and clear enough. They should
be distributed alongside the software's source distribution (as part of an
\texttt{INSTALL} file or \texttt{README}), and/or be addressed in the documentation. If the
source distribution is the primary means of distribution, then build
instructions should be prominently displayed on the project website as well.

%\CheckTable

\newcommand{\bTwoID}{B2}
\newcommand{\bTwoText}{Is an established automated build system used?}
\criterion{\bTwoID}{\bTwoText}

Established build systems should be used. For example the GNU Build
System\footnote{Also known as the autotools.} or CMake for C/C++ (this is
preferred over a static Makefile); or Ant or Maven for Java. 

Solutions that are not tied to a single IDE are always preferred.

%\CheckTable

\newcommand{\bThreeID}{B3}
\newcommand{\bThreeText}{Are the dependencies listed and available?}
\criterion{\bThreeID}{\bThreeText}
(related to \refcrit{IS2})

All required or optional dependencies necessary to build the software should be
listed, including those by third parties (with references to their websites).
If the build system supports automatically obtaining dependencies, then this is
a preferred solution. If the platform has a package manager that can install
these dependencies, then add instructions (i.e. package names) to accomplish
this. Ideally all dependencies should be installable by one command.

Moreover, all listed dependencies should be available, unobtainable software
can not be used as a dependency. Higher-order dependencies need not be listed.

%\CheckTable

\newcommand{\bFourID}{B4}
\newcommand{\bFourText}{Are there tests to verify the build has succeeded?}
\criterion{\bFourID}{\bFourText}

A build process should terminate with a testing stage that verifies its success.

%\CheckTable

\subsection{Installability}\label{sec:ins}

Installability concerns the deployment of software on the target platform, also
including configuration of the software to the user's needs.

\newcommand{\isOneID}{IS1}
\newcommand{\isOneText}{Are there easily accessible installation instructions?}
\criterion{\isOneID}{\isOneText}

All software must come with installation instructions. Those instructions should
be easily accessible and presented on the project website as well as shipped
with the software in the README or INSTALL file (see also \refcrit{IS7}). Build and installation
instructions can be combined if the software is published as a source
distribution.

%\CheckTable

\newcommand{\isTwoID}{IS2}
\newcommand{\isTwoText}{Are the dependencies listed and available?}
\criterion{\isTwoID}{\isTwoText}
(related to \refcrit{B3})

All required or optional dependencies should be listed, including those by
third parties (with references to their websites). If the installation
procedure supports automatically obtaining dependencies, then this is a
preferred solution (see also \refcrit{IS3}). If the platform has a package manager that
can install these dependencies, then add instructions (i.e. package names) to
accomplish this. Ideally all dependencies should be installable with one
command.

Moreover, all listed dependencies should be available, unobtainable software
can not be used as a dependency. Higher-order dependencies need not be listed.
% * <reinierdevalk@gmail.com> 2016-03-30T15:54:41.426Z:
%
% This whole paragraph is almost a verbatim repetition of B3.  Maybe just cross-refer?
%
% ^.
%<proycon@anaproy.nl> 2016-04-06: Technically, build and installation
%dependencies may differ (though there will be substantial overlap).
%The text is much the same indeed though, cross-referencing may be a solution
%here, not sure what would be best.

%\CheckTable

\newcommand{\isThreeID}{IS3}
\newcommand{\isThreeText}{Are programming language's best installation practices followed?}
\criterion{\isThreeID}{\isThreeText}
%\criterion{IS3 -- Does the installation procedure comply to best practices in the ecosystem around the programming language?}

The ecosystem surrounding programming languages may come with standard
procedures for installation. Software should comply to these rather than use
ad-hoc mechanisms. If the ecosystem has facilities for dependency management
(automatic download of dependencies), these should be used.

\begin{itemize}
 \item Python software should have a \texttt{setup.py} based on Distutils, Setuptools, or their relatives.
 \item Installation of C/C++ software is generally an extension of the build process, often culminating in a \texttt{make install}.
 \item Java software is delivered as a \texttt{jar} file.
 \item Perl software should use systems such as \texttt{Module::Build}, \texttt{ExtUtils::MakeMaker}, or \texttt{Module::Install}. Resulting in a \texttt{Build.PL} or \texttt{Makefile.PL}.
 %TODO: add more languages (R?)
\end{itemize}

Complying to these installation practices often means your installable package
is fit for inclusion into the programming language's public package repository
(see \refcrit{AC5}).

%\CheckTable

\newcommand{\isFourID}{IS4}
\newcommand{\isFourText}{Is the software packaged according to standards for the target platform?}
\criterion{\isFourID}{\isFourText}

If the software has a well-defined target platform and is intended for adoption
by a wider audience, it is recommended to package it in the appropriate form.
This may however conflict with IS3, be rendered obsolete by \refcrit{IS3}, or
render \refcrit{IS3}
obsolete. The decision whether to use packaging methods for the target platform
and/or packaging methods for the ecosystem surrounding the programming language
(\refcrit{IS3}) should be
made on an individual basis, considering the nature and audience of the
software.

\begin{itemize}
    \item Linux/BSD/Unix -- Distributions generally have their own package manager
    \begin{itemize}
        \item Arch Linux: Use the Arch User Repository (AUR)
        \item Debian/Ubuntu (and other derivates): Use Debian Packages (deb)
        \item RedHat/CentOS/Fedora (and other derivates): Use Red Hat Packages (rpm)
    \end{itemize}
    \item Mac OS X: Use PKG or DMG for traditional Mac applications intended for end-users; for the more Unix-style software, use a system such as Homebrew or MacPorts
    \item Android: Use Android Application Packages (APK)
    \item iOS: (TODO) 
    \item Windows: (TODO) 
\end{itemize}

It is not always time- and cost-effective to package the software, especially not
when there are multiple target platforms. 

%\CheckTable

\newcommand{\isFiveID}{IS5}
\newcommand{\isFiveText}{Is the software package properly structured?}
\criterion{\isFiveID}{\isFiveText}

The contents of the package should be properly organised in sub-directories
(for documentation, headers, source, etc..). Conventions may differ between
programming languages.

If software is distributed as a plain archive (tar.gz, tar.bz2, zip or otherwise),
it must create a single directory when unpacked rather than spread its contents
over the current working directory.

%\CheckTable

\newcommand{\isSixID}{IS6}
\newcommand{\isSixText}{Is the software properly structured when installed?}
\criterion{\isSixID}{\isSixText}

When the software is installed, it should adhere to file placement standards set
by the target platform. On Unix-like operating systems (excluding Mac OS X), the Filesystem
Hierarchy Standard\footnote{\url{http://refspecs.linuxfoundation.org/fhs.shtml}} (FHS) should be followed.

%\CheckTable

\newcommand{\isSevenID}{IS7}
\newcommand{\isSevenText}{The software must include a README.}
\criterion{\isSevenID}{\isSevenText}

The \texttt{README} file must contain links to the project website, as well as
license and copyright information. The README should be either plain-text or
preferably in an unobtrusive mark-up format such as
Markdown\footnote{\url{http://daringfireball.net/projects/markdown/}} or
ReStructuredText\footnote{\url{http://docutils.sourceforge.net/rst.html}}.

If a public version control platform such as GitHub is used (see \refcrit{AC2}), the
README will usually be visualised on the repository page.

%\CheckTable

\newcommand{\isEightID}{IS8}
\newcommand{\isEightText}{Are there facilities to uninstall the software?}
\criterion{\isEightID}{\isEightText}

This is usually trivial when the software complies to IS4 and/or IS3, or if the
software installation is self-contained in a single directory. In all other
instances, proper uninstallation facilities should be explicitly implemented.

%\CheckTable

\newcommand{\isNineID}{IS9}
\newcommand{\isNineText}{Are the system requirements such as target platform clearly advertised?}
\criterion{\isNineID}{\isNineText}

The project website must clearly state what target platform(s) the software is expected to run on.
System requirements in terms of computing resources must be stated if the
software employs more than insignificant resources (see \refcrit{PF2}).
(related also to \refcrit{B3} and \refcrit{IS2})

\newcommand{\isTenID}{IS10}
\newcommand{\isTenText}{Is the software deployable through an established container or virtualisation solution?}
\criterion{\isTenID}{\isTenText}

Certain software projects benefit from being installable either in a
containerised form, or in fully virtualised form (i.e. a virtual machine). This
applies particularly to software that instead of being a standalone tool, is a
complex interplay between different components. Such software is often a web
application or webservice that builds on a large number of dependencies that
may be non-trivial to install or configure. A container or virtual machine
offers a solution to facilitate deployment. For CLARIAH this usually implies
easier deployment by a CLARIN centre.

Containerisation solutions:

\begin{itemize}
    \item Docker -- \url{https://docker.com}
    \item Ansible -- \url{https://ansible.com}
\end{itemize}

Virtualisation solutions:

\begin{itemize}
    \item Vagrant -- \url{https://www.vagrantup.com} (builds on VirtualBox or VMWare)
\end{itemize}

This criterion can either complement or completely replace \refcrit{IS3} and
\refcrit{IS4}. In such a case, the solutions offered by \refcrit{IS3} and
\refcrit{IS4} are often not sufficient for a simple deployment. For simple
standalone tools, however, this guideline is generally not applicable.

%\CheckTable

\subsection{Performance}\label{sec:per}

The guidelines in this category are typically hard to assess, as they require
an intricate knowledge of the design of the system. 

\newcommand{\pfOneID}{PF1}
\newcommand{\pfOneText}{Does the software perform its function(s) efficiently?}
\criterion{\pfOneID}{\pfOneText}

Software should employ efficient algorithms to perform its task. This is quite
non-trivial to assess, especially from a third party perspective, but obvious
performance bottlenecks \emph{may} be an indication of sub-optimal design
choices.

% * <reinierdevalk@gmail.com> 2016-03-30T16:05:28.991Z:
%
% > should employ efficient algorithms
%
% Agreed, but is it feasible to check this? (Applies both to the party providing the software and the one preserving it.) 
%
% ^.
%<proycon@anaproy.nl> 2016-04-06: This is indeed hard to check and
%would require a proper source code audit, obvious performance bottlenecks
%during normal operation (at least this can be identified more easily) may be
%an indication that such a deeper check is warranted.
(Consider also U4)

%\CheckTable

\newcommand{\pfTwoID}{PF2}
\newcommand{\pfTwoText}{Does the software make a reasonable demand on computing resources?}
\criterion{\pfTwoID}{\pfTwoText}

Software should not place an unreasonable demand on computing resource (memory,
CPU time). Overuse of resources \emph{may} be an indication of sub-optimal
design choices.  This is not easy to assess, but if the software does make high
demand on particular resources, then this should be clearly advertised and
explained (in the documentation for instance) and also stated as part of the
system requirements.

Be aware that there is often a trade-off between speed and memory usage. The
developer of the decision must make an appropriate decision favouring one or the other
given the task at hand. 

%\CheckTable

\newcommand{\pfThreeID}{PF3}
\newcommand{\pfThreeText}{Does the software make efficient use of available resources?}
\criterion{\pfThreeID}{\pfThreeText}

Software should make use of available resources if it helps getting the job
done faster. For instance, if multiple CPU cores are available and the task at hand
would benefit significantly from parallelisation, then the implementation should be
multithreaded to make use of said cores.

%\CheckTable

\newcommand{\pfFourID}{PF4}
\newcommand{\pfFourText}{Is the interface responsive?}
\criterion{\pfFourID}{\pfFourText}

GUIs and WUIs should be responsive and deliver clear feedback when the user is
to await the completion of the task. GUIs and WUIs should use asynchronous methods for
handling user-interface interaction. Interaction with the interface should not
be blocked needlessly.

Webservice interfaces need to be similarly responsive and not time out. Server interfaces furthermore demand multi-threading and need to be able to handle
multiple clients concurrently.

%\CheckTable

\newcommand{\pfFiveID}{PF5}
\newcommand{\pfFiveText}{Does the software scale as intended?}
\criterion{\pfFiveID}{\pfFiveText}

Proper facilities should be implemented to allow scaling \emph{if} the software is
intended to handle big data and/or large user bases. Performance should remain
acceptable at such levels.  Scaling can be horizontal, where multiple CPU cores
or multiple machines are put to work to distribute the load, or vertically by
simply adding more resources. In a data-sense, horizontal scaling often implies
a partitioning of the data.

%\CheckTable

\section{Quality Assessment Criteria - Sustainability and Maintainability}\label{sec:sustAndMaint}

\subsection{Identity}\label{sec:ide}

\newcommand{\idOneID}{ID1}
\newcommand{\idOneText}{Does the software provide a clear and unique identity?}
\criterion{\idOneID}{\idOneText}

Software should be identifiable by a clear name that does not clash with others
in its application domain and in the wider generic software domain. 

%\CheckTable

\newcommand{\idTwoID}{ID2}
\newcommand{\idTwoText}{Does the software have a website?}
\criterion{\idTwoID}{\idTwoText}

The software should have a website that describes it, allows users to obtain
it as well as its documentation. The website should be a portal to everything
related to the software.

It is recommended to have a dedicated domain name or subdomain.

%\CheckTable

\newcommand{\idThreeID}{ID3}
\newcommand{\idThreeText}{Does the project name not violate an existing trade-mark?}
\criterion{\idThreeID}{\idThreeText}

The project name should not infringe on existing trade-marks.

%\CheckTable

\subsection{Copyright \& Licensing}\label{sec:cop}

\newcommand{\cpOneID}{CP1}
\newcommand{\cpOneText}{Has an appropriate open-source license been adopted?}
\criterion{\cpOneID}{\cpOneText}

Software should be released as open source under an appropriate license. The
license should be recognised by the Open Source Initiative\footnote{https://opensource.org} or Free Software
Foundation\footnote{http://fsf.org}. Open source licenses generally fall somewhere on a spectrum between the following two flavours:

\begin{itemize}
 \item Copyleft license (GNU Public License (GPL) and variants): If the code is modified and distributed, the modified code must be distributed under the same license. Also, if your code is licensed in this way and is used as a library or imported module by other software, then that software too is considered a derivate work and must be distributed under the same license. This ensures that anybody who derives a work from your code is obliged to distribute it as open source (free\footnote{Free as in free speech, not free beer.} software) as well.
 \item Permissive license (MIT, Apache License, BSD license): Allows modifications and reuse with fewer restrictions. Generally anybody is free to use the code and use it in closed-source (non-free) software.
\end{itemize}

The decision which license is most appropriate should be made on an individual
basis. Note that none of the major licenses prohibits commercial use, nor is it
recommended to do so.

The license chosen for the software may never violate the licenses of any of
the software's dependencies, if there are linking clauses in those. \footnote{This implies you that if you use GPL libraries, you can no longer release your software under a more permissive license such as MIT.}.

Any form of closed-source software is strongly discouraged as it is found to be
at odds with a scientific method that relies on methodological transparency,
reproducibility and peer review. The CLARIAH project is explicitly committed to
open source and ``aims to create an inclusive open-source engineering community
that will carry on providing new tools and support for users after the end of
the CLARIAH project'' \citep{CLARIAHTECHPLAN1}.

%\CheckTable

\newcommand{\cpTwoID}{CP2}
\newcommand{\cpTwoText}{Is it clear who wrote the software, who owns the copyright, and what the license is?}
\criterion{\cpTwoID}{\cpTwoText}

Authors, copyright and license should be clearly mentioned on the project
website, as well as in the source code.  We strongly recommend each source file to contain a comment header stating this information.

%\CheckTable

\newcommand{\cpThreeID}{CP3}
\newcommand{\cpThreeText}{Are the funders acknowledged?}
\criterion{\cpThreeID}{\cpThreeText}

Any external funders of the software should be publicly acknowledged on the project
website, and in the \texttt{README}.

%\CheckTable

\subsection{Accessibility}\label{sec:acc}

\newcommand{\acOneID}{AC1}
\newcommand{\acOneText}{Is the source code maintained in a version control system?}
\criterion{\acOneID}{\acOneText}

Source code must always be kept under version control. This enables developers
to collaborate, maintains a perfect version history, and allows everyone keep a
track of changes. Absence of version control is bad development practice and
unsustainable.

%\CheckTable

\newcommand{\acTwoID}{AC2}
\newcommand{\acTwoText}{Is the source code in a public version-controlled repository?}
\criterion{\acTwoID}{\acTwoText}

The version control repository should be public (read only) in the spirit of both open
source, as well as the scientific method (transparency, peer review,
reproducibility). 

It is recommended to use a large sustainable third-party platform
(GitHub\footnote{\url{https://github.com}},
GitLab\footnote{\url{https://gitlab.com}},
BitBucket\footnote{\url{https://bitbucket.org}},
Launchpad\footnote{\url{https://launchpad.net}}), as it offers a social
dimension and is more likely to live beyond the lifetime of any current project
funding line. Use of sourceforge.net, de-facto standard prior to the advent of
current leader Github, is strongly discouraged as it is under ill management
and no longer deemed secure. Public repositories tend to be free on these platforms and
often come with many additional benefits, such as a good public issue tracker
(see \refcrit{SP2}).

We motivate our reasoning in favour of public version control repositories as follows:

\begin{enumerate}
    \item increased visibility for the project, especially if a major external platform is used.
    \item facilitates judgement of development activity of a software project.
    \item facilitates collaboration, especially from external partners.
    \item invites peer review
    \item allows users to be more intimately aware of changes, which in an
        academic context may impact their experimental results.
\end{enumerate}

(depends on \refcrit{AC1})

%\CheckTable

\newcommand{\acThreeID}{AC3}
\newcommand{\acThreeText}{Is there no restricted data in the public source code repository?}
\criterion{\acThreeID}{\acThreeText}

Privacy and security sensitive data (passwords, API keys), as well as other
restricted data, should never be checked into public version control systems.

%\CheckTable

\newcommand{\acFourID}{AC4}
\newcommand{\acFourText}{Are there clearly marked formal releases of the software?}
\criterion{\acFourID}{\acFourText}

The software should be formally released when the developers deem it a good
time. Each release should be clearly marked with the version number of the
software, according to a consistent scheme. The version control system should
also have clearly identifiable tags that mark the state of the repository at
the time of this release.

%\CheckTable

\newcommand{\acFiveID}{AC5}
\newcommand{\acFiveText}{Is the software deposited in a public repository for the language?}
\criterion{\acFiveID}{\acFiveText}

The ecosystems surrounding various high-level programming languages offer
public repositories in which installable software release packages should be
deposited. These repositories function as a primary source for installation (see also
\refcrit{IS3}) and generally offer automatic dependency management.

Use of these repositories is strongly recommended practice if available.

\begin{itemize}
    \item \textbf{Python} -- Releases of Python software should be deposited in
        the Python Package Index: \url{https://pypi.python.org}
    \item \textbf{Perl} --  Perl modules should be deposited in the Comprehensive
        Perl Archive Network (CPAN): \url{http://www.cpan.org}.
    \item \textbf{Java} -- Java components built using Maven and various other
        build systems should be deposited in The Central Repository: \url{http://central.sonatype.org/}
    \item \textbf{R} -- R packages should be deposited in the Comprehensive R
        Archive Network (CRAN): \url{https://cran.r-project.org}
    \item \textbf{Javascript} -- Javascript software should be deposited in
        NPM: \url{https://www.npmjs.com}
    \item \textbf{Ruby} -- Ruby software should be deposited to 
        RubyGems: \url{https://rubygems.org}
\end{itemize}

%\CheckTable

\newcommand{\acSixID}{AC6}
\newcommand{\acSixText}{Is the software available in the target platform's software repository?}
\criterion{\acSixID}{\acSixText}

If the software has a well-defined target platform and the target platform has
a public software repository available, then it is recommended to submit the
packages to this public repository.

This may however conflict with \refcrit{AC5}, be rendered obsolete by
\refcrit{AC5}, or render \refcrit{AC5} obsolete. The decision which form of
installation takes precedence should be made on an individual basis,
considering the nature and audience of the software.

\begin{itemize}
    \item Linux/BSD/Unix -- Distributions generally have their own official repositories. Package inclusion can be a lengthy process.
    \begin{itemize}
        \item Arch Linux: Submit your package to the Arch User Repository\footnote{https://aur.archlinux.org} (AUR)
		\item CentOS: Consult https://wiki.centos.org/HowTos/Packages/ContributeYourRPMs
        \item Debian: Consult https://wiki.debian.org/Packaging
        \item Fedora: Consult https://fedoraproject.org/wiki/Packaging:Guidelines
        \item Ubuntu: Ubuntu inherits Debian's packages, therefore submitting Debian is recommended. Users can submit software  to Package Archive (PPA).
    \end{itemize}
    \item Mac OS X: use the Apple Store for traditional Mac applications intended for end-users; for the more Unix-style
        software, use a system such as Homebrew or MacPorts.
    \item Android: use the Google Store, fully open-source software may also be submitted to F-Droid\footnote{https://f-droid.org}
    \item iOS: Use the Apple Store
    \item Windows: (TODO) 
\end{itemize}

It is not always time -and cost-effective to package and deposit the software,
especially not when there are multiple target platforms. When the software is
not intended for adoption by a wider audience, it can be considered to ignore
this guideline.

%\CheckTable

\newcommand{\acSevenID}{AC7}
\newcommand{\acSevenText}{Is each software release deposited in a persistent store with a unique DOI?}
\criterion{\acSevenID}{\acSevenText}

Persistent stores aim to preserve research output. They aim for a longevity
that can not be guaranteed by normal software repositories. Each software
release should be deposited in a persistent store and receive a unique Digital
Object Identifier (DOI),  a persistent ID that is ubiquitous in the academic
world. The DOI can in turn be referenced in citations from publications.

Users using Github (see \refcrit{AC2}) and Zenodo\footnote{\url{https://zenodo.org}, a
research data repository.} can set this up with little effort. \footnote{A guide
for this is provided at https://guides.github.com/activities/citable-code/.}.
Any releases made on GitHub will then automatically transfer to Zenodo and
receive a DOI, no user intervention is necessary. 

%\CheckTable

\subsection{Community}\label{sec:com}

This aspect concerns to what extent an active user community exists for the
software.

\newcommand{\cmOneID}{CM1}
\newcommand{\cmOneText}{Is there evidence of the software being in use by others?}
\criterion{\cmOneID}{\cmOneText}

It is recommended to list on the project website if the software has 
users aside from the primary developers and their immediate institution, this
may facilitate further adoption. It is also recommended to mention (on the
project website) a citable publication that discusses software, and to actively
recommend or even require others to cite this if they use the software.

A list of third-party publications citing the software is also recommended for
the project page. Alternatively, it is lightly recommended to
incorporate success stories or quotes from satisfied users on the project
website.

In addition to the aforementioned, evidence for this activity may also be sought
in contributions to the software by external developers (assuming
compliance with \refcrit{AC2}), or questions in the issue/bug tracker or mailing lists.

%\CheckTable

\newcommand{\cmTwoID}{CM2}
\newcommand{\cmTwoText}{Is there evidence of external developers?}
\criterion{\cmTwoID}{\cmTwoText}

External developers, not related to the original developers or their
institution, are a good sign of community interest. The threshold for
contributing is relatively high. Compliance with this criterion is greatly
facilitated by compliance with \refcrit{AC2}.

%\CheckTable

\newcommand{\cmThreeID}{CM3}
\newcommand{\cmThreeText}{Are there statistics available on software use?}
\criterion{\cmThreeID}{\cmThreeText}

To gain an impression of user interest, it is recommended to have statistics
available on how often the software is downloaded or how often the project page
is visited. Public availability of these statistics (stripped of any privacy
sensitive information), is recommended.

%\CheckTable

\subsection{Testability}\label{sec:tes}

\newcommand{\tsOneID}{TS1}
\newcommand{\tsOneText}{Does the project have unit tests, is there sufficient coverage?}
\criterion{\tsOneID}{\tsOneText}

Software should have unit tests that automatically test individual units of the
source code. They verify the data and logic flow by testing whether the output,
given certain input, confirms to expectation.

It is important that the tests cover enough of the source code. This is not
trivial to assess, but automated tools
and platforms are available that can help in this assessment, such as
\url{https://coveralls.io} and \url{https://codacy.com}.

%\CheckTable

\newcommand{\tsTwoID}{TS2}
\newcommand{\tsTwoText}{Does the project have integration tests, is there sufficient coverage?}
\criterion{\tsTwoID}{\tsTwoText}

Software should have integration tests that combine invididual parts of modules and see how they function
as a group.

%\CheckTable

\newcommand{\tsThreeID}{TS3}
\newcommand{\tsThreeText}{Does the project have automated GUI tests?}
\criterion{\tsThreeID}{\tsThreeText}

If the software offers a non-trivial GUI or WUI, it should have automated tests
that verify whether the interface functions as intended.

%\CheckTable

\newcommand{\tsFourID}{TS4}
\newcommand{\tsFourText}{Are tests run automatically?}
\criterion{\tsFourID}{\tsFourText}

It is recommended that tests be run automatically at the end of the build
or installation process. This is best accomplished through continuous
integration testing. Free public continuous integration
platforms such as Travis-CI\footnote{https://travis-ci.org},
Gitlab-CI\footnote{https://gitlab.com},
Jenkins\footnote{https://jenkins-ci.org} can be hooked into version control
systems (see \refcrit{AC1}, \refcrit{AC2}) with minimal effort, resulting in an
automatic run of the test suite upon each commit and notifying the developer
when the test suite fails.

It is strongly recommended that test results are publicly available, so users
can more quickly assess software quality.

\subsection{Portability}\label{sec:por}

Portability concerns the extent to which software can be used on multiple
platforms. 

\newcommand{\pbOneID}{PB1}
\newcommand{\pbOneText}{Is it clear for what platforms the software is written?}
\criterion{\pbOneID}{\pbOneText}

It should be clear, from at least the project website, for what platforms the software
is intended.

%\CheckTable

\newcommand{\pbTwoID}{PB2}
\newcommand{\pbTwoText}{Is the software portable for multiple platforms?}
\criterion{\pbTwoID}{\pbTwoText}

It is recommended to support a wide variety of platforms rather than a single
one. This, however, is not always feasible or cost and time effective.

%\CheckTable

\newcommand{\pbThreeID}{PB3}
\newcommand{\pbThreeText}{Does the software work on multiple browsers?}
\criterion{\pbThreeID}{\pbThreeText}

This concerns only web-based software with a significant client-side component.
Such software should function under recent versions of all major browsers
(Mozilla Firefox, Google Chrome, Internet Explorer / Edge, Safari, Opera), and
never be limited to just one.

Moreover, it is recommended that such software does not rely on browser plugins that
are themselves not portable. Adobe Flash or Microsoft Silverlight are two
examples of badly portable legacy technologies that should be avoided, always use
modern substitutes (HTML5) instead.

%\CheckTable

\subsection{Supportability}\label{sec:sup}

To what extent will the product be supported currently and in the future?

\newcommand{\spOneID}{SP1}
\newcommand{\spOneText}{Is it clear whom to contact for support?}
\criterion{\spOneID}{\spOneText}

It should be clear where to go for support. A project must have a contact
e-mail address. If an issue tracker is present (SP2), it should be clearly advertised
as well.

%\CheckTable

\newcommand{\spTwoID}{SP2}
\newcommand{\spTwoText}{Are there public support channels available?}
\criterion{\spTwoID}{\spTwoText}

A public issue/bug tracker is strongly recommended. It allows everyone to post
bugs or features requests and allows users to see what issues are current, how
they are resolved, and if they are resolved in a timely fashion. It also
prevents duplication of issues, and gives a platform for
tracking feature requests.

Alternatively, a mailing list may as a lesser substitute for an issue tracker.
It can also serve as an extra line of communications between users
and developers. The mailing list must allow anyone to subscribe and must have a
public archive allowing users to follow any issues and follow development.

Other extra support channels may take the form of an IRC channel or services
such as Slack\footnote{\url{https://slack.com/}} or
Gitter\footnote{\url{https://gitter.com}}.

%\CheckTable

\subsection{Analysability}\label{sec:ana}

This section concerns the extent to which the source code can be understood. At
this level, the source code is inspected in closer detail.

\newcommand{\anOneID}{AN1}
\newcommand{\anOneText}{Is the source code structured adequately?}
\criterion{\anOneID}{\anOneText}

Source code should be modular, i.e. it should be structured into multiple
modules/packages, following the requirements and conventions of the programming
language. 

The structure of the source code should bear a clear relationship to the
architecture or design of the software.

(See also \refcrit{R3})

%\CheckTable

\newcommand{\anTwoID}{AN2}
\newcommand{\anTwoText}{Is the source code commented adequately?}
\criterion{\anTwoID}{\anTwoText}

The source code should contain comments explaining what major blocks do.

%\CheckTable

\newcommand{\anThreeID}{AN3}
\newcommand{\anThreeText}{Do the comments generate API documentation?}
\criterion{\anThreeID}{\anThreeText}

The comments use a mark-up that allows them to be used directly as the
source for the generation of the API reference documentation. This is
% * <reinierdevalk@gmail.com> 2016-03-31T08:39:14.635Z:
%
% > use a mark-up that allows them to be used directly as the
% > source for the generation of the API reference documentation
%
% This does not always apply to all comments
%
% ^.
accomplished using document generation tools such as Doxygen\footnote{\url{http://www.doxygen.org}}, Sphinx\footnote{\url{http://www.sphinx-doc.org}} or
Javadoc\footnote{\url{http://www.oracle.com/technetwork/java/javase/documentation/index-jsp-135444.html}}.

%\CheckTable

\newcommand{\anFourID}{AN4}
\newcommand{\anFourText}{Is the source code cleanly laid out?}
\criterion{\anFourID}{\anFourText}

The source code should follow a proper indentation convention. 

%\CheckTable

\newcommand{\anFiveID}{AN5}
\newcommand{\anFiveText}{Are sensible names used?}
\criterion{\anFiveID}{\anFiveText}

Do the classes, functions and variables in the source code use sensible names
and do they follow a consistent naming scheme that is conventional for the
programming language?

%\CheckTable

\newcommand{\anSixID}{AN6}
\newcommand{\anSixText}{Are there no (large) blocks of commented out code or obsolete files?}
\criterion{\anSixID}{\anSixText}

There should be no large blocks of commented-out code, nor obsolete or
alternate versions of files that bypass proper version control.
(this depends on compliance with \refcrit{AC1})

%\CheckTable

\newcommand{\anSevenID}{AN7}
\newcommand{\anSevenText}{Are all TODO comments resolved?}
\criterion{\anSevenID}{\anSevenText}

There should be no important TODO comments, if there are they should at least be
clearly described in the issue tracker (See \refcrit{SP2}).

%\CheckTable

\newcommand{\anEightID}{AN8}
\newcommand{\anEightText}{Does the project have recommended coding standards?}
\criterion{\anEightID}{\anEightText}

A project should have recommended coding standards to which contributors
should adhere. These standards should be consistent with the larger community
of generic coding standards for the programming language.

\subsection{Changeability}\label{sec:cha}
% * <reinierdevalk@gmail.com> 2016-03-31T08:57:05.619Z:
%
% > Changeability
%
% This seems very important to me. Contribution of outside parties can
% complicate sustainability, preservation, or even just keeping things under
% control, quite a bit.
%
% ^.
%<proycon@anaproy.nl> 2016-04-06: For me this is an important issue as well,
%but I'm of the view that outside contribution is generally a very good thing
%for sustainability. It means the software is 'alive' and people take an active
%interest in it, which is not hard to accomplish and should be cherished. If proper
%infrastructure is set up (public version control, pull requests, issue
%trackers), then projects are easier to manage and keep under control. Major
%open source software projects often have dedicated people judging and
%accepting/rejecting patches of others and merging them back into the master branch, Linus Torvalds is a
%good example of this. I think, however, in the scope we operate, that we
%should already be happy if we receive the occassional pull request and
%definitely not see this as a threat.
%The main sustainability problem, in my view, is that of software dying out and going stale in
%repositories because nobody develops it anymore and nobody uses it anymore.
%Outside contribution also generally improves the quality as people generally
%come with a fresher perspective and see things the original developers may not have spotted.

%\CheckTable

\newcommand{\chOneID}{CH1}
\newcommand{\chOneText}{Is the project open to contributions from third parties?}
\criterion{\chOneID}{\chOneText}

A project is recommended to be open to outside contributions. Community
involvement is a major factor in the success of a software project. 

This depends on \refcrit{AC1}, \refcrit{AC2}, and facilitates \refcrit{CM1} and \refcrit{CM2}. 

%\CheckTable

\newcommand{\chTwoID}{CH2}
\newcommand{\chTwoText}{Does the project have guidelines for contributions?}
\criterion{\chTwoID}{\chTwoText}

If contributions are desired, a project is recommended to have guidelines for
contributors. These must be publicly available.
See also \refcrit{AN8}.

%\CheckTable

\newcommand{\chThreeID}{CH3}
\newcommand{\chThreeText}{Are all source code changes, and their authorship, publicly visible?}
\criterion{\chThreeID}{\chThreeText}

Collaboration requires awareness of what changes, when it changes, and who
changes it. This depends on AC1, heavily facilitated by AC2.

Alternatively, commit messages can be forwarded to mailing lists, Slack, IRC chat, or
whatever is deemed appropriate.

%\CheckTable

\newcommand{\chFourID}{CH4}
\newcommand{\chFourText}{Is the software sufficiently backward compatible?}
\criterion{\chFourID}{\chFourText}

Software should be backward compatible with old versions of itself with respect
to public interfaces and data input. Backward compatibility changes and
deprecation may occur but should always be clearly announced ahead of time.

%\CheckTable

\newcommand{\chFiveID}{CH5}
\newcommand{\chFiveText}{Is there a roadmap for future changes?}
\criterion{\chFiveID}{\chFiveText}

Software still under active development should have a roadmap. This may take
shape either as an explicit roadmap or implicit in the issue tracker
(SP2) through the assignment of milestones.

%\CheckTable

\newcommand{\chSixID}{CH6}
\newcommand{\chSixText}{Does the website mention how the software is funded and when funding ends?}
\criterion{\chSixID}{\chSixText}

Funders should be acknowledged publicly and people should be aware when
software is no longer actively developed.

%\CheckTable

\subsection{Reusability}\label{sec:reu}

\newcommand{\rOneID}{R1}
\newcommand{\rOneText}{Does the software offer all the appropriate interfaces?}
\criterion{\rOneID}{\rOneText}

The use of multiple interfaces enables reusability. Different users should be
able to access the software at different levels. End-users usually require a
GUI or WUI, developer-users prefer a command line interface (CLI) and/or
webservice interface, developers needs a software library with API.

Assessment of this depends greatly on the intended audience of the software
(see \refcrit{U2}) and desired level of reusability.

%\CheckTable

\newcommand{\rTwoID}{R2}
\newcommand{\rTwoText}{Is the software modular, can the software support multiple interfaces?}
\criterion{\rTwoID}{\rTwoText}

Software should be set up in such a manner that higher-level interfaces can be
constructed on its lower-level components. This implies that there should be a
clear separation between front-end and back-end. In order to achieve this it we
strongly recommend software to be set up in a modular fashion, allowing reuse
of its components without the need to modify these components.

Modularity can be expressed as layers, from low-level to high-level:

\begin{itemize}
 \item Classes and functions are defined at the source code level. (see
     \refcrit{R3})
 \item Libraries group and expose these publicly, described by APIs. 
 \item Command Line Interfaces use the libraries.
 \item Servers/daemons use the libraries, networked clients use the server/daemons.
 \item GUIs and WUIs use either the libraries, the CLI tools, or act as a networked client to the servers/daemons.
\end{itemize}

Whenever two or more of these layers are intrinsically merged, reusability potential is lost.
For instance, monolithic software that offers only a GUI interface can not be
readily adapted to add a CLI or web interface.

Maximum reusability is not always desired or time- and cost-effective. The
desired degree of modularity and reusability is to be assessed on an
individual basis.

%\CheckTable

\newcommand{\rThreeID}{R3}
\newcommand{\rThreeText}{Is the software's source code set up in a modular fashion?}
\criterion{\rThreeID}{\rThreeText}
% * <reinierdevalk@gmail.com> 2016-03-31T09:08:08.123Z:
%
% > R3 -- Is the software's source code set up in a modular fashion?
%
% Maybe superfluous (as already covered by R2)?
%
% ^.
%<proycon@anaproy.nl> 2016-04-06:  It's included in R2 yes, but this one is a
%bit more specific and narrow-scoped. Even if R2 is not met, R3 should be.

The software's source code should define clearly delimited classes and
functions, following the paradigm of the programming language. 

(See also \refcrit{AN1})

%\CheckTable

\subsection{Security \& Privacy}\label{sec:sec}

\newcommand{\scOneID}{SC1}
\newcommand{\scOneText}{Is the software free of obvious security flaws?}
\criterion{\scOneID}{\scOneText}

Software should be secure and have no holes that allow unauthorised users to
gain access. Developers should take care to avoid common attack vectors such as
shell injection, SQL injection, cross-site request forgery, buffer overflow.
Proper validation of user input is a major factor in preventing security holes

Assessment of security is an art in itself and non-trivial. 

%\CheckTable

\newcommand{\scTwoID}{SC2}
\newcommand{\scTwoText}{Is user privacy secured effectively?}
\criterion{\scTwoID}{\scTwoText}

Privacy-sensitive user data must be treated with care. Passwords must never be
stored in unhashed form, private keys must never be shared. Any compromises to
privacy must be clearly laid out in a privacy policy.  Strongly related to SC1.
Also see \refcrit{AC3}.

%\CheckTable

\subsection{Interoperability}\label{sec:int}

\newcommand{\ipOneID}{IP1}
\newcommand{\ipOneText}{Does the software use appropriate open standards for data?}
\criterion{\ipOneID}{\ipOneText}

Software should adhere to appropriate open standards as much as possible, i.e.
it should be able to read input files and write output files in open
standards. Support of multiple open standards is recommended. Conversion may
also be mediated through other third party tools. A counter indication for this
is when software uses its own ad hoc format when decent open alternatives
already exist.

%\CheckTable

\subsection{Interoperability for CLARIAH}\label{sec:intCLA}


\begin{TODO} 
What specific interoperabilities does CLARIAH demand from software for
integration into the larger infrastructure?  External input is greatly
appreciated here!
\end{TODO}

%<proycon@anaproy.nl> 2016-04-06: What does CLARIAH specifically demand from
%software for integration into the infrastructure? This is something to
%discuss, which I personally don't have a clear image of, and for which we will need input from others in CLARIAH.

\subsection{Governance}\label{sec:gov}

%<proycon@anaproy.nl> 2016-04-06: What are quality indicators for how software
%projects are run and managed? This is more on a high-level organizational and financial
%level on which I personally don't have much input.

%(Reinier) Can we find examples how these guidelines are implemented, how they are
%executed? Can we translate them into a survey, spreadsheet to compute the score
%if we could do so, and than run iterations, we could - hopefully - see that
%software quality is improving.


\begin{TODO} 
What are quality criteria for how software projects are run and
managed? This is more on a high-level organizational level.
\end{TODO}

%ENDCRITERIA

\section{Quality Measurement}

To achieve a measure of quality, evaluators can compute a quality score ($Q$,
expressed as a percentage) per dimension, by taking the sum of the scores for
the individual criteria. Responses yield scores on a scale from 0 (No) to 4 (Perfect). 

A score for the dimension can be computed following the simple formula in
Equation~\ref{eq1}, here $s$ corresponds to the cummulative score, and $n$ is
the number of questions, except those that are deemed not applicable.

\begin{equation}
\label{eq1}
Q = (s / (4n)) \cdot 100
\end{equation}

A score of 100 is achieved if all criteria are adhered to perfectly. This
threshold may be lowered by using for instance $2n$ or $3n$ instead of $4n$, in
which case scores over $100$ are possible.

\begin{TODO}
What quality thresholds do we set for CLARIAH, or is it up
to institutions or even projects themselves? Do we need pre-set thresholds at
all?
\end{TODO}



\section{Implementation}

The guidelines have been implemented as an interactive survey on
\url{http://softwarequality.clariah.nl}. Developers are
encouraged to use this to self-assess their projects. Project managers or end users can use
this to assess whether others' software meets their quality expectations.

For developers, filling the interactive survey will result in a
MarkDown document which is to be checked in alongside the code in the
software's version control repository.


%\section{TODO: section not finished, will become Implementation and a report how we used this later, for the time being it is our workflow with the document}
%AS April 5
%next steps: skype after April 12, possible steps:
%identify main software developers in each WP (with GertJan)
    %(<proycon@anaproy.nl> 2016-04-06: This may be a good starting place: https://github.com/orgs/CLARIAH/people)
%share text with them for comments - do they understand it
%set up a google sheet to collect tools by names - get an overview how much software we would like to have tested; 
%set up a survey using the criteria and run this through those tools which have been identified prior as very important - otherwise we create a major software testing operation 

%Content-wise: how is software quality related to software sustainability? How is software sustainability related to the expected lifetime of a software and its user community?
%To be discussed by skype can we come up with a classification of tools: tools for internal (ICT) use versus tools for user with no knowledge in code; tools which are inherent part of the infrastructure and should work in xxx years versus tools which are experimental or only supposed to work for shorter time periods and be replaced afterwards; could we organise a workshop and make people draw where their software tool live in such a space?

%workflow:
%1. Reinier (look link to data quality) April 22, Patrick/Peter (sustainability aspect) by end of April latest, Andrea would have another go on the document and make implementation section (use of the document until end of ….) - By April 22
%2. Andrea asks Peter who we know at the SSI in UK - could they look at our guidelines? [we added some dimensions, their feedback would be great] - April 15
%2. go back to Maarten for end-check - flag out where we need comments and where we need input [here - make visible in the PDF, i.e. Single-signon
%3. send it to Geert-Jan May 10-15 and ask him with whom to share further - would also ask him to make it more specific if needed; give scores to criteria depending on CLARIAH (may also depend on software) Emphasis given by the implementation for CLARIAH => thresholds we want to set 
%4.  share text with managers and developers (including implementation section) workshop (awareness raising, coupled to a more general infrastructure meeting, May 18 next/testing with specific people
%first run done pre-summer 

%ideally, developers have an assessment routine and we plugin 
%promote software which has run through it 
%interactive survey is a great idea
%SSI has also an interactive survey  



\bibliographystyle{plainnat}
\bibliography{softwareguidelines}
\addcontentsline{toc}{section}{References}

\pagebreak % force appendix on new page for printing convenience
\appendix
\newcommand{\appname}{Appendix A: Assessment Form}
\section*{\appname}

Assessment is best conducted using our online survey at
\url{http://softwarequality.clariah.nl} instead.
Alternatively, you can use this paper form presented here. Note that criteria
that are deemed not applicable can be striked through.

\addcontentsline{toc}{section}{\appname}
\begin{table}[hbt]
	\begin{tabular}
		{ p{0.1\textwidth-2\tabcolsep} | 
		  p{0.5\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} }
%		{ p{narr} p{wide} p{narr} p{narr} p{narr} }
		\hline
		\textbf{ID} & \textbf{Criterion} & \textbf{No} & \textbf{Min} & \textbf{Adq} & \textbf{Good} & \textbf{Perfect} \\
                                        & & \emph{0} & \emph{1} & \emph{2} & \emph{3} & \emph{4} \\
		\hline
		\multicolumn{7}{l}{\ref{sec:und}{ }\nameref{sec:und}} \\
		\hline
        \hyperref[id:U1]{\uOneID} & \uOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:U2]{\uTwoID} & \uTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:U3]{\uThreeID} & \uThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:U4]{\uFourID} & \uFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:U5]{\uFiveID} & \uFiveText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:U6]{\uSixID} & \uSixText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:doc}{ }\nameref{sec:doc}} \\
		\hline
		\hyperref[id:D1]{\dOneID} & \dOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D2]{\dTwoID} & \dTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D3]{\dThreeID} & \dThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D4]{\dFourID} & \dFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D5]{\dFiveID} & \dFiveText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D6]{\dSixID} & \dSixText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D7]{\dSevenID} & \dSevenText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D8]{\dEightID} & \dEightText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D9]{\dNineID} & \dNineText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D10]{\dTenID} & \dTenText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D11]{\dElevenID} & \dElevenText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:D12]{\dTwelveID} & \dTwelveText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:lea}{ }\nameref{sec:lea}} \\
		\hline
		\hyperref[id:L1]{\lOneID} & \lOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:L2]{\lTwoID} & \lTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:L3]{\lThreeID} & \lThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:L4]{\lFourID} & \lFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:L5]{\lFiveID} & \lFiveText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\end{tabular}
		\caption{Quality Assessment Criteria - Usability (1)}
		\label{tab:criteriaUsab1}
\end{table}	
	
\begin{table}[hbt]
	\begin{tabular}
		{ p{0.1\textwidth-2\tabcolsep} | 
		  p{0.5\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} }
		%		{ p{narr} p{wide} p{narr} p{narr} p{narr} }
		\hline
		\textbf{ID} & \textbf{Criterion} & \textbf{No} & \textbf{Min} & \textbf{Adq} & \textbf{Good} & \textbf{Perfect} \\
                                        & & \emph{0} & \emph{1} & \emph{2} & \emph{3} & \emph{4} \\
        \hline
        \multicolumn{7}{l}{\ref{sec:bui}{ }\nameref{sec:bui}} \\
		\hline
		\hyperref[id:B1]{\bOneID} & \bOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:B2]{\bTwoID} & \bTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:B3]{\bThreeID} & \bThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:B4]{\bFourID} & \bFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline			
		\multicolumn{7}{l}{\ref{sec:ins}{ }\nameref{sec:ins}} \\
		\hline
		\hyperref[id:IS1]{\isOneID} & \isOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:IS2]{\isTwoID} & \isTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:IS3]{\isThreeID} & \isThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:IS4]{\isFourID} & \isFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:IS5]{\isFiveID} & \isFiveText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:IS6]{\isSixID} & \isSixText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:IS7]{\isSevenID} & \isSevenText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:IS8]{\isEightID} & \isEightText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:IS9]{\isNineID} & \isNineText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:IS10]{\isTenID} & \isTenText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:per}{ }\nameref{sec:per}} \\
		\hline	
		\hyperref[id:PF1]{\pfOneID} & \pfOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:PF2]{\pfTwoID} & \pfTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:PF3]{\pfThreeID} & \pfThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:PF4]{\pfFourID} & \pfFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:PF5]{\pfFiveID} & \pfFiveText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline		
		\end{tabular}
		\caption{Quality Assessment Criteria - Usability (2)}
		\label{tab:criteriaUsab2}
\end{table}

\begin{table}[hbt]
	\begin{tabular}
		{ p{0.1\textwidth-2\tabcolsep} | 
		  p{0.5\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} }
		%		{ p{narr} p{wide} p{narr} p{narr} p{narr} }
		\hline
		\textbf{ID} & \textbf{Criterion} & \textbf{No} & \textbf{Min} & \textbf{Adq} & \textbf{Good} & \textbf{Perfect} \\
                                        & & \emph{0} & \emph{1} & \emph{2} & \emph{3} & \emph{4} \\
		\hline
		\multicolumn{7}{l}{\ref{sec:ide}{ }\nameref{sec:ide}} \\
		\hline
		\hyperref[id:ID1]{\idOneID} & \idOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:ID2]{\idTwoID} & \idTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:ID3]{\idThreeID} & \idThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:cop}{ }\nameref{sec:cop}} \\
		\hline
		\hyperref[id:CP1]{\cpOneID} & \cpOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:CP2]{\cpTwoID} & \cpTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:CP3]{\cpThreeID} & \cpThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:acc}{ }\nameref{sec:acc}} \\
		\hline
		\hyperref[id:AC1]{\acOneID} & \acOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AC2]{\acTwoID} & \acTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AC3]{\acThreeID} & \acThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AC4]{\acFourID} & \acFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AC5]{\acFiveID} & \acFiveText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AC6]{\acSixID} & \acSixText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AC7]{\acSevenID} & \acSevenText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:com}{ }\nameref{sec:com}} \\
		\hline
		\hyperref[id:CM1]{\cmOneID} & \cmOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:CM2]{\cmTwoID} & \cmTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:CM3]{\cmThreeID} & \cmThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
	\end{tabular}
	\caption{Quality Assessment Criteria - Sustainability and Maintainability (1)}
	\label{tab:criteriaSustMaint1}
\end{table}

\begin{table}[hbt]
	\begin{tabular}
		{ p{0.1\textwidth-2\tabcolsep} | 
		  p{0.5\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} }
        \hline
		\textbf{ID} & \textbf{Criterion} & \textbf{No} & \textbf{Min} & \textbf{Adq} & \textbf{Good} & \textbf{Perfect} \\
                                        & & \emph{0} & \emph{1} & \emph{2} & \emph{3} & \emph{4} \\
        \hline 
        		\multicolumn{7}{l}{\ref{sec:tes}{ }\nameref{sec:tes}} \\
		\hline
		\hyperref[id:TS1]{\tsOneID} & \tsOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:TS2]{\tsTwoID} & \tsTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:TS3]{\tsThreeID} & \tsThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:TS4]{\tsFourID} & \tsFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
        \multicolumn{7}{l}{\ref{sec:por}{ }\nameref{sec:por}} \\
		\hline	
		\hyperref[id:PB1]{\pbOneID} & \pbOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:PB2]{\pbTwoID} & \pbTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:PB3]{\pbThreeID} & \pbThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline	
		\multicolumn{7}{l}{\ref{sec:sup}{ }\nameref{sec:sup}} \\
		\hline	
		\hyperref[id:SP1]{\spOneID} & \spOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:SP2]{\spTwoID} & \spTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:ana}{ }\nameref{sec:ana}} \\
		\hline	
		\hyperref[id:AN1]{\anOneID} & \anOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AN2]{\anTwoID} & \anTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AN3]{\anThreeID} & \anThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AN4]{\anFourID} & \anFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AN5]{\anFiveID} & \anFiveText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AN6]{\anSixID} & \anSixText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AN7]{\anSevenID} & \anSevenText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:AN8]{\anEightID} & \anEightText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
	\end{tabular}
	\caption{Quality Assessment Criteria - Sustainability and Maintainability (2)}
	\label{tab:criteriaSustMaint2}
\end{table}

\begin{table}[hbt]
	\begin{tabular}
		{ p{0.1\textwidth-2\tabcolsep} | 
		  p{0.5\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} |
		  p{0.08\textwidth-2\tabcolsep} }
        \hline
		\textbf{ID} & \textbf{Criterion} & \textbf{No} & \textbf{Min} & \textbf{Adq} & \textbf{Good} & \textbf{Perfect} \\
                                        & & \emph{0} & \emph{1} & \emph{2} & \emph{3} & \emph{4} \\
        \hline 
		\multicolumn{7}{l}{\ref{sec:cha}{ }\nameref{sec:cha}} \\
		\hline	
		\hyperref[id:CH1]{\chOneID} & \chOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:CH2]{\chTwoID} & \chTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:CH3]{\chThreeID} & \chThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:CH4]{\chFourID} & \chFourText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:CH5]{\chFiveID} & \chFiveText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:CH6]{\chSixID} & \chSixText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
        \multicolumn{7}{l}{\ref{sec:reu}{ }\nameref{sec:reu}} \\
		\hline	
		\hyperref[id:R1]{\rOneID} & \rOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:R2]{\rTwoID} & \rTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:R3]{\rThreeID} & \rThreeText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:sec}{ }\nameref{sec:sec}} \\
		\hline	
		\hyperref[id:SC1]{\scOneID} & \scOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hyperref[id:SC2]{\scTwoID} & \scTwoText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:int}{ }\nameref{sec:int}} \\
		\hline	
		\hyperref[id:IP1]{\ipOneID} & \ipOneText & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:intCLA}{ }\nameref{sec:intCLA}} \\
		\hline	
		&  [TODO] & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
		\multicolumn{7}{l}{\ref{sec:gov}{ }\nameref{sec:gov}} \\
		\hline	
		& [TODO] & $\square$ & $\square$ & $\square$ & $\square$ & $\square$ \\
		\hline
\end{tabular}
	\caption{Quality Assessment Criteria - Sustainability and Maintainability (3)}
	\label{tab:criteriaSustMaint3}
\end{table}
\end{document}