\documentclass{scrartcl}

\usepackage{cleveref}
\usepackage{enumerate}
\usepackage[hidelinks]{hyperref}
\usepackage{minted}
\usepackage{skak}
\usepackage{tikz}
	\usetikzlibrary{arrows, matrix, positioning}

\title{The {\tt chesshs} Haskell Package}
\subtitle{Test Plan, Manual Testing \& Automated Testing}
\author{
	\begin{tabular}{r l}
		Ren\'e den Hertog & {\tt 1007823} \\
		Camil Staps       & {\tt 4498062} \\
		Erin van der Veen & {\tt 4431200}
	\end{tabular}
}
% Remove the date from '\maketitle'.
\makeatletter
	\patchcmd
		{\@maketitle}
		{{\usekomafont{date}{\@date\par}}\vskip\z@\@plus 1em}
		{}
		{}
		{}
\makeatother

\begin{document}

\maketitle

\section{Test Plan}

\subsection*{System Under Test (SUT) Description}

The chosen SUT is the {\tt chesshs} Haskell package. \cite{SUT} It should be able to run in any software environment and hardware platform which supports Haskell. The latest version of the SUT, since the 13\textsuperscript{th} of March 2015, is {\tt 0.2.1}.

\paragraph{Executing/Running the SUT} \label{P:ERSUT}

It is required to be able to compile Haskell, typically through the Glasgow Haskell Compiler (GHC), and having the {\tt chesshs} package installed, normally through Cabal via the \mintinline{bash}{cabal install chesshs} command. \cite{GHC, C} Both the GHC and Cabal are part of the Haskell Platform. \cite{HP} Under our computer systems, compiling and starting the SUT is done with the \mintinline{bash}{make} and \mintinline{bash}{./runchess} commands, respectively.

The SUT is used by providing it with, via the standard input, a text file containing a game of Chess in the Portable Game Notation (PGN). \cite{PGN} After computation, via the standard output, some general information about the Chess match is displayed, including the end result. Followed by the state of the board after each move. In any other case, an error message is displayed. \\

As is common with everything software, the SUT is likely to contain faults. However, the risks surrounding the program are near zero due to the very low impact of the program faulting. As of our expectation, the most ``risky" scenario for the SUT is at a notable Chess competition. In the case the program does not behave as intended, it is almost certain that a human referee would take over its place. Furthermore, we expect such a system to be used as a supporting tool for match officials and not as a replacement for regulation.

\section{Test method}
\subsection*{Test Goals}

\paragraph{Specification}

The specification of the SUT is for the most part similar to the input-output behavior described in \S\ref{P:ERSUT}. Because the SUT is strongly related to Chess, many finer details of its specification can be derived from documentation of the world of Chess.

\subparagraph{F\'ed\'eration Internationale Des Echecs (FIDE)} Due to the popularity of Chess, most people are familiar with how it is played. Hence, a summary of the rules would be excesive. However, due to the competitve nature of the game, the World Chess Federation describes the rules painfully precise. \cite{FIDE}

\subparagraph{Portable Game Notation (PGN)} Many Chess matches are recorded with the PGN. \cite{PGN} As some of the test cases consist of text files describing Chess games in PGN, one of our test cases is included in listing \ref{L:PGNE} as an example for the notation.

\begin{listing}
	\inputminted{text}{../test/database/m0.in}
	\caption{Portable Game Notation (PGN) Example} \label{L:PGNE}
\end{listing}

PGN consists of two parts: first, the tag pairs and, second, the movetext.

The tag pairs encode properties of the game of Chess in question, such as the name of the tournament or match event, the player of the white pieces and the player of the black pieces, and the result of the game. It is possible to add self defined tag pairs. During our testing, we have added the name of the test with a \mintinline{text}{TestName} tag pair. Tag pairs always follow the structure

\begin{minted}{text}
	['Tag Name' "'Tag Value'"]
\end{minted}

where \mintinline{text}{'Tag Name'} holds the name of the tag (\mintinline{text}{Event}, \mintinline{text}{White}, \mintinline{text}{Black}, \mintinline{text}{Result}, ...) and \mintinline{text}{'Tag Value'} the value of the tag (\mintinline{text}{Third Rosenwald Trophy}, \mintinline{text}{Donald Byrne}, \mintinline{text}{Robert James Fischer}, \mintinline{text}{0-1}, ...).

The movetext encodes the actual game of Chess in question. It mainly consists of the moves made during the match, which are encode with

\begin{minted}{text}
	'Round Number'. 'White's Move' 'Black's Move'
\end{minted}

where each player's move is recorded in the Standard Algebraic Notation (SAN).

\subparagraph{Standard Algebraic Notation (SAN)}

Most moves are recorded with the SAN, which adheres to the following structure. \cite{SAN}

\begin{minted}{text}
	'Piece Identifier'x?'End Position'
\end{minted}

\mintinline{text}{'Piece Identifier'} denotes the piece being moved by a single capital letter: \mintinline{text}{B} (Bishop), \mintinline{text}{K} (King), \mintinline{text}{N} (Knight), \mintinline{text}{Q} (Queen) and \mintinline{text}{R} (Rook). Pawns either do not have an abbreviation or are denoted with \mintinline{text}{P}. In the case the abbreviation alone is not sufficient to identify the piece being moved, (partial) initial position information, according to the notation explained in figure \ref{F:SN}, is added to solve ambiguity. \mintinline{text}{'Piece Identifier'} is followed by \mintinline{text}{x} only if the move resulted in a capture. \mintinline{text}{'End Position'} records the final position of the moved piece using the notation explained in figure \ref{F:SN}. Additional notation exists to describe more specialized aspects of Chess moves. For example: \mintinline{text}{+} is appended to checking moves and \mintinline{text}{O-O} alone denotes King side Castling moves.

\begin{figure}
	\centering
	\newgame
	\showboard
	\caption{Square Notation: Each square is noted by its file (column letter) and rank (row number).} \label{F:SN}
\end{figure}

\paragraph{Testing Scope}

The SUT consists of a wrapper and the {\tt chesshs} library which the wrapper uses.

In order to transform the {\tt chesshs} package into a system with a command line interface, we have written a Haskell program which interacts with the library in such a way that the desired input-output behavior as described in \S\ref{P:ERSUT} of the SUT is achieved. In essence, it ``wraps" around the library to make it interactive. Hence the term `wrapper'.

Due to its strong influence on the behavior of the SUT, we will also be testing the wrapper. Nonetheless, testing will still focus mainly on the {\tt chesshs} package and its individual components, as this is the (sub)system of interest. Specifically, we will be testing

\begin{enumerate}
	\setcounter{enumi}{7}
	\item
		Unit, Integration. The scale and the number of sub-components of the project limits us to these types of tests.
	\item
		Mostly Black box testing, for the input/output of entire games in pgn notation.
		%TODO: Fix sentence
		The functions of the library will also be tested using a White box model.
	\item
		\begin{enumerate}[a)]
			\item
				For test generation of the black box testing, we will use a mixture of error guessing and the parsing of PGN files in a database.
				White box testing will be done through statement coverage and branch/condition coverage.
			\item
				Equivalence partitioning and boundary value analysis is hard to implement given our SUT, given that all possible chess games are nearly impossible to enumerate.
				Use case testing is also very hard to test, given that Chess.hs is a library, no user will likely every interact with the library directly.
				Path coverage and condition coverage are not implemented in Quickcheck, making it substantially harder to implement.
		\end{enumerate}
	\item the wrapper, \label{EI:W}
	\item PGN parsing, \label{EI:PGNP}
	\item move legality verification and \label{EI:MLV}
	\item checkmate position determination. \label{EI:CPD}
\end{enumerate}

This means that any rule of the game of Chess not implemented in the {\tt chesshs} library, such as timing, can and will not be tested.

The input-output specification in \S\ref{P:ERSUT} has effect on \ref{EI:W}, the PGN specification has effect on \ref{EI:PGNP} and the FIDE Chess rules have an effect on \ref{EI:MLV} and \ref{EI:CPD}. \cite{PGN, FIDE}

Because every system component will be tested both manually and automatically, no special distinctions with regards to the system elements and test methods are necessary. \\

Functionality is the most relevant quality characteristic of the tests. We would like to have a reasonable amount of certainty that the SUT behaves as specified. Reliability is also relevant to the SUT. Because it is a library, many other systems will depend on its continuous availability, even after a ``crash" of some sort. For the same reason, usability is also relevant to the SUT. Packages are components of a programming language that will be \emph{used} by other systems, hence it is paramount for the library to have an \emph{``usable"} interface. In short, many of the quality characteristics that apply to libraries, will also apply neatly to the SUT. \\

White box and gray box testing would look somewhat similar when testing the SUT, due to the fact that the system is a package. Gray box testing would only look at the components available to other programs, that is, the exported elements. White box testing would also look at every component of the library including their implementation. Gray box testing would consider every component as a black box, that is, it will only focus on testing if each component adheres to its related specification. White box testing would consider every component by its literal code. In conclusion, white box testing would focus on the actual implementation of the library's functionality and gray box testing would only focus on the library's exported component's functionality without considering any actual code.

\begin{figure}
	\centering
	\begin{tikzpicture}[
			every node/.style={draw,rectangle},
			title/.style={draw=none},
			->,>=open triangle 60]
		\node (bbtest) at (4,0)  {Black-box tester (\texttt{test.sh})};
		\node[matrix,row sep=.5em] (prog) at (0,0) {
			\node[title] {Wrapper};\\
			\node (chesshs) {Chess.hs};\\
		};
		\node[left=2em of chesshs] (wbtest) {White-box tester (\texttt{test.hs})};
		\node[below of=wbtest] (quickcheck) {Quickcheck};
		\draw (bbtest) -- (prog);
		\draw (wbtest) -- (chesshs);
		\draw (quickcheck) -- (wbtest);
	\end{tikzpicture}
	\caption{A graphical overview of the testing setup.\label{fig:testing-setup}}
\end{figure}

\begin{listing}
	\inputminted[fontsize=\footnotesize,firstline=13]{haskell}{../src/runchess.hs}
	\caption{The basic wrapper.\label{lst:wrapper}}
\end{listing}

\begin{thebibliography}{8}
	\bibitem[C]    {C}    \url{http://haskell.org/cabal}
	\bibitem[FIDE] {FIDE} \url{http://fide.com/fide/handbook.html}
	\bibitem[GHC]  {GHC}  \url{http://haskell.org/ghc}
	\bibitem[HP]   {HP}   \url{http://haskell.org/platform}
	\bibitem[PGN]  {PGN}  \url{http://saremba.de/chessgml/standards/pgn/pgn-complete.htm}
	\bibitem[SAN]  {SAN}  \url{http://cfajohnson.com/chess/SAN}
	\bibitem[SUT]  {SUT}  \url{http://hackage.haskell.org/package/chesshs}
\end{thebibliography}

\section{Test Environment}
\begin{enumerate}
	\setcounter{enumi}{10}
	\item 
		A wrapper written by us will be used to interact with the SUT.
		See \cref{lst:wrapper} for the implementation.
	\item
		Tests will be automatically performed using a combination of bash and Quickcheck.
	\item
		\begin{enumerate}[a)]
			\item
				The software environment GHC 8, Quickcheck 2.10, Linux Debian (derived).
				The hardware environment is a set of three amd64 laptops.
			\item
				See \cref{fig:testing-setup}.
		\end{enumerate}
	\item
		A Git repository, the results will be stored in individual files.
\end{enumerate}
\end{document}