Test results

2 files changed, 114 insertions, 14 deletions

diff --git a/assignments/assignment1-test-run.txt b/assignments/assignment1-test-run.txt
new file mode 100644
index 0000000..06c460f
--- /dev/null
+++ b/assignments/assignment1-test-run.txt
@@ -0,0 +1,51 @@
+Test: "Normal Legal Game" passed
+##################################################
+Test: "Two Games, single file" passed
+##################################################
+Test: "Double Castling on Kingside" passed
+##################################################
+Test: "White Moving Black piece" failed
+InvalidMove	      |	NoParse
+##################################################
+Test: "Castling both Queen and Kingside" passed
+##################################################
+Test: "Taking En Passent on second move" failed
+InvalidMove	      |	NoParse
+##################################################
+Test: "Mate when impossible" failed
+InvalidMove	      |	rnbqkbnr
+		      >	pppppppp
+		      >	........
+		      >	........
+		      >	....P...
+		      >	........
+		      >	PPPP.PPP
+		      >	RNBQKBNR
+		      >
+##################################################
+Test: "Castling Through Check" passed
+##################################################
+Test: "Castling In Check" failed
+InvalidMove	      |	rnbqk.nr
+		      >	pppp...p
+		      >	....ppp.
+		      >	........
+		      >	.b......
+		      >	...P.NP.
+		      >	PPP.PPBP
+		      >	RNBQ.RK.
+		      >
+##################################################
+Test: ";-comment in {}-comment" passed
+##################################################
+Test: "Insufficient Material Draw" failed
+InvalidMove	      |	........
+		      >	.....k..
+		      >	...b....
+		      >	........
+		      >	......K.
+		      >	........
+		      >	........
+		      >	........
+		      >
+##################################################
diff --git a/assignments/assignment1.tex b/assignments/assignment1.tex
index 937d164..6275e9b 100644
--- a/assignments/assignment1.tex
+++ b/assignments/assignment1.tex
@@ -12,7 +12,7 @@
 \usepackage{cleveref}
 
 \let\oldurl\url
-\def\url#1{\small{\oldurl{#1}}}
+\def\url#1{{\small\oldurl{#1}}}
 
 \title{Testing the \texttt{chesshs} Haskell Package}
 \subtitle{Test Plan, Manual Testing and Automated Testing}
@@ -269,19 +269,32 @@ Most, if not all, documentation surrounding the tests, test results and test env
 
 The tests will either be stored in the examples database (directory) or will be part of the QuickCheck unit test module(s) (source file). The test results will be stored, if necessary, in the same files as their related test or in another easy to locate file. The test environment will be stored, as every other testing component, in a Git repository found at \url{https://gitlab.science.ru.nl/eveen/Testing-Techniques.git}.
 
-Since Haskell, as most functional programming languages, is \enquote{stable} in terms of side effects and randomness, reproductivity should no be an issue. Nevertheless, we will make sure every test case can be reproduced, that is, run again leading to the same results as is documented.
+Since Haskell is a pure functional language and we have fixed version numbers,
+	the tests are by construction reproducible.
 
 \section{Manual Testing}
 
-Our tests are stored in the form of input files that are fed to the wrapper.
+Our tests are stored in the form of input files that are fed to the wrapper.%
+	\footnote{These are just PGN files, an example of which was already shown in \cref{L:PGNE}.}
 In turn, the wrapper uses the \texttt{chesshs} API to construct the position sequence.
-This position sequence is then compared to a user defined expected output.
+This position sequence is then compared to a user defined expected output, an example of which is shown in \cref{L:OUTE}.
+The first two lines contain metadata, after which a list of board configurations is printed,
+	either until the match is finished or an invalid move is detected.
 
-In essence, this implies that the Manual Tests and Automated Tests are the same.
+\begin{listing}
+	\inputminted{text}{../test/database/m2.out}
+	\caption{An example output file.\label{L:OUTE}}
+\end{listing}
 
+Because we do not test an interactive program,
+	our manual tests are in a sense already automated.
 If the wrapper were extended to support interactivity,
 	manual testing would have a more prominent place.
 
+\subsection*{Database}
+We have created 11 test cases for both valid and invalid games.
+As described above, the invalid games are derived from the FIDE rules
+	in that they attempt to find borderline cases where the library may deviate from those rules~\cite{FIDE}.
 The concrete tests we have used are shown in \cref{T:T}.
 
 \begin{table}
@@ -305,25 +318,61 @@ The concrete tests we have used are shown in \cref{T:T}.
 
 \section{Automated Testing}
 
-The input of the automated tests are stored in individual files.
-These files contain one or more PGN games that are fed to the wrapper.
-The wrapper then parses these strings using the chesshs library,
-	applies the moves one by one
-	and outputs the board state after every move.
+To automate our manual tests, we use a small bash script (\cref{L:ATS}).
+This script runs all tests and compares the output that is produced by the wrapper with a user defined expected output.
+If there is no difference between the two, the test is considered \enquote{passed}.
+If there is a difference, the test is considered \enquote{failed} and the script shows the difference.
 
 \begin{listing}
 	\inputminted{bash}{../test/test.sh}
 	\caption{Automated Test script\label{L:ATS}}
 \end{listing}
 
-A small bash script, shown in \cref{L:ATS}, is used
-	to run all tests and to compare the output that is produced by the wrapper with a user defined expected output.
-If there is no difference between the two, the test is considered \enquote{passed}.
-If there is a difference, the script shows it.
+\begin{listing}
+	\inputminted[fontsize=\footnotesize]{text}{assignment1-test-run.txt}
+	\caption{Results of the test run.\label{L:TR}}
+\end{listing}
+
+The output of the test run on the tests in \cref{T:T} is given in \cref{L:TR}.
+We were able to find several failures:
+
+\begin{description}
+	\item[White moving black piece]
+		This test had the white player moving a black piece.
+		We would expect that this causes an \texttt{InvalidMove}.
+		Instead, \texttt{chesshs} gives a \texttt{NoParse} error.
+		That is odd, because the PGN file is valid according to the specification~\cite{PGN}.
+
+	\item[Taking En Passent on second move]
+		This test has the white player taking a black piece \emph{en passent}.%
+			\footnote{In this move, a pawn that moved two ranks forward is captured by an enemy pawn on the field one rank before the destination field.}
+		This special pawn capture is only allowed in the move directly after the target pawn moved.
+		The test attempts to capture the pawn one round later.
+		Like the test above, we would expect an \texttt{InvalidMove} error here,
+			but \texttt{chesshs} gives a \texttt{NoParse} error.
+
+	\item[Mate when impossible]
+		This test includes the \texttt{\#} annotation in the PGN file to annotate a checkmate.
+		However, it includes it in the wrong position (after a move that does not produce checkmate).
+		The library ignores the \texttt{\#} annotation,
+			while it is not a valid move.
+
+	\item[Castling in check]
+		In this test, the white player attempts to get out of a check configuration by castling,
+			which is not allowed~\cite[3.8.2.2.2]{FIDERULES}.
+		The library does not take this into account, and allows the castling.
+
+	\item[Insufficient material draw]
+		This test contains a game that ends with so few pieces that a checkmate position is impossible
+			(one bishop, besides the kings).
+		In such a situation, the game is drawn~\cite[1.5]{FIDERULES}.
+		However, \texttt{chesshs} allows further moves.
+\end{description}
 
 \begin{thebibliography}{8}
 	\bibitem[C]    {C}    \url{http://haskell.org/cabal}
 	\bibitem[FIDE] {FIDE} \url{http://fide.com/fide/handbook.html}
+	\bibitem[FIDERULES] {FIDERULES} \url{http://www.fide.com/fide/handbook.html?id=207&view=article}
 	\bibitem[GHC]  {GHC}  \url{http://haskell.org/ghc}
 	\bibitem[HP]   {HP}   \url{http://haskell.org/platform}
 	\bibitem[PGN]  {PGN}  \url{http://saremba.de/chessgml/standards/pgn/pgn-complete.htm}