Continue with text

2 files changed, 79 insertions, 43 deletions

diff --git a/assignments/assignment1.tex b/assignments/assignment1.tex
index 9be82e2..937d164 100644
--- a/assignments/assignment1.tex
+++ b/assignments/assignment1.tex
@@ -5,12 +5,15 @@
 \usepackage{enumerate}
 \usepackage[hidelinks]{hyperref}
 \usepackage{minted}
-\setminted{fontsize=\small,breaklines}
+\setminted{fontsize=\small,breaklines,tabsize=4}
 \usepackage{skak}
 \usepackage{tikz}
 \usetikzlibrary{arrows, matrix, positioning}
 \usepackage{cleveref}
 
+\let\oldurl\url
+\def\url#1{\small{\oldurl{#1}}}
+
 \title{Testing the \texttt{chesshs} Haskell Package}
 \subtitle{Test Plan, Manual Testing and Automated Testing}
 \author{Ren\'e den Hertog \and Camil Staps \and Erin van der Veen}
@@ -112,7 +115,7 @@ It mainly consists of the moves made during the match, which are encoded with
 
 where each player's move is recorded in the Standard Algebraic Notation (SAN).
 
-\subparagraph{Standard Algebraic Notation (SAN)}
+\paragraph{Standard Algebraic Notation (SAN)}
 
 Most moves are recorded with the SAN, which adheres to the following structure~\cite{SAN}:
 
@@ -142,8 +145,8 @@ For example, \texttt{+} is appended to checking moves and \texttt{O-O} alone den
 
 The SUT is the \texttt{chesshs} library.
 However, it is useful to have an actual program that can be run.
-To this end, we have written a small wrapper program that acts as a command-line interface to the library.
-The source code of the wrapper is shown in \cref{L:WI}.
+To this end, we have written a small wrapper program that acts as a command-line interface to the library.%
+	\footnote{The source code of the wrapper will be elaborated on later.}
 This program is kept as minimal as possible to contain (almost) no logic that might influence our tests.
 
 The \texttt{chesshs} package contains several subsystems.
@@ -175,16 +178,6 @@ We will be able to comment on the usability as we have written the small wrapper
 	--- that would require finding a reasonably large set of programmers using \texttt{chesshs},
 		and we suspect that may be hard.
 
-White box and grey box testing would look somewhat similar when testing the SUT, due to the fact that the system is a package.
-Grey box testing would only look at the components available to other programs, that is, the exported elements.
-For instance, we might look at the types of functions, but not their implementation.
-White box testing would also look at every component of the library including their implementation.
-Grey box testing would consider every component as a black box:
-	it would only focus on testing if each component adheres to its related specification.
-White box testing would consider every component by its literal code.
-In conclusion, white box testing would focus on the actual implementation of the library's functionality
-	and grey box testing would only focus on the library's exported component's functionality without considering any actual code.
-
 \subsection*{Test Method}
 
 Our tests are mainly at the unit level, and somewhat at the integration level.
@@ -193,44 +186,62 @@ Both are \enquote{atomic} enough to be tested via unit testing.
 The interaction between these two, mostly the usage of the library by the wrapper, will be tested via integration testing.
 Testing at the module, system and acceptance level is not reasonable in the case of our SUT, as its scale and number of subsystems limits us to lower testing levels.
 
-\paragraph{Test Generation Techniques}
-
-The majority of our test generation techniques are based upon or similar to black box methods. However, we also will apply some white box testing techniques.
-
-\subparagraph{Black Box Techniques}
+\subsubsection*{Test Generation Techniques}
 
-The focus of the black box testing is on the SUT in its entirety, that is, the \texttt{chesshs} package surrounded by the wrapper. The black box tester will try to find faults in the input-output behavior of the program as described in \S\ref{S:ERSUT}. In this type, test generation is based on error guessing and an examples database.
+The majority of our test generation techniques are based upon or similar to black box methods.
 
-Error guessing is based mostly on our collective experience of developing and testing software in the past. But also somewhat on our experience with the game of Chess and using other similar SUTs.
+\paragraph{Black Box Techniques}
 
-The examples database consists of a collection of text files describing a game of Chess in (possibly illegal) PGN (the `\texttt{.in}' files) and the expected output of the SUT (the `\texttt{.out}' files).
+The focus of the black box testing is on the SUT in its entirety, that is, the \texttt{chesshs} package surrounded by the wrapper.
+The black box tester will try to find faults in the input-output behaviour of the program.
+In this type, test generation is based on error guessing and an examples database.
 
-We will manually create test cases of this form, where we will be possibly introducing illegal PGN guided by error guessing.
+Error guessing is based mostly on our collective experience of developing and testing software in the past.
+But also somewhat on our experience with the game of Chess and using other similar SUTs.
 
-Sadly, equivalence partitioning and boundary value analysis are hard to apply to our SUT, given that the collection of all possible Chess games is nearly impossible to categorize. Use case testing is also tricky to apply. The wrapper is very limited in its interaction and is designed purely to make the \texttt{chesshs} package ``usable" for testing. Since the core of our SUT is a library, no \emph{end} user will likely interact with it directly. In summary, it does not feel reasonable to apply use case testing to the SUT.
+The examples database consists of a collection of text files describing a game of Chess in (possibly illegal) PGN (the \enquote{\texttt{.in}} files) and the expected output of the SUT (the \enquote{\texttt{.out}} files).
+To create these examples, we look at the FIDE rules in~\cite{FIDE} and the PGN specification~\cite{PGN}
+	for border cases, where we expect the library may deviate from the standard.
 
-\subparagraph{White Box Techniques}
+Sadly, equivalence partitioning and boundary value analysis are hard to apply to our SUT,
+	given that the collection of all possible Chess games is nearly impossible to categorise.
+Use case testing is also tricky to apply.
+The wrapper is intentionally very limited in its interaction and is designed purely to make the \texttt{chesshs} package \enquote{usable} for testing.
+Since the core of our SUT is a library, no end user will likely interact with it directly.
+In summary, it does not feel reasonable to apply use case testing to the SUT.
 
-The focus of the white box testing is on the \texttt{chesshs} package only, that is, the components of the library. The white box tester will try to find faults in the implementation of the package with regards to its specification.
+\paragraph{White Box Techniques}
 
-Sadly, path, branch and condition coverage are not included in the tools we will be using during the testing. Hence, it is substantially harder to implement these test generation techniques and will not be applied during our testing. \\
+White box and grey box testing would look somewhat similar when testing the SUT, due to the fact that the system is a package.
+Grey box testing would only look at the components available to other programs, that is, the exported elements.
+For instance, we might look at the types of functions, but not their implementation.
+White box testing would also look at every component of the library including their implementation.
+Grey box testing would consider every component as a black box:
+	it would only focus on testing if each component adheres to its related specification.
+White box testing would consider every component by its literal code.
+In conclusion, white box testing would focus on the actual implementation of the library's functionality
+	and grey box testing would only focus on the library's exported component's functionality without considering any actual code.
 
-As every software developer and tester knows, it is very hard, if not impossible, to completely test a given system. Noting that the risks surrounding our SUT are almost negligible (see \S\ref{S:ERSUT}), the pressure on the testing is not noticeable. However, we still feel the tests as described should give a more than sufficient scrutiny. The black box testing will likely find, if any, faults in the wrapper and the interaction with the \texttt{chesshs} package. The white box testing will likely find, if any, errors in the implementation of the library.
+Sadly, path, branch and condition coverage are not included in the tools we will be using during the testing.
+Hence, it is substantially harder to implement these test generation techniques and will not be applied during our testing.
 
 \subsection*{Test Environment}
 
-Since the SUT is in actuality the \texttt{chesshs} package, a user interface is clearly lacking. Thats why we have written a wrapper which makes the library interactive. See listing \ref{L:WI} for the full implementation. \\
+Since the SUT is in actuality the \texttt{chesshs} package, a user interface is clearly lacking. Thats why we have written a wrapper which makes the library interactive. See \cref{L:WI} for the full implementation.
 
 \begin{listing}
 	\inputminted{haskell}{../src/runchess.hs}
 	\caption{Wrapper Implementation} \label{L:WI}
 \end{listing}
 
-Black box tests will be automatically performed using shell scripting. In our case, we will be using bash and Linux compatible scripts. Test automation for white box testing is achieved by functionalities available in the to be used testing tool Quick Check. \cite{QC} \\
+Black box tests will be automatically performed using shell scripting.
+In our case, we will be using bash and Linux compatible scripts.
+Test automation for white box testing is achieved by functionalities available in the to be used testing tool QuickCheck~\cite{QC}.
 
-The software environments we will be using for testing our SUT are the operating system Linux, specifically Debian and Ubuntu, and the Haskell Platform \texttt{2014.2.0.0}, which includes, the Glasgow Haskell Compiler (GHC) \texttt{7.10.3}, Cabal \texttt{1.22.6.0} (\texttt{1.22.5.0} for the library) and Quick Check \texttt{2.8.1}. \cite{HP, GHC, C, QC} The hardware platforms are three differently powerful laptops, notably Acer and Toshiba.
+The software environments we will be using for testing our SUT are the operating system Linux, specifically Debian and Ubuntu, and the Haskell Platform \texttt{2014.2.0.0}, which includes the Glasgow Haskell Compiler (GHC) \texttt{7.10.3}, Cabal \texttt{1.22.6.0} (\texttt{1.22.5.0} for the library) and QuickCheck \texttt{2.8.1}~\cite{HP, GHC, C, QC}.
+The hardware platforms are three differently powerful laptops, notably Acer and Toshiba.
 
-The testing architecture is best described by the graphic shown in figure \ref{F:TAGO}.
+The testing architecture is best described by the graphic shown in \cref{F:TAGO}.
 
 \begin{figure}
 	\centering
@@ -244,7 +255,7 @@ The testing architecture is best described by the graphic shown in figure \ref{F
 			\node (chesshs) {\texttt{chesshs}};\\
 		};
 		\node[left=2em of chesshs] (wbtest) {White Box Tester (\texttt{test.hs})};
-		\node[below of=wbtest] (quickcheck) {Quick Check};
+		\node[below of=wbtest] (quickcheck) {QuickCheck};
 		\draw (bbtest) -- (prog);
 		\draw (wbtest) -- (chesshs);
 		\draw (quickcheck) -- (wbtest);
@@ -252,38 +263,63 @@ The testing architecture is best described by the graphic shown in figure \ref{F
 	\caption{Testing Architecture Graphical Overview} \label{F:TAGO}
 \end{figure}
 
-No stubs nor drivers are necessary for executing the testing architecture. \\
+No stubs nor drivers are necessary for executing the testing architecture.
 
 Most, if not all, documentation surrounding the tests, test results and test environment will be included in this document. It is possible that some tests and test results will contain information not stated here, such as the name of a test case and the possible flaw that it (tries to) detect(s). However, any external information will easily be found in the appropriate files.
 
-The tests will either be stored in the examples database (directory) or will be part of the Quick Check unit test module(s) (source file). The test results will be stored, if necessary, in the same files as their related test or in another easy to locate file. The test environment will be stored, as every other testing component, in a Git repository found at \url{https://gitlab.science.ru.nl/eveen/Testing-Techniques.git}.
+The tests will either be stored in the examples database (directory) or will be part of the QuickCheck unit test module(s) (source file). The test results will be stored, if necessary, in the same files as their related test or in another easy to locate file. The test environment will be stored, as every other testing component, in a Git repository found at \url{https://gitlab.science.ru.nl/eveen/Testing-Techniques.git}.
 
-Since Haskell, as most functional programming languages, is ``stable" in terms of side effects and randomness, reproductivity should no be an issue. Nevertheless, we will make sure every test case can be reproduced, that is, run again leading to the same results as is documented.
+Since Haskell, as most functional programming languages, is \enquote{stable} in terms of side effects and randomness, reproductivity should no be an issue. Nevertheless, we will make sure every test case can be reproduced, that is, run again leading to the same results as is documented.
 
 \section{Manual Testing}
 
-Since the SUT is an Application Programming Interface all tests are constructed not as an sequence of interactions, but rather as something that can be fed to the wrapper.
-In turn, the wrapper uses the API of \texttt{chesshs} to construct the position sequence.
+Our tests are stored in the form of input files that are fed to the wrapper.
+In turn, the wrapper uses the \texttt{chesshs} API to construct the position sequence.
 This position sequence is then compared to a user defined expected output.
 
 In essence, this implies that the Manual Tests and Automated Tests are the same.
 
-If the wrapper were extended to support interactivity, manual testing would have a more prominent place.
+If the wrapper were extended to support interactivity,
+	manual testing would have a more prominent place.
+
+The concrete tests we have used are shown in \cref{T:T}.
+
+\begin{table}
+	\centering
+	\begin{tabular}{r l l}
+		Test & Description & Expected result\\\hline
+		 0 & A normal legal game & Accepted\\
+		 1 & Double castling on kingside & \texttt{InvalidMove}\\
+		 2 & White moving a black piece & \texttt{InvalidMove}\\
+		 3 & Castling on both queen and kingside & \texttt{InvalidMove}\\
+		 4 & Taking \emph{en passent} on second move & \texttt{InvalidMove}\\
+		 5 & Illegal mate annotation & \texttt{InvalidMove}\\
+		 6 & Castling through check & \texttt{InvalidMove}\\
+		 7 & Castling in check & \texttt{InvalidMove}\\
+		 8 & Mixed PGN comment notations & Accepted\\
+		 9 & Continuing after insufficient material draw & \texttt{InvalidMove}\\
+		10 & Two games in a single file & The outputs for both games\\
+	\end{tabular}
+	\caption{Descriptions of the manual tests.\label{T:T}}
+\end{table}
 
 \section{Automated Testing}
 
 The input of the automated tests are stored in individual files.
 These files contain one or more PGN games that are fed to the wrapper.
-The wrapper then passes these PGN games to the chesshs library, and outputs the board state after every move.
+The wrapper then parses these strings using the chesshs library,
+	applies the moves one by one
+	and outputs the board state after every move.
+
 \begin{listing}
 	\inputminted{bash}{../test/test.sh}
 	\caption{Automated Test script\label{L:ATS}}
 \end{listing}
 
 A small bash script, shown in \cref{L:ATS}, is used
-	to compare the output that is produced by the wrapper with a user defined expected output.
+	to run all tests and to compare the output that is produced by the wrapper with a user defined expected output.
 If there is no difference between the two, the test is considered \enquote{passed}.
-If there is a difference, this is output by the script.
+If there is a difference, the script shows it.
 
 \begin{thebibliography}{8}
 	\bibitem[C]    {C}    \url{http://haskell.org/cabal}
diff --git a/test/test.sh b/test/test.sh
index 17728aa..e050aaa 100755
--- a/test/test.sh
+++ b/test/test.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 RED='\033[0;31m'
 GREEN='\033[0;32m'
-RESET='\033[0;30m'
+RESET='\033[0m'
 
 for f in database/*.in;
 do