diff options
| -rw-r--r-- | assignments/assignment1.tex | 140 |
1 files changed, 93 insertions, 47 deletions
diff --git a/assignments/assignment1.tex b/assignments/assignment1.tex index ebe7f8a..06b160d 100644 --- a/assignments/assignment1.tex +++ b/assignments/assignment1.tex @@ -1,21 +1,21 @@ -\documentclass{scrartcl} +\documentclass[british]{scrartcl} -\usepackage{cleveref} +\usepackage[british]{babel} +\usepackage{csquotes} \usepackage{enumerate} \usepackage[hidelinks]{hyperref} \usepackage{minted} \usepackage{skak} \usepackage{tikz} - \usetikzlibrary{arrows, matrix, positioning} +\usetikzlibrary{arrows, matrix, positioning} +\usepackage{cleveref} -\title{The {\tt chesshs} Haskell Package} +\title{The \texttt{chesshs} Haskell Package} \subtitle{Test Plan, Manual Testing \& Automated Testing} \author{ - \begin{tabular}{r l} - Ren\'e den Hertog & {\tt 1007823} \\ - Camil Staps & {\tt 4498062} \\ - Erin van der Veen & {\tt 4431200} - \end{tabular} + Ren\'e den Hertog\footnote{s1007823} \and + Camil Staps\footnote{s4498062} \and + Erin van der Veen\footnote{s4431200} } % Remove the date from '\maketitle'. \makeatletter @@ -35,58 +35,104 @@ \subsection*{System Under Test (SUT) Description} -The chosen SUT is the {\tt chesshs} Haskell package. \cite{SUT} It should be able to run in any software environment and hardware platform which supports Haskell. The latest version of the SUT, since the 13\textsuperscript{th} of March 2015, is {\tt 0.2.1}. - -\paragraph{Executing/Running the SUT} \label{P:ERSUT} - -It is required to be able to compile Haskell, typically through the Glasgow Haskell Compiler (GHC), and having the {\tt chesshs} package installed, normally through Cabal via the \mintinline{bash}{cabal install chesshs} command. \cite{GHC, C} Both the GHC and Cabal are part of the Haskell Platform. \cite{HP} Under our computer systems, compiling and starting the SUT is done with the \mintinline{bash}{make} and \mintinline{bash}{./runchess} commands, respectively. - -The SUT is used by providing it with, via the standard input, a text file containing a game of Chess in the Portable Game Notation (PGN). \cite{PGN} After computation, via the standard output, some general information about the Chess match is displayed, including the end result. Followed by the state of the board after each move. In any other case, an error message is displayed. \\ - -As is common with everything software, the SUT is likely to contain faults. However, the risks surrounding the program are near zero due to the very low impact of the program faulting. As of our expectation, the most ``risky" scenario for the SUT is at a notable Chess competition. In the case the program does not behave as intended, it is almost certain that a human referee would take over its place. Furthermore, we expect such a system to be used as a supporting tool for match officials and not as a replacement for regulation. +The chosen SUT is the \texttt{chesshs} Haskell package. +\cite{SUT} It should be able to run in any software environment and hardware platform which supports Haskell. +The latest version of the SUT, since the 13\textsuperscript{th} of March 2015, is \texttt{0.2.1}. + +\subsubsection*{Executing/Running the SUT} + +It is required to be able to compile Haskell, + typically through the Glasgow Haskell Compiler (GHC), + and having the \texttt{chesshs} package installed, + normally through Cabal via the \mintinline{bash}{cabal install chesshs} command~\cite{GHC, C}. +On our systems, + compiling and starting the SUT is done with the \mintinline{bash}{make} and \mintinline{bash}{./runchess} commands, respectively. + +The SUT is used by providing it with standard input containing a game of Chess in Portable Game Notation~\cite{PGN}. +After computation, + some general information about the Chess match is given on standard output, + including the end result, + followed by the state of the board after each move. +In case of a parse error or an invalid move, an error message is displayed. + +As is common with all software, the SUT is likely to contain faults. +However, few risks are associated with the program, due to the very low impact of the program faulting. +As for our expectation, the most \enquote{risky} scenario for the SUT is at a notable Chess competition. +In the case the program does not behave as intended, + it is almost certain that a human referee would take over its place. +Furthermore, we expect such a system to be used as a supporting tool for match officials + and not as a replacement for regulation. \subsection*{Test Goals} -\paragraph{Specification} +\subsubsection*{Specification} -The specification of the SUT is for the most part similar to the input-output behavior described in \S\ref{P:ERSUT}. Because the SUT is strongly related to Chess, many finer details of its specification can be derived from documentation of the world of Chess. +The specification of the SUT is for the most part similar to the input-output behaviour described above. +Because the SUT is strongly related to Chess, + many finer details of its specification can be derived from documentation of the world of Chess. -\subparagraph{F\'ed\'eration Internationale Des Echecs (FIDE)} Due to the popularity of Chess, most people are familiar with how it is played. Hence, a summary of the rules would be excesive. However, due to the competitve nature of the game, the World Chess Federation describes the rules painfully precise. \cite{FIDE} +\paragraph{F\'ed\'eration Internationale Des Echecs (FIDE)} +Due to the popularity of Chess, most people are familiar with how it is played. +Hence, a summary of the rules would be excesive. +However, due to the competitve nature of the game, + the World Chess Federation describes the rules painfully precise~\cite{FIDE}. -\subparagraph{Portable Game Notation (PGN)} Many Chess matches are recorded with the PGN. \cite{PGN} As some of the test cases consist of text files describing Chess games in PGN, one of our test cases is included in listing \ref{L:PGNE} as an example for the notation. +\paragraph{Portable Game Notation (PGN)} +Many Chess matches are recorded with the PGN~\cite{PGN}. +As some of the test cases consist of text files describing Chess games in PGN, + one of our test cases is included in \cref{L:PGNE} as an example for the notation. \begin{listing} \inputminted{text}{../test/database/m0.in} - \caption{Portable Game Notation (PGN) Example} \label{L:PGNE} + \caption{Portable Game Notation (PGN) Example\label{L:PGNE}} \end{listing} -PGN consists of two parts: first, the tag pairs and, second, the movetext. +PGN consists of two parts: the tag pairs and the movetext. -The tag pairs encode properties of the game of Chess in question, such as the name of the tournament or match event, the player of the white pieces and the player of the black pieces, and the result of the game. It is possible to add self defined tag pairs. During our testing, we have added the name of the test with a \mintinline{text}{TestName} tag pair. Tag pairs always follow the structure +The tag pairs encode properties of the game of Chess in question, + such as the name of the tournament or match event, + the player of the white pieces and the player of the black pieces, + and the result of the game. +It is possible to add self defined tag pairs. +During our testing, we have added the name of the test with a \texttt{TestName} tag pair. +Tag pairs always follow the structure \begin{minted}{text} - ['Tag Name' "'Tag Value'"] + [<Tag Name> "<Tag Value>"] \end{minted} -where \mintinline{text}{'Tag Name'} holds the name of the tag (\mintinline{text}{Event}, \mintinline{text}{White}, \mintinline{text}{Black}, \mintinline{text}{Result}, ...) and \mintinline{text}{'Tag Value'} the value of the tag (\mintinline{text}{Third Rosenwald Trophy}, \mintinline{text}{Donald Byrne}, \mintinline{text}{Robert James Fischer}, \mintinline{text}{0-1}, ...). +where \texttt{'Tag Name'} holds the name of the tag + (\texttt{Event}, \texttt{White}, \texttt{Black}, \texttt{Result}, ...) + and \texttt{'Tag Value'} the value of the tag + (\texttt{Third Rosenwald Trophy}, \texttt{Donald Byrne}, \texttt{Robert James Fischer}, \texttt{0-1}, ...). -The movetext encodes the actual game of Chess in question. It mainly consists of the moves made during the match, which are encode with +The movetext encodes the actual game of Chess in question. +It mainly consists of the moves made during the match, which are encoded with \begin{minted}{text} - 'Round Number'. 'White's Move' 'Black's Move' + <Round Number>. <White's Move> <Black's Move> \end{minted} where each player's move is recorded in the Standard Algebraic Notation (SAN). \subparagraph{Standard Algebraic Notation (SAN)} -Most moves are recorded with the SAN, which adheres to the following structure. \cite{SAN} +Most moves are recorded with the SAN, which adheres to the following structure~\cite{SAN}: \begin{minted}{text} - 'Piece Identifier'x?'End Position' + <Piece Identifier>[x]<End Position> \end{minted} -\mintinline{text}{'Piece Identifier'} denotes the piece being moved by a single capital letter: \mintinline{text}{B} (Bishop), \mintinline{text}{K} (King), \mintinline{text}{N} (Knight), \mintinline{text}{Q} (Queen) and \mintinline{text}{R} (Rook). Pawns either do not have an abbreviation or are denoted with \mintinline{text}{P}. In the case the abbreviation alone is not sufficient to identify the piece being moved, (partial) initial position information, according to the notation explained in figure \ref{F:SN}, is added to solve ambiguity. \mintinline{text}{'Piece Identifier'} is followed by \mintinline{text}{x} only if the move resulted in a capture. \mintinline{text}{'End Position'} records the final position of the moved piece using the notation explained in figure \ref{F:SN}. Additional notation exists to describe more specialized aspects of Chess moves. For example: \mintinline{text}{+} is appended to checking moves and \mintinline{text}{O-O} alone denotes King side Castling moves. +\texttt{Piece Identifier} denotes the piece being moved by a single capital letter: + \texttt{B} (Bishop), \texttt{K} (King), \texttt{N} (Knight), \texttt{Q} (Queen) and \texttt{R} (Rook). +Pawns either do not have an abbreviation or are denoted with \texttt{P}. +In the case the abbreviation alone is not sufficient to identify the piece being moved, + (partial) initial position information is added to resolve ambiguity, + according to the notation explained in \cref{F:SN}. +\texttt{Piece Identifier} is followed by \texttt{x} only if the move resulted in a capture. +\texttt{End Position} records the final position of the moved piece using the notation explained in figure \ref{F:SN}. +Additional notation exists to describe more specialized aspects of Chess moves. +For example, \texttt{+} is appended to checking moves and \texttt{O-O} alone denotes king side castling moves. \begin{figure} \centering @@ -97,11 +143,11 @@ Most moves are recorded with the SAN, which adheres to the following structure. \paragraph{Testing Scope} -The SUT consists of a wrapper and the {\tt chesshs} library which the wrapper uses. +The SUT consists of a wrapper and the \texttt{chesshs} library which the wrapper uses. -In order to transform the {\tt chesshs} package into a system with a command line interface, we have written a Haskell program which interacts with the library in such a way that the desired input-output behavior as described in \S\ref{P:ERSUT} of the SUT is achieved. In essence, it ``wraps" around the library to make it interactive. Hence the term `wrapper'. +In order to transform the \texttt{chesshs} package into a system with a command line interface, we have written a Haskell program which interacts with the library in such a way that the desired input-output behavior as described in \S\ref{P:ERSUT} of the SUT is achieved. In essence, it ``wraps" around the library to make it interactive. Hence the term `wrapper'. -Due to its strong influence on the behavior of the SUT, we will also be testing the wrapper. Nonetheless, testing will still focus mainly on the {\tt chesshs} package and its individual components, as this is the (sub)system of interest. Specifically, we will be testing +Due to its strong influence on the behavior of the SUT, we will also be testing the wrapper. Nonetheless, testing will still focus mainly on the \texttt{chesshs} package and its individual components, as this is the (sub)system of interest. Specifically, we will be testing \begin{enumerate} \item the wrapper, \label{EI:W} @@ -110,7 +156,7 @@ Due to its strong influence on the behavior of the SUT, we will also be testing \item checkmate position determination. \label{EI:CPD} \end{enumerate} -This means that any rule of the game of Chess not implemented in the {\tt chesshs} library, such as timing, can and will not be tested. +This means that any rule of the game of Chess not implemented in the \texttt{chesshs} library, such as timing, can and will not be tested. The input-output specification in \S\ref{P:ERSUT} has effect on \ref{EI:W}, the PGN specification has effect on \ref{EI:PGNP} and the FIDE Chess rules have an effect on \ref{EI:MLV} and \ref{EI:CPD}. \cite{PGN, FIDE} @@ -122,7 +168,7 @@ White box and gray box testing would look somewhat similar when testing the SUT, \subsection*{Test Method} -The testing level is mainly at the unit level, and somewhat at the integration level. The SUT only consists of two components: the wrapper and the {\tt chesshs} package. Both are ``atomic" enough to be tested via unit testing. The interaction between these two, mostly the usage of the library by the wrapper, will be tested via integration testing. Testing at the module, system and acceptance level is not reasonable in the case of our SUT, as its scale and number of subsystems limits us to lower testing levels. +The testing level is mainly at the unit level, and somewhat at the integration level. The SUT only consists of two components: the wrapper and the \texttt{chesshs} package. Both are ``atomic" enough to be tested via unit testing. The interaction between these two, mostly the usage of the library by the wrapper, will be tested via integration testing. Testing at the module, system and acceptance level is not reasonable in the case of our SUT, as its scale and number of subsystems limits us to lower testing levels. \paragraph{Test Generation Techniques} @@ -130,19 +176,19 @@ The majority of our test generation techniques are based upon or similar to blac \subparagraph{Black Box Techniques} -The focus of the black box testing is on the SUT in its entirety, that is, the {\tt chesshs} package surrounded by the wrapper. The black box tester will try to find faults in the input-output behavior of the program as described in \S\ref{P:ERSUT}. In this type, test generation is based on error guessing and an examples database. +The focus of the black box testing is on the SUT in its entirety, that is, the \texttt{chesshs} package surrounded by the wrapper. The black box tester will try to find faults in the input-output behavior of the program as described in \S\ref{P:ERSUT}. In this type, test generation is based on error guessing and an examples database. Error guessing is based mostly on our collective experience of developing and testing software in the past. But also somewhat on our experience with the game of Chess and using other similar SUTs. -The examples database consists of a collection of text files describing a game of Chess in (possibly illegal) PGN (the `{\tt .in}' files) and the expected output of the SUT (the `{\tt .out}' files). +The examples database consists of a collection of text files describing a game of Chess in (possibly illegal) PGN (the `\texttt{.in}' files) and the expected output of the SUT (the `\texttt{.out}' files). We will manually create test cases of this form, where we will be possibly introducing illegal PGN guided by error guessing. -Sadly, equivalence partitioning and boundary value analysis are hard to apply to our SUT, given that the collection of all possible Chess games is nearly impossible to categorize. Use case testing is also tricky to apply. The wrapper is very limited in its interaction and is designed purely to make the {\tt chesshs} package ``usable" for testing. Since the core of our SUT is a library, no \emph{end} user will likely interact with it directly. In summary, it does not feel reasonable to apply use case testing to the SUT. +Sadly, equivalence partitioning and boundary value analysis are hard to apply to our SUT, given that the collection of all possible Chess games is nearly impossible to categorize. Use case testing is also tricky to apply. The wrapper is very limited in its interaction and is designed purely to make the \texttt{chesshs} package ``usable" for testing. Since the core of our SUT is a library, no \emph{end} user will likely interact with it directly. In summary, it does not feel reasonable to apply use case testing to the SUT. \subparagraph{White Box Techniques} -The focus of the white box testing is on the {\tt chesshs} package only, that is, the components of the library. The white box tester will try to find faults in the implementation of the package with regards to its specification. In this type, test generation is based on statement coverage and manual path based coverage. +The focus of the white box testing is on the \texttt{chesshs} package only, that is, the components of the library. The white box tester will try to find faults in the implementation of the package with regards to its specification. In this type, test generation is based on statement coverage and manual path based coverage. Statement coverage, in Haskell's case, marks the lines of code that have been executed at least once. @@ -152,11 +198,11 @@ We will continuously adapt the white box tester until it reaches a sufficient le Sadly, path, branch and condition coverage are not included in the tools we will be using during the testing. Hence, it is substantially harder to implement these test generation techniques and will not be applied during our testing. \\ -As every software developer and tester knows, it is very hard, if not impossible, to completely test a given system. Noting that the risks surrounding our SUT are almost negligible (see \S\ref{P:ERSUT}), the pressure on the testing is not noticeable. However, we still feel the tests as described should give a more than sufficient scrutiny. The black box testing will likely find, if any, faults in the wrapper and the interaction with the {\tt chesshs} package. The white box testing will likely find, if any, errors in the implementation of the library. +As every software developer and tester knows, it is very hard, if not impossible, to completely test a given system. Noting that the risks surrounding our SUT are almost negligible (see \S\ref{P:ERSUT}), the pressure on the testing is not noticeable. However, we still feel the tests as described should give a more than sufficient scrutiny. The black box testing will likely find, if any, faults in the wrapper and the interaction with the \texttt{chesshs} package. The white box testing will likely find, if any, errors in the implementation of the library. \subsection*{Test Environment} -Since the SUT is in actuality the {\tt chesshs} package, a user interface is clearly lacking. Thats why we have written a wrapper which makes the library interactive. See listing \ref{L:WI} for the full implementation. \\ +Since the SUT is in actuality the \texttt{chesshs} package, a user interface is clearly lacking. Thats why we have written a wrapper which makes the library interactive. See listing \ref{L:WI} for the full implementation. \\ \begin{listing} \inputminted[fontsize=\footnotesize]{haskell}{../src/runchess.hs} @@ -165,7 +211,7 @@ Since the SUT is in actuality the {\tt chesshs} package, a user interface is cle Black box tests will be automatically performed using shell scripting. In our case, we will be using bash and Linux compatible scripts. Test automation for white box testing is achieved by functionalities available in the to be used testing tool Quick Check. \cite{QC} \\ -The software environments we will be using for testing our SUT are the operating system Linux, specifically Debian and Ubuntu, and the Haskell Platform {\tt 2014.2.0.0}, which includes, the Glasgow Haskell Compiler (GHC) {\tt 7.10.3}, Cabal {\tt 1.22.6.0} ({\tt 1.22.5.0} for the library) and Quick Check {\tt 2.8.1}. \cite{HP, GHC, C, QC} The hardware platforms are three differently powerful laptops, notably Acer and Toshiba. +The software environments we will be using for testing our SUT are the operating system Linux, specifically Debian and Ubuntu, and the Haskell Platform \texttt{2014.2.0.0}, which includes, the Glasgow Haskell Compiler (GHC) \texttt{7.10.3}, Cabal \texttt{1.22.6.0} (\texttt{1.22.5.0} for the library) and Quick Check \texttt{2.8.1}. \cite{HP, GHC, C, QC} The hardware platforms are three differently powerful laptops, notably Acer and Toshiba. The testing architecture is best described by the graphic shown in figure \ref{F:TAGO}. @@ -175,12 +221,12 @@ The testing architecture is best described by the graphic shown in figure \ref{F every node/.style={draw,rectangle}, title/.style={draw=none}, ->,>=open triangle 60] - \node (bbtest) at (4,0) {Black Box Tester ({\tt test.sh})}; + \node (bbtest) at (4,0) {Black Box Tester (\texttt{test.sh})}; \node[matrix,row sep=.5em] (prog) at (0,0) { \node[title] {Wrapper};\\ - \node (chesshs) {{\tt chesshs}};\\ + \node (chesshs) {\texttt{chesshs}};\\ }; - \node[left=2em of chesshs] (wbtest) {White Box Tester ({\tt test.hs})}; + \node[left=2em of chesshs] (wbtest) {White Box Tester (\texttt{test.hs})}; \node[below of=wbtest] (quickcheck) {Quick Check}; \draw (bbtest) -- (prog); \draw (wbtest) -- (chesshs); |