\documentclass{scrartcl} \usepackage{cleveref} \usepackage{enumerate} \usepackage[hidelinks]{hyperref} \usepackage{minted} \usepackage{skak} \usepackage{tikz} \usetikzlibrary{arrows, matrix, positioning} \title{The {\tt chesshs} Haskell Package} \subtitle{Test Plan, Manual Testing \& Automated Testing} \author{ \begin{tabular}{r l} Ren\'e den Hertog & {\tt 1007823} \\ Camil Staps & {\tt 4498062} \\ Erin van der Veen & {\tt 4431200} \end{tabular} } % Remove the date from '\maketitle'. \makeatletter \patchcmd {\@maketitle} {{\usekomafont{date}{\@date\par}}\vskip\z@\@plus 1em} {} {} {} \makeatother \begin{document} \maketitle \section{Test Plan} \subsection*{System Under Test (SUT) Description} The chosen SUT is the {\tt chesshs} Haskell package. \cite{SUT} It should be able to run in any software environment and hardware platform which supports Haskell. The latest version of the SUT, since the 13\textsuperscript{th} of March 2015, is {\tt 0.2.1}. \paragraph{Executing/Running the SUT} \label{P:ERSUT} It is required to be able to compile Haskell, typically through the Glasgow Haskell Compiler (GHC), and having the {\tt chesshs} package installed, normally through Cabal via the \mintinline{bash}{cabal install chesshs} command. \cite{GHC, C} Both the GHC and Cabal are part of the Haskell Platform. \cite{HP} Under our computer systems, compiling and starting the SUT is done with the \mintinline{bash}{make} and \mintinline{bash}{./runchess} commands, respectively. The SUT is used by providing it with, via the standard input, a text file containing a game of Chess in the Portable Game Notation (PGN). \cite{PGN} After computation, via the standard output, some general information about the Chess match is displayed, including the end result. Followed by the state of the board after each move. In any other case, an error message is displayed. \\ As is common with everything software, the SUT is likely to contain faults. However, the risks surrounding the program are near zero due to the very low impact of the program faulting. As of our expectation, the most ``risky" scenario for the SUT is at a notable Chess competition. In the case the program does not behave as intended, it is almost certain that a human referee would take over its place. Furthermore, we expect such a system to be used as a supporting tool for match officials and not as a replacement for regulation. \subsection*{Test Goals} \paragraph{Specification} The specification of the SUT is for the most part similar to the input-output behavior described in \S\ref{P:ERSUT}. Because the SUT is strongly related to Chess, many finer details of its specification can be derived from documentation of the world of Chess. \subparagraph{F\'ed\'eration Internationale Des Echecs (FIDE)} Due to the popularity of Chess, most people are familiar with how it is played. Hence, a summary of the rules would be excesive. However, due to the competitve nature of the game, the World Chess Federation describes the rules painfully precise. \cite{FIDE} \subparagraph{Portable Game Notation (PGN)} Many Chess matches are recorded with the PGN. \cite{PGN} As some of the test cases consist of text files describing Chess games in PGN, one of our test cases is included in listing \ref{L:PGNE} as an example for the notation. \begin{listing} \inputminted{text}{../test/database/m0.in} \caption{Portable Game Notation (PGN) Example} \label{L:PGNE} \end{listing} PGN consists of two parts: first, the tag pairs and, second, the movetext. The tag pairs encode properties of the game of Chess in question, such as the name of the tournament or match event, the player of the white pieces and the player of the black pieces, and the result of the game. It is possible to add self defined tag pairs. During our testing, we have added the name of the test with a \mintinline{text}{TestName} tag pair. Tag pairs always follow the structure \begin{minted}{text} ['Tag Name' "'Tag Value'"] \end{minted} where \mintinline{text}{'Tag Name'} holds the name of the tag (\mintinline{text}{Event}, \mintinline{text}{White}, \mintinline{text}{Black}, \mintinline{text}{Result}, ...) and \mintinline{text}{'Tag Value'} the value of the tag (\mintinline{text}{Third Rosenwald Trophy}, \mintinline{text}{Donald Byrne}, \mintinline{text}{Robert James Fischer}, \mintinline{text}{0-1}, ...). The movetext encodes the actual game of Chess in question. It mainly consists of the moves made during the match, which are encode with \begin{minted}{text} 'Round Number'. 'White's Move' 'Black's Move' \end{minted} where each player's move is recorded in the Standard Algebraic Notation (SAN). \subparagraph{Standard Algebraic Notation (SAN)} Most moves are recorded with the SAN, which adheres to the following structure. \cite{SAN} \begin{minted}{text} 'Piece Identifier'x?'End Position' \end{minted} \mintinline{text}{'Piece Identifier'} denotes the piece being moved by a single capital letter: \mintinline{text}{B} (Bishop), \mintinline{text}{K} (King), \mintinline{text}{N} (Knight), \mintinline{text}{Q} (Queen) and \mintinline{text}{R} (Rook). Pawns either do not have an abbreviation or are denoted with \mintinline{text}{P}. In the case the abbreviation alone is not sufficient to identify the piece being moved, (partial) initial position information, according to the notation explained in figure \ref{F:SN}, is added to solve ambiguity. \mintinline{text}{'Piece Identifier'} is followed by \mintinline{text}{x} only if the move resulted in a capture. \mintinline{text}{'End Position'} records the final position of the moved piece using the notation explained in figure \ref{F:SN}. Additional notation exists to describe more specialized aspects of Chess moves. For example: \mintinline{text}{+} is appended to checking moves and \mintinline{text}{O-O} alone denotes King side Castling moves. \begin{figure} \centering \newgame \showboard \caption{Square Notation: Each square is noted by its file (column letter) and rank (row number).} \label{F:SN} \end{figure} \paragraph{Testing Scope} The SUT consists of a wrapper and the {\tt chesshs} library which the wrapper uses. In order to transform the {\tt chesshs} package into a system with a command line interface, we have written a Haskell program which interacts with the library in such a way that the desired input-output behavior as described in \S\ref{P:ERSUT} of the SUT is achieved. In essence, it ``wraps" around the library to make it interactive. Hence the term `wrapper'. Due to its strong influence on the behavior of the SUT, we will also be testing the wrapper. Nonetheless, testing will still focus mainly on the {\tt chesshs} package and its individual components, as this is the (sub)system of interest. Specifically, we will be testing \begin{enumerate} \item the wrapper, \label{EI:W} \item PGN parsing, \label{EI:PGNP} \item move legality verification and \label{EI:MLV} \item checkmate position determination. \label{EI:CPD} \end{enumerate} This means that any rule of the game of Chess not implemented in the {\tt chesshs} library, such as timing, can and will not be tested. The input-output specification in \S\ref{P:ERSUT} has effect on \ref{EI:W}, the PGN specification has effect on \ref{EI:PGNP} and the FIDE Chess rules have an effect on \ref{EI:MLV} and \ref{EI:CPD}. \cite{PGN, FIDE} Because every system component will be tested both manually and automatically, no special distinctions with regards to the system elements and test methods are necessary. \\ Functionality is the most relevant quality characteristic of the tests. We would like to have a reasonable amount of certainty that the SUT behaves as specified. Reliability is also relevant to the SUT. Because it is a library, many other systems will depend on its continuous availability, even after a ``crash" of some sort. For the same reason, usability is also relevant to the SUT. Packages are components of a programming language that will be \emph{used} by other systems, hence it is paramount for the library to have an \emph{``usable"} interface. In short, many of the quality characteristics that apply to libraries, will also apply neatly to the SUT. \\ White box and gray box testing would look somewhat similar when testing the SUT, due to the fact that the system is a package. Gray box testing would only look at the components available to other programs, that is, the exported elements. White box testing would also look at every component of the library including their implementation. Gray box testing would consider every component as a black box, that is, it will only focus on testing if each component adheres to its related specification. White box testing would consider every component by its literal code. In conclusion, white box testing would focus on the actual implementation of the library's functionality and gray box testing would only focus on the library's exported component's functionality without considering any actual code. \subsection*{Test Method} The testing level is mainly at the unit level, and somewhat at the integration level. The SUT only consists of two components: the wrapper and the {\tt chesshs} package. Both are ``atomic" enough to be tested via unit testing. The interaction between these two, mostly the usage of the library by the wrapper, will be tested via integration testing. Testing at the module, system and acceptance level is not reasonable in the case of our SUT, as its scale and number of subsystems limits us to lower testing levels. \paragraph{Test Generation Techniques} The majority of our test generation techniques are based upon or similar to black box methods. However, we also will apply some white box testing techniques. \subparagraph{Black Box Techniques} The focus of the black box testing is on the SUT in its entirety, that is, the {\tt chesshs} package surrounded by the wrapper. The black box tester will try to find faults in the input-output behavior of the program as described in \S\ref{P:ERSUT}. In this type, test generation is based on error guessing and an examples database. Error guessing is based mostly on our collective experience of developing and testing software in the past. But also somewhat on our experience with the game of Chess and using other similar SUTs. The examples database consists of a collection of text files describing a game of Chess in (possibly illegal) PGN (the `{\tt .in}' files) and the expected output of the SUT (the `{\tt .out}' files). We will manually create test cases of this form, where we will be possibly introducing illegal PGN guided by error guessing. Sadly, equivalence partitioning and boundary value analysis are hard to apply to our SUT, given that the collection of all possible Chess games is nearly impossible to categorize. Use case testing is also tricky to apply. The wrapper is very limited in its interaction and is designed purely to make the {\tt chesshs} package ``usable" for testing. Since the core of our SUT is a library, no \emph{end} user will likely interact with it directly. In summary, it does not feel reasonable to apply use case testing to the SUT. \subparagraph{White Box Techniques} The focus of the white box testing is on the {\tt chesshs} package only, that is, the components of the library. The white box tester will try to find faults in the implementation of the package with regards to its specification. In this type, test generation is based on statement coverage and manual path based coverage. Statement coverage, in Haskell's case, marks the lines of code that have been executed at least once. Path based coverage analysis is not included in the testing tools used. To compensate, we will measure path based coverage manually. By hand, the implementation is transformed into a control flow graph, and we manually determine how the test cases ``traverse" the code. We will continuously adapt the white box tester until it reaches a sufficient level of coverage, that is full coverage, unless achieving so is unfeasible or unreasonable. Sadly, path, branch and condition coverage are not included in the tools we will be using during the testing. Hence, it is substantially harder to implement these test generation techniques and will not be applied during our testing. \\ As every software developer and tester knows, it is very hard, if not impossible, to completely test a given system. Noting that the risks surrounding our SUT are almost negligible (see \S\ref{P:ERSUT}), the pressure on the testing is not noticeable. However, we still feel the tests as described should give a more than sufficient scrutiny. The black box testing will likely find, if any, faults in the wrapper and the interaction with the {\tt chesshs} package. The white box testing will likely find, if any, errors in the implementation of the library. \subsection*{Test Environment} Since the SUT is in actuality the {\tt chesshs} package, a user interface is clearly lacking. Thats why we have written a wrapper which makes the library interactive. See listing \ref{L:WI} for the full implementation. \\ \begin{listing} \inputminted[fontsize=\footnotesize]{haskell}{../src/runchess.hs} \caption{Wrapper Implementation} \label{L:WI} \end{listing} Black box tests will be automatically performed using shell scripting. In our case, we will be using bash and Linux compatible scripts. Test automation for white box testing is achieved by functionalities available in the to be used testing tool Quick Check. \cite{QC} \\ The software environments we will be using for testing our SUT are the operating system Linux, specifically Debian and Ubuntu, and the Haskell Platform {\tt 2014.2.0.0}, which includes, the Glasgow Haskell Compiler (GHC) {\tt 7.10.3}, Cabal {\tt 1.22.6.0} ({\tt 1.22.5.0} for the library) and Quick Check {\tt 2.8.1}. \cite{HP, GHC, C, QC} The hardware platforms are three differently powerful laptops, notably Acer and Toshiba. The testing architecture is best described by the graphic shown in figure \ref{F:TAGO}. \begin{figure} \centering \begin{tikzpicture}[ every node/.style={draw,rectangle}, title/.style={draw=none}, ->,>=open triangle 60] \node (bbtest) at (4,0) {Black Box Tester ({\tt test.sh})}; \node[matrix,row sep=.5em] (prog) at (0,0) { \node[title] {Wrapper};\\ \node (chesshs) {{\tt chesshs}};\\ }; \node[left=2em of chesshs] (wbtest) {White Box Tester ({\tt test.hs})}; \node[below of=wbtest] (quickcheck) {Quick Check}; \draw (bbtest) -- (prog); \draw (wbtest) -- (chesshs); \draw (quickcheck) -- (wbtest); \end{tikzpicture} \caption{Testing Architecture Graphical Overview} \label{F:TAGO} \end{figure} No stubs nor drivers are necessary for executing the testing architecture. \\ Most, if not all, documentation surrounding the tests, test results and test environment will be included in this document. It is possible that some tests and test results will contain information not stated here, such as the name of a test case and the possible flaw that it (tries to) detect(s). However, any external information will easily be found in the appropriate files. The tests will either be stored in the examples database (directory) or will be part of the Quick Check unit test module(s) (source file). The test results will be stored, if necessary, in the same files as their related test or in another easy to locate file. The test environment will be stored, as every other testing component, in a Git repository found at \url{https://gitlab.science.ru.nl/eveen/Testing-Techniques.git}. Since Haskell, as most functional programming languages, is ``stable" in terms of side effects and randomness, reproductivity should no be an issue. Nevertheless, we will make sure every test case can be reproduced, that is, run again leading to the same results as is documented. \section{Manual Testing} % TODO : Add the "results" of the exercises of section 2 of the assignment. \section{Automated Testing} % TODO : Add the "results" of the exercises of section 3 of the assignment. \begin{thebibliography}{8} \bibitem[C] {C} \url{http://haskell.org/cabal} \bibitem[FIDE] {FIDE} \url{http://fide.com/fide/handbook.html} \bibitem[GHC] {GHC} \url{http://haskell.org/ghc} \bibitem[HP] {HP} \url{http://haskell.org/platform} \bibitem[PGN] {PGN} \url{http://saremba.de/chessgml/standards/pgn/pgn-complete.htm} \bibitem[QC] {QC} \url{http://hackage.haskell.org/package/QuickCheck} \bibitem[SAN] {SAN} \url{http://cfajohnson.com/chess/SAN} \bibitem[SUT] {SUT} \url{http://hackage.haskell.org/package/chesshs} \end{thebibliography} \end{document}