diff options
| -rw-r--r-- | assignments/assignment1.tex | 89 |
1 files changed, 58 insertions, 31 deletions
diff --git a/assignments/assignment1.tex b/assignments/assignment1.tex index e4453b2..9f9e7e8 100644 --- a/assignments/assignment1.tex +++ b/assignments/assignment1.tex @@ -5,13 +5,14 @@ \usepackage{enumerate} \usepackage[hidelinks]{hyperref} \usepackage{minted} +\setminted{fontsize=\small,breaklines} \usepackage{skak} \usepackage{tikz} \usetikzlibrary{arrows, matrix, positioning} \usepackage{cleveref} -\title{The \texttt{chesshs} Haskell Package} -\subtitle{Test Plan, Manual Testing \& Automated Testing} +\title{Testing the \texttt{chesshs} Haskell Package} +\subtitle{Test Plan, Manual Testing and Automated Testing} \author{ Ren\'e den Hertog\footnote{s1007823} \and Camil Staps\footnote{s4498062} \and @@ -69,22 +70,22 @@ Furthermore, we expect such a system to be used as a supporting tool for match o The specification of the SUT is for the most part similar to the input-output behaviour described above. Because the SUT is strongly related to Chess, - many finer details of its specification can be derived from documentation of the world of Chess. + many finer details of its specification can be derived from documentation from the world of Chess. \paragraph{F\'ed\'eration Internationale Des Echecs (FIDE)} Due to the popularity of Chess, most people are familiar with how it is played. -Hence, a summary of the rules would be excesive. -However, due to the competitve nature of the game, +Hence, a summary of the rules is omitted here. +However, due to the competitive nature of the game, the World Chess Federation describes the rules painfully precise~\cite{FIDE}. \paragraph{Portable Game Notation (PGN)} -Many Chess matches are recorded with the PGN~\cite{PGN}. -As some of the test cases consist of text files describing Chess games in PGN, - one of our test cases is included in \cref{L:PGNE} as an example for the notation. +Many Chess matches are recorded in PGN~\cite{PGN}, an example of which is shown in \cref{L:PGNE}. \begin{listing} - \inputminted{text}{../test/database/m0.in} - \caption{Portable Game Notation (PGN) Example\label{L:PGNE}} + \inputminted[lastline=19]{text}{../test/database/m0.in} + \dots + \inputminted[firstline=42]{text}{../test/database/m0.in} + \caption{An example of Portable Game Notation (PGN).\label{L:PGNE}} \end{listing} PGN consists of two parts: the tag pairs and the movetext. @@ -123,14 +124,14 @@ Most moves are recorded with the SAN, which adheres to the following structure~\ <Piece Identifier>[x]<End Position> \end{minted} -\texttt{Piece Identifier} denotes the piece being moved by a single capital letter: +The \texttt{Piece Identifier} denotes the piece being moved by a single capital letter: \texttt{B} (Bishop), \texttt{K} (King), \texttt{N} (Knight), \texttt{Q} (Queen) and \texttt{R} (Rook). Pawns either do not have an abbreviation or are denoted with \texttt{P}. In the case the abbreviation alone is not sufficient to identify the piece being moved, (partial) initial position information is added to resolve ambiguity, according to the notation explained in \cref{F:SN}. -\texttt{Piece Identifier} is followed by \texttt{x} only if the move resulted in a capture. -\texttt{End Position} records the final position of the moved piece using the notation explained in figure \ref{F:SN}. +The \texttt{Piece Identifier} is followed by \texttt{x} only if the move resulted in a capture. +The \texttt{End Position} records the final position of the moved piece using the notation explained in \cref{F:SN}. Additional notation exists to describe more specialized aspects of Chess moves. For example, \texttt{+} is appended to checking moves and \texttt{O-O} alone denotes king side castling moves. @@ -138,37 +139,63 @@ For example, \texttt{+} is appended to checking moves and \texttt{O-O} alone den \centering \newgame \showboard - \caption{Square Notation: Each square is noted by its file (column letter) and rank (row number).} \label{F:SN} + \caption{Square Notation: Each square is noted by its file (column letter) and rank (row number).\label{F:SN}} \end{figure} \paragraph{Testing Scope} -The SUT consists of a wrapper and the \texttt{chesshs} library which the wrapper uses. +The SUT is the \texttt{chesshs} library. +However, it is useful to have an actual program that can be run. +To this end, we have written a small wrapper program that acts as a command-line interface to the library. +The source code of the wrapper is shown in \cref{L:WI}. +This program is kept as minimal as possible to contain (almost) no logic that might influence our tests. -In order to transform the \texttt{chesshs} package into a system with a command line interface, we have written a Haskell program which interacts with the library in such a way that the desired input-output behavior as described in \S\ref{P:ERSUT} of the SUT is achieved. In essence, it ``wraps" around the library to make it interactive. Hence the term `wrapper'. - -Due to its strong influence on the behavior of the SUT, we will also be testing the wrapper. Nonetheless, testing will still focus mainly on the \texttt{chesshs} package and its individual components, as this is the (sub)system of interest. Specifically, we will be testing +The \texttt{chesshs} package contains several subsystems. +Specifically, we will test \begin{enumerate} - \item the wrapper, \label{EI:W} - \item PGN parsing, \label{EI:PGNP} + \item PGN parsing; \label{EI:PGNP} \item move legality verification and \label{EI:MLV} \item checkmate position determination. \label{EI:CPD} \end{enumerate} -This means that any rule of the game of Chess not implemented in the \texttt{chesshs} library, such as timing, can and will not be tested. - -The input-output specification in \S\ref{P:ERSUT} has effect on \ref{EI:W}, the PGN specification has effect on \ref{EI:PGNP} and the FIDE Chess rules have an effect on \ref{EI:MLV} and \ref{EI:CPD}. \cite{PGN, FIDE} - -Because every system component will be tested both manually and automatically, no special distinctions with regards to the system elements and test methods are necessary. \\ - -Functionality is the most relevant quality characteristic of the tests. We would like to have a reasonable amount of certainty that the SUT behaves as specified. Reliability is also relevant to the SUT. Because it is a library, many other systems will depend on its continuous availability, even after a ``crash" of some sort. For the same reason, usability is also relevant to the SUT. Packages are components of a programming language that will be \emph{used} by other systems, hence it is paramount for the library to have an \emph{``usable"} interface. In short, many of the quality characteristics that apply to libraries, will also apply neatly to the SUT. \\ - -White box and gray box testing would look somewhat similar when testing the SUT, due to the fact that the system is a package. Gray box testing would only look at the components available to other programs, that is, the exported elements. White box testing would also look at every component of the library including their implementation. Gray box testing would consider every component as a black box, that is, it will only focus on testing if each component adheres to its related specification. White box testing would consider every component by its literal code. In conclusion, white box testing would focus on the actual implementation of the library's functionality and gray box testing would only focus on the library's exported component's functionality without considering any actual code. +The PGN specification has effect on \ref{EI:PGNP}; + the FIDE Chess rules have an effect on \ref{EI:MLV} and \ref{EI:CPD}. +Some Chess rules not implemented in the \texttt{chesshs} library, + like those concerning timing, + can and will not be tested. + +Our main focus will lie on the quality characteristic of functionality. +We would like to have a reasonable level of certainty that the SUT behaves as specified + and correctly implements the rules of Chess. +Reliability is also relevant to the SUT, + especially because it is a library that other systems depend on: + we do not expect any crashes; exceptions should be handled using \texttt{Maybe}s and the like. +Usability does not apply in the same way to libraries as it does to end user programs. +The actual users are the programmers that use the library in their code. +Hence, usability mainly concerns whether the API has sensible, well-scoped functions. +We will be able to comment on the usability as we have written the small wrapper, + but will not attempt to achieve an objective judgement in this aspect + --- that would require finding a reasonably large set of programmers using \texttt{chesshs}, + and we suspect that may be hard. + +White box and grey box testing would look somewhat similar when testing the SUT, due to the fact that the system is a package. +Grey box testing would only look at the components available to other programs, that is, the exported elements. +For instance, we might look at the types of functions, but not their implementation. +White box testing would also look at every component of the library including their implementation. +Grey box testing would consider every component as a black box: + it would only focus on testing if each component adheres to its related specification. +White box testing would consider every component by its literal code. +In conclusion, white box testing would focus on the actual implementation of the library's functionality + and grey box testing would only focus on the library's exported component's functionality without considering any actual code. \subsection*{Test Method} -The testing level is mainly at the unit level, and somewhat at the integration level. The SUT only consists of two components: the wrapper and the \texttt{chesshs} package. Both are ``atomic" enough to be tested via unit testing. The interaction between these two, mostly the usage of the library by the wrapper, will be tested via integration testing. Testing at the module, system and acceptance level is not reasonable in the case of our SUT, as its scale and number of subsystems limits us to lower testing levels. +Our tests are mainly at the unit level, and somewhat at the integration level. +The SUT only consists of two components: the wrapper and the \texttt{chesshs} package. +Both are \enquote{atomic} enough to be tested via unit testing. +The interaction between these two, mostly the usage of the library by the wrapper, will be tested via integration testing. +Testing at the module, system and acceptance level is not reasonable in the case of our SUT, as its scale and number of subsystems limits us to lower testing levels. \paragraph{Test Generation Techniques} @@ -205,7 +232,7 @@ As every software developer and tester knows, it is very hard, if not impossible Since the SUT is in actuality the \texttt{chesshs} package, a user interface is clearly lacking. Thats why we have written a wrapper which makes the library interactive. See listing \ref{L:WI} for the full implementation. \\ \begin{listing} - \inputminted[fontsize=\footnotesize]{haskell}{../src/runchess.hs} + \inputminted{haskell}{../src/runchess.hs} \caption{Wrapper Implementation} \label{L:WI} \end{listing} |