%  validate.tex from ObjectProDSP 0.2
%  Copyright (C) 1994 1995, Mountain Math Software. All rights reserved.
%  
%  This file is part of ObjectProDSP, a tool for Digital Signal
%  Processing design, development and implementation. It is free
%  software provided you use and distribute it under the terms of
%  version 2 of the GNU General Public License as published
%  by the Free Software Foundation. You may NOT distribute it or
%  works derived from it or code that it generates under ANY
%  OTHER terms.  In particular NONE of the ObjectProDSP system is
%  licensed for use under the GNU General Public LIBRARY License.
%  Mountain Math Software plans to offer a commercial version of
%  ObjectProDSP for a fee. That version will allow redistribution
%  of generated code under standard commercial terms.
%  
%  This program is distributed in the hope that it will be useful,
%  but WITHOUT ANY WARRANTY; without even the implied warranty of
%  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
%  GNU General Public License for more details.
%  
%  You should have received a copy of version 2 of the GNU General
%  Public License along with this program. See file COPYING. If not
%  or if you wish information on commercial versions and licensing
%  write Mountain Math Software, P. O. Box 2124, Saratoga, CA 95070,
%  USA, or send us e-mail at: support@mtnmath.com.
%  
%  You may also obtain the GNU General Public License by writing the
%  Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139,
%  USA.  However if you received a copy of this program without the
%  file COPYING or without a copyright notice attached to all text
%  files, libraries and executables please inform Mountain Math Software.
%  
%  ObjectProDSP is a trademark of Mountain Math Software.
%  
\section{Regression tests }
\label{Sec:validation}

Regression tests can assure that under controlled inputs a program
produces exactly the output it had previously. The original output
that the test is run against must be manually checked. One difficulty
is insuring the correctness of this base line test output.
Another difficulty
is making the regression tests `orthogonal' so a small
change in the program will have a known small change in the test output.
Regression tests allow one to make code changes and have a reasonable assurance
that one has not fixed one bug only to introduce several new ones.
However this is only possible
if the tests can be made sufficiently independent of each other.
This can be particularly difficult in user interface code.

Table~\PageRef{Tab:AffectTest} shows what elements of the user
interface can and cannot be changed without invalidating the existing
regression test suite. Of course these lists are not exhaustive.
However they describe important areas where you cannot make changes
unless you are willing to recreate or heavily edit the regression tests.
They describe other important areas where you can change the user interface without
breaking the existing validation suite.

\begin{table}
\begin{center}
\begin{tabular}{|ll|} \hline
Feature & Affect \\ \hline
changing menu data base structure & affects everything lower in tree \\
new menu data base items or menus & no affect \\
delete menu data base items & only affects item deleted \\
changing menu bar key codes & affects items changed \\
new menu bar items & no affect \\
new menu bar top level menus & no affect \\
deleting menu bar items and menus & affects only items deleted \\
adding new classes or member functions & no affect \\
changing class names & affects classes changed \\
\hline
\end{tabular}
\caption{User interface changes that break and do not break validation}
\label{Tab:AffectTest}
\end{center}
\end{table}


\subsection{Running and creating regression tests}

Script \DppNm{\$OPD\_ROOT/scripts/master\_validate.sh} creates and
runs the
standard suite of regression tests. This is most conveniently used
with the \DppNm{Makefile} in \DppNm{\$OPD\_ROOT/build}.
For instructions execute the shell script or do a \DppNm{make}
with no arguments.
The rest of this section describes how
to add tests to the standard suite created and run by this script.


The standard tests are determined by the
files in \DppNm{\$OPD\_ROOT/validate} of the forms
\DppNm{make\_*\_validate.rec},\\
\DppNm{make\_*\_validate\_float.rec} and\\
\DppNm{make\_*\_validate\_int16.rec}.\\
Files of the first
type are used to generate tests for both simulators. The other
types generate tests for the floating point and 16 bit simulators
individually. To add new tests create an action file with the appropriate
name and add it to this directory. This action file must construct
a network and do a \DppNm{TargetValidate} on the network.
It should then exit
(preferably without saving the state). Make sure your network does
not conflict with any of the networks already used. (For each network
there is directory\\
{\tt \$OPD\_ROOT/validate/test\_nodes/val\_}{\it network\_name}{\tt \_float}\\
and/or a directory\\
{\tt \$OPD\_ROOT/validate/test\_nodes/val\_}{\it network\_name}{\tt \_int16}\\
created. Make sure your new network is not the same as {\it network\_name} in
any of these directory names.) (To assign a name to a network create
a nondefault instance of a network. You will be prompted for the name
to use. You must do this before you create any instances of objects
or a network will be created with a default name and the objects associated
with that network. You can move objects between networks, if they are
not linked, but you cannot change the name of a network once it has
been created.)

\subsection{Regression tests created with {\protect\tt TargetValidate}}

The \DppNm{Network} member function \DppNm{TargetValidate} 
generates \Dpp{.dpp} files and shell scripts to create
and execute regression tests. 
In these scripts the output of all \DppNm{Listing} and \DppNm{Plot} nodes
is automatically tested.
Two networks are generated from the original.
In the first
each occurrence of \DppNm{Listing} or \DppNm{Plot}
is replaced with an \DppNm{OutputNode}. In the 
second the same nodes are replaced with a \DppNm{CompareDisk} node.
\DppNm{CompareDisk} reads the
data generated by the \DppNm{OutputNode} and compares it with the
current input. 
The scripts created by \DppNm{TargetValidate}
are used for both target code and interactive code validation.)


\subsection{Writing and reading a file in different tests}

There are two ways to test output files (other than those
created by \DppNm{OutputNode}). 
You can write a file in one test and then read it in a subsequent
test. The second test should send the data to a \DppNm{Plot} or \DppNm{Listing} node.
so the data will be verified.
You can also do byte by byte comparison against base line validation
files by writing a file that ends with suffix \DppNm{.cmp}.

If you write a file in one test and read it in another you must be
sure the two tests are done in the correct order.
Tests are done in alphabetical order (as defined by the \DppNm{sort} command).
They are alphabetized by network name and by the \DppNm{make*validate.rec}
name.
To have one test generate output and a second read it
you must choose these names so the
tests that creates the data runs first.

Different tests
are run in different directories. You must specify an absolute path
name if different tests are to access the same file.
The standard place to put the
data is directory \DppNm{\$OPD\_ROOT/validate/test\_data}.
(You can use environmental variables
when entering file names.)
If this directory
does not exist it will be created when validation starts. You
may need to create the directory manually for debugging your tests.

\subsection{Byte by byte comparison files}

Some files cannot be easily tested by writing and then reading
a file. Such files can be tested by doing a byte by byte comparison
against a base line version of the file.
To test files in this way
write them in the
current directory and give them a name
ending with \DppNm{.cmp}. When you do a \DppNm{make VAL\_DATA}
to integrate your tests (as described below) a base line version of these
files will be created with suffix \DppNm{.cmpb}.
Subsequent \DppNm{make VALIDATE}
executions will test the newly generated \DppNm{.cmp} files against the
\DppNm{.cmpb} files by doing a comparison on every byte in both files.

All tests must contain at least one \DppNm{Plot} or \DppNm{Listing}
node and must
include the \DppNm{MakeTarget} operation. If no output is generated
the test will abort.  \DppNm{MakeTarget} is necessary to create
the test scripts.


\subsection{Documenting tests}

Each \DppNm{make\_*\_validate.rec} file must start with documentation
of the tests. If a file does not contain a line similar to the following the
validation script will abort:\\
\verb"######################################### END OF VALIDATION LIST"

This indicates the end of the information that annotates the tests.
Program \DppNm{\$OPD\_ROOT/report\_test} reads
the information above this line in each test file and uses it to
print a summary of what nodes and other features were tested and
how completely these features were tested. If \DppNm{report\_test} does
not find the line {\it exactly} as it expects, it will abort with an
error message. Copy the line from an existing file to make sure it is
exactly correct. 

Above this line you should include a single line for each node that
occurs in your test of the form:\\
\verb"#	node_name	tests performed or other comments" \\
Do not include \DppNm{Listing} or \DppNm{Plot}
nodes in the above list as these are replaced with other nodes
in generating the tests. Make sure that every thread
ends with either a \DppNm{Plot} or \DppNm{Listing} node so at least
the final output of the thread will be verified.


Following all nodes you can list other features tested one
per line. These lines should start with two sharp signs: \verb"##".
See any of the standard validation files, \DppNm{make\_*\_validate*.rec},
for examples. In these features list include the name of any \DppNm{.cmp}
files that the test generates. This will make it easy to go from the
\DppNm{.cmp} file name to the validation script that generated the file.

\subsection{Creating base line test data}

To create base line test data run \DppNm{\$OPD\_ROOT/scripts/master\_validate.sh}
with argument \DppNm{VAL\_DATA}.  You can temporarily move all of the existing
\DppNm{make*validate.rec} files in \DppNm{\$OPD\_ROOT/validate} to a different directory
to just do a \DppNm{make VAL\_DATA} with your new validation files. This will not
change the base line test data for the existing tests. (If you just add
your files and then do a \DppNm{make VAL\_DATA} all base line test data will
be recreated.) You can then transfer the original \DppNm{make*validate.rec} files
back to \DppNm{\$OPD\_ROOT/validate} and the next \DppNm{make VALIDATE} will run the full
suite including those tests you added. Make sure you manually check the
correctness of the base line test data you have added.

When you next do \DppNm{make VALIDATE} the validation log should report no
errors but additional test cases not in the previous log. Copy this log
to the appropriate base line validation log to complete the update.
File \DppNm{base\_log} is the log for floating point and 16 bit integer tests.
\DppNm{base\_float\_log} is for floating point tests only. (There is no capability
to run only 16 bit integer tests.) If your update includes floating
point tests you should update both base line log files.

\subsection{Make many mistakes in recording your test}

These are tests of both the DSP process and the GUI since the networks
are built interactively. It is good to make mistakes 
in construction the actions files for these tests. It makes the GUI
test more effective.

