<!--  -->
<sect1 id="testing"> <title> &slony1; Test Suites </title>

<para> &slony1; has had (thus far) three test suites:
<itemizedlist>

<listitem><para> Ducttape tests</para>

<para> These were introduced as part of the original &slony1;
distribution, and induced load via running
<application>pgbench</application>.</para>

<para> Unfortunately, the tests required human intervention to control
invokation and shutdown of tests, so running them could not be readily
automated.</para></listitem>

<listitem><para> <link linkend="oldtestbed"> Test bed framework
</link></para>

<para> &slony1; version 1.1.5, introduced a test framework intended to
better support automation of the tests.  It eliminated the use of
<application>xterm</application>, and tests were self-contained and
self-controlled, so that one could run a series of tests.</para>

<para> Unfortunately, the framework did not include any way of
inducing distributed load, so as to test scenarios involving
sophisticated concurrent activity.</para></listitem>

<listitem><para> <link linkend="clustertest"> clustertest framework
</link></para>

<para> Introduced during testing of &slony1; version 2.0 during 2010,
and released in early 2011, this framework is intended to be a better
replacement for all of the preceding test
frameworks.</para></listitem>

</itemizedlist>

<sect1 id="clustertest"> <title>Clustertest Test Framework</title>

<sect2><title>Introduction and Overview</title>

<para> The clustertest framework is implemented in Java, where tests
are implemented in the interpreted JavaScript language.  The use of
Java made it much easier to implement tests involving concurrent
activities, both in terms of inducing test load, and in, concurrently
changing configuration of the replication cluster.

<para> It consists of two physical portions:

<itemizedlist>
<listitem><para> A framework, implemented in Java </para> 

<para> This software is available at <ulink
url="https://github.com/clustertest/clustertest-framework">
clustertest-framework @ GitHub </ulink>.

<para> This framework makes use of libraries from several other open source projects:</para>

<itemizedlist>
<listitem><para><filename>js.jar</filename></para>

<para> This is for <envar>org.mozilla.javascript</envar>, the Mozilla JavaScript interpreter</para></listitem>

<listitem><para><filename>junit-4.8.1.jar</filename></para>

<para> JUnit, a unit test framework.</para></listitem>

<listitem><para> <filename>log4j-1.2.15.jar</filename></para>

<para> Log4J is a popular Java-based framework for generating event
logs.</para></listitem>

<listitem><para><filename>postgresql-8.4-701.jdbc3.jar</filename></para>

<para> This is the &postgres; JDBC driver.</para></listitem>
</itemizedlist>

<para> To build the framework, it is necessary to have a Java compiler
and the build tool, <application>Ant</application>, installed.  To
build all the <filename>.jar</filename> files used by the framework,
one will run the command, with output similar to the following:

<screen>
% ant jar
Buildfile: /var/lib/postgresql/PostgreSQL/clustertest-framework/build.xml

compile-common:
    [mkdir] Created dir: /var/lib/postgresql/PostgreSQL/clustertest-framework/build/classes
    [javac] /var/lib/postgresql/PostgreSQL/clustertest-framework/build.xml:23: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

compile-testcoordinator:
    [javac] /var/lib/postgresql/PostgreSQL/clustertest-framework/build.xml:44: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
    [javac] Compiling 25 source files to /var/lib/postgresql/PostgreSQL/clustertest-framework/build/classes
    [javac] Note: /var/lib/postgresql/PostgreSQL/clustertest-framework/src/info/slony/clustertest/testcoordinator/script/ClientScript.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
     [copy] Copying 1 file to /var/lib/postgresql/PostgreSQL/clustertest-framework/build/classes/info/slony/clustertest/testcoordinator

jar-common:
    [mkdir] Created dir: /var/lib/postgresql/PostgreSQL/clustertest-framework/build/jar
      [jar] Building MANIFEST-only jar: /var/lib/postgresql/PostgreSQL/clustertest-framework/build/jar/clustertest-common.jar

compile-client:
    [javac] /var/lib/postgresql/PostgreSQL/clustertest-framework/build.xml:30: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
    [javac] Compiling 1 source file to /var/lib/postgresql/PostgreSQL/clustertest-framework/build/classes
     [copy] Copying 2 files to /var/lib/postgresql/PostgreSQL/clustertest-framework/build/classes/info/slony/clustertest/client

jar-client:
      [jar] Building jar: /var/lib/postgresql/PostgreSQL/clustertest-framework/build/jar/clustertest-client.jar

jar-coordinator:
      [jar] Building jar: /var/lib/postgresql/PostgreSQL/clustertest-framework/build/jar/clustertest-coordinator.jar

jar:

BUILD SUCCESSFUL
Total time: 2 seconds
</screen></para>

<para> At this time, there is no <quote>regression test</quote> for
the test framework; to validate that it works requires running tests
that use it.</para>

<para> It includes classes supporting &postgres;- and &slony1;-specific functionality such as:</para>
<itemizedlist>
<listitem><para> <classname> CreateDbScript</classname></para>
<para> Creates a database </para> </listitem>
<listitem><para> <classname> DropDbScript</classname></para>
<para> Drops a database </para> </listitem>
<listitem><para> <classname> LogShippingDaemon</classname></para>
<para> Starts up &slony1; logshipping daemon </para> </listitem>
<listitem><para> <classname> LogShippingDumpScript</classname></para>
<para> Dumps and loads logshipping-based schema </para> </listitem>
<listitem><para> <classname> PgCommand</classname></para>
<para> Run a &postgres; shell command (such as <command>psql</command>, <command>createdb</command>, and such)</para> </listitem>
<listitem><para> <classname> PgDumpCommand</classname></para>
<para> Dump a &postgres; database </para> </listitem>
<listitem><para> <classname> PsqlCommandExec</classname></para>
<para> Run SQL </para> </listitem>
<listitem><para> <classname> ShellExecScript</classname></para>
<para> Run a shell script/command</para> </listitem>
<listitem><para> <classname> SlonLauncher</classname></para>
<para> Start up a &lslon; process </para> </listitem>
<listitem><para> <classname> SlonikScript</classname></para>
<para> Run a &lslonik; script </para> </listitem>
</itemizedlist>

</listitem>

<listitem><para> Tests integrated into the &slony1; software
distribution, that consist of a combination of shell scripts,
JavaScript, and SQL scripts. </para>

<para> See the directory <filename>clustertest</filename> in the
&slony1; software distribution, which has two sets of tests:</para>
<itemizedlist>
<listitem><para> <xref linkend="disorder"> </para></listitem>
<listitem><para> <xref linkend="regression"> </para></listitem>
</itemizedlist>
</listitem>

</itemizedlist>

<sect2 id="disorder"> <title>DISORDER - DIStributed ORDER test</title>

<para> The <emphasis>DISORDER</emphasis> or <emphasis>DIStributed
ORDER</emphasis> test is intended to provide concurrency tests
involving a reasonably sophisticated schema to validate various
aspects of &slony1; behavior under concurrent load.</para>

<para> It consists of:
<itemizedlist>
<listitem><para> A schema for an inventory management application.</para>
<para> Major objects include customers, inventory items, orders, order lines, and shipments. </para>

<para> There are foreign key relationships between the various items,
as well as triggers that maintain inventory and customer balances.
Some of these relationships involve <command>ON DELETE CASCADE</command>, and
so some actions may induce large numbers of cascaded updates.</para>

</listitem>

<listitem><para> Stored procedures to induce creation of the various
sorts of objects, purchases, shipments, and additions and removals of
customers and products. </para></listitem>

<listitem><para> Some tests are intended to be run against replicas,
validating that balances add up.  We believe that &postgres; applies
changes in a transactional fashion such that they will always
<command>COMMIT</command> leaving the visible state consistent;
certain of the tests look for inconsistencies. </para></listitem>

<listitem><para> There are JavaScript test scripts that induce all
sorts of manipulations of replication clusters to validate that
replication configuration changes succeed and fail as expected.</para>
</listitem>

</itemizedlist>

<sect3 id="disorderconfig"> <title>Configuring DISORDER</title>

<para> DISORDER test configuration may be found in the following
files:</para>
<itemizedlist>
<listitem><para><filename>conf/disorder.properties.sample</filename></para>

<para> This file contains Java style properties indicating how to
connect to the various databases used by the DISORDER tests, including paths to
tools such as &lslon; and &lslonik;</para>

<para> The sample file is to be copied to
<filename>conf/disorder.properties</filename>, and customized to
indicate your local configuration.  By using a
<filename>.sample</filename> file, a developer may run tests within a
Git tree, and not need to worry about their customizations interfering
with the <quote>canonical</quote> sample configuration
provided.</para>

</listitem>

<listitem><para><filename>conf/java.conf.sample</filename></para>

<para> This is a shell script containing a path indicating where the
clustertest Java code (<emphasis>e.g.</emphasis> - the
<filename>clustertest-coordinator.jar</filename> file) may be found.  This is
also used, indirectly to determine where additional Java .jar files
such as the JDBC driver are located.  </para>

<para> As with the disorder properties, above, this needs to be copied
to <filename>conf/java.conf</filename>, and customized to indicate one's own local
configuration.</para>

</listitem>

<listitem><para><filename>conf/log4j.properties</filename></para>

<para> See documentation for <application>Log4J</application> for more
details as to how this is configured; the defaults provided likely do
not need to be altered. </para></listitem>
</itemizedlist>

</sect3>

<sect2 id="regression"> <title>Regression Tests</title>

<para> These tests represent a re-coding of the tests previously
implemented as shell scripts using the clustertest framework.</para>

<para> These tests have gradually been enhanced to provide coverage of
scenarios with which &slony1; has had problems; it is to be expected
that new bugs may lead to the addition of further tests.</para>

<sect3> <title>Configuring Regression Tests</title>

<para> Similar to the <xref linkend="disorderconfig"> for DISORDER tests, there are three configuration parameters:</para>
<itemizedlist>
<listitem><para><filename>conf/slonyregress.properties.sample</filename></para>

<para> This file contains Java style properties indicating how to
connect to the various databases used by the regression tests,
including paths to tools such as &lslon; and &lslonik;</para>

<para> The sample file is to be copied to
<filename>conf/slonyregress.properties</filename>, and customized to
indicate your local configuration.  By using a
<filename>.sample</filename> file, a developer may run tests within a
Git tree, and not need to worry about their customizations interfering
with the <quote>canonical</quote> sample configuration
provided.</para>

</listitem>

<listitem><para><filename>conf/java.conf.sample</filename></para>

<para> This is a shell script containing a path indicating where the
clustertest Java code (<emphasis>e.g.</emphasis> - the
<filename>clustertest-coordinator.jar</filename> file) may be found.
This is also used, indirectly to determine where additional Java .jar
files such as the JDBC driver are located.  </para>

</listitem>

<listitem><para><filename>conf/log4j.properties</filename></para>

<para> Identical to configuration for DISORDER.
</itemizedlist>

</sect3>
</sect2>

</sect1>
<sect1 id="oldtestbed"><title> &slony1; Test Bed Framework </title>

<indexterm><primary>test bed framework</primary></indexterm>

<para> Version 1.1.5 of &slony1; introduced a common test bed
framework intended to better support running a comprehensive set of
tests at least somewhat automatically.  </para>

<para> The new test framework is mostly written in Bourne shell, and
is intended to be portable to both Bash (widely used on Linux) and
Korn shell (widely found on commercial UNIX systems).  The code lives
in the source tree under the <filename> tests </filename>
directory.</para>

<para> At present, nearly all of the tests make use of only two
databases that, by default, are on a single &postgres; postmaster on
one host.  This is perfectly fine for those tests that involve
verifying that &slony1; functions properly on various sorts of data.
Those tests do things like varying date styles, and creating tables
and sequences that involve unusual names to verify that quoting is
being handled properly. </para>

<para> It is also possible to configure environment variables so that
the replicated nodes will be placed on different database backends,
optionally on remote hosts, running varying versions of
&postgres;.</para>

<para>Here are some of the vital files...</para>

<itemizedlist>

<listitem><para> <filename>run_test.sh</filename> </para></listitem>

</itemizedlist>

<para> This is the central script for running tests.  Typical usage is
thus:</para>

<para> <command> ./run_test.sh </command></para>
<screen>
usage ./run_test.sh testname
</screen>

<para> You need to specify the subdirectory name of the test set to be
run; each such set is stored in a subdirectory of
<filename>tests</filename>.</para>

<para> You may need to set one or more of the following environment
variables to reflect your local configuration.  For instance, the
writer runs <quote>test1</quote> against &postgres; 8.0.x using the
following command line:</para>

<screen> PGBINDIR=/opt/OXRS/dbs/pgsql8/bin PGPORT=5532 PGUSER=cbbrowne ./run_test.sh test1 </screen>

<glosslist>
<glossentry><glossterm> <envar>PGBINDIR</envar> </glossterm>

<glossdef><para> This determines where the test scripts look for
&postgres; and &slony1; binaries.  The default is <filename>
/usr/local/pgsql/bin</filename>.</para>

<para> There are also variables <envar>PGBINDIR1</envar> thru
<envar>PGBINDIR13</envar> which allows you to specify a separate path
for each database instance.  That will be particularly useful when
testing interoperability of &slony1; across different versions of
&postgres; on different platforms. In order to create a database of
each respective version, you need to point to
an <application>initdb</application> of the appropriate
version.</para> </glossdef> </glossentry>

<glossentry><glossterm> <envar>PGPORT</envar> </glossterm>
<glossdef><para> This indicates what port the backend is on.  By
default, 5432 is used. </para> 

<para> There are also variables <envar>PORT1</envar> thru
<envar>PORT13</envar> which allow you to specify a separate port
number for each database instance.  That will be particularly useful
when testing interoperability of &slony1; across different versions of
&postgres;. </para> </glossdef> </glossentry>

<glossentry><glossterm> <envar>PGUSER</envar> </glossterm>
<glossdef><para> By default, the user <filename>postgres</filename> is
used; this is taken as the default user ID to use for all of the
databases. </para>

<para> There are also variables <envar>USER1</envar> thru
<envar>USER13</envar> which allow specifying a separate user name for
each database instance.  The tests assume this to be a &postgres;
<quote>superuser.</quote> </para> </glossdef> </glossentry>

<glossentry><glossterm> <envar>WEAKUSER</envar> </glossterm>
<glossdef><para> By default, the user <filename>postgres</filename> is
used; this is taken as the default user ID to use for the <xref linkend="stmtstorepath"> connections to all of the
databases. </para>

<para> There are also variables <envar>WEAKUSER1</envar> thru
<envar>WEAKUSER13</envar> which allow specifying a separate user name
for each database instance.  This user <emphasis> does not </emphasis>
need to be a &postgres; <quote>superuser.</quote> This user can start
out with no permissions; it winds up granted read permissions on the
tables that the test uses, plus read access throughout the &slony1;
schema, plus write access to one table and sequence used to manage
node locks. </para> </glossdef> </glossentry>

<glossentry><glossterm> <envar>HOST</envar> </glossterm>
<glossdef><para> By default, <filename>localhost</filename> is used.
</para>

<para> There are also variables <envar>HOST1</envar> thru
<envar>HOST13</envar> which allow specifying a separate host for
each database instance.</para></glossdef>
</glossentry>

<glossentry><glossterm> <envar>DB1</envar> thru <envar>DB13 </envar> </glossterm> 

<glossdef><para> By default, <filename>slonyregress1</filename> thru
<filename>slonyregress13</filename> are used.
</para>

<para> You may override these from the environment if you have some
reason to use different names. </para></glossdef>
</glossentry>

<glossentry>
<glossterm><envar>ENCODING</envar></glossterm>

<glossdef><para> By default, <filename>UNICODE</filename> is used, so
that tests can create UTF8 tables and test the multibyte capabilities.
</para></glossdef>

</glossentry>

<glossentry>
<glossterm><envar>MY_MKTEMP_IS_DECREPIT</envar></glossterm>

<glossdef><para> If your version of Linux uses a variation of
<application>mktemp</application> that does not generate a full path
to the location of the desired temporary file/directory, then set this
value.  </para></glossdef>

</glossentry>

<glossentry>
<glossterm><envar>TMPDIR</envar></glossterm>

<glossdef><para> By default, the tests will generate their output in
<filename>/tmp</filename>, <filename>/usr/tmp</filename>, or
<filename>/var/tmp</filename>, unless you set your own value for this
environment variable.  </para></glossdef>

</glossentry>

<glossentry>
<glossterm><envar>SLTOOLDIR</envar></glossterm>

<glossdef><para> Where to look for &slony1; tools such as
<application>slony1_dump.sh</application>.  </para></glossdef>

</glossentry>

<glossentry>
<glossterm><envar>ARCHIVE[n]</envar></glossterm>

<glossdef><para> If set to <quote>true</quote>, for a particular node,
which will normally get configured out of human sight in the
generic-to-a-particular-test file <filename>settings.ik</filename>,
then this node will be used as a data source for <xref
linkend="logshipping">, and this causes the test tools to set up a
directory for the <link linkend="slon-config-archive-dir">
archive_dir</link> option.  </para></glossdef>

</glossentry>

<glossentry>
<glossterm><envar>LOGSHIP[n]</envar></glossterm>

<glossdef><para> If set to <quote>true</quote>, for a particular node,
which will normally get configured out of human sight in
<filename>settings.ik</filename> for a particular test, then this
indicates that this node is being created via <xref
linkend="logshipping">, and a &lslon; is not required for this node.
</para></glossdef>

</glossentry>

<glossentry>
<glossterm><envar>SLONCONF[n]</envar></glossterm>

<glossdef><para> If set to <quote>true</quote>, for a particular node,
typically handled in <filename>settings.ik</filename> for a given
test, then configuration will be set up in a <link
linkend="runtime-config"> per-node <filename>slon.conf</filename>
runtime config file. </link> </para> </glossdef>
</glossentry>

<glossentry>
<glossterm><envar>SLONYTESTER</envar></glossterm>

<glossdef><para> Email address of the person who might be
contacted about the test results. This is stored in the
<envar>SLONYTESTFILE</envar>, and may eventually be aggregated in some
sort of buildfarm-like registry. </para> </glossdef>
</glossentry>

<glossentry>
<glossterm><envar>SLONYTESTFILE</envar></glossterm>

<glossdef><para> File in which to store summary results from tests.
Eventually, this may be used to construct a buildfarm-like repository of
aggregated test results. </para> </glossdef>
</glossentry>

<glossentry>
<glossterm><filename>random_number</filename> and <filename>random_string</filename> </glossterm>

<glossdef><para> If you run <command>make</command> in the
<filename>test</filename> directory, C programs
<application>random_number</application> and
<application>random_string</application> will be built which will then
be used when generating random data in lieu of using shell/SQL
capabilities that are much slower than the C programs.  </para>
</glossdef>
</glossentry>

</glosslist>

<para> Within each test, you will find the following files: </para>

<itemizedlist>
<listitem><para> <filename>README</filename> </para> 

<para> This file contains a description of the test, and is displayed
to the reader when the test is invoked. </para> </listitem>

<listitem><para> <filename>generate_dml.sh</filename> </para> 
<para> This contains script code that generates SQL to perform updates. </para> </listitem>
<listitem><para> <filename>init_add_tables.ik</filename> </para> 
<para>  This is a <xref linkend="slonik"> script for adding the tables for the test to repliation. </para> </listitem>
<listitem><para> <filename>init_cluster.ik</filename> </para> 
<para> <xref linkend="slonik"> to initialize the cluster for the test.</para> </listitem>
<listitem><para> <filename>init_create_set.ik</filename> </para> 
<para> <xref linkend="slonik"> to initialize additional nodes to be used in the test. </para> </listitem>
<listitem><para> <filename>init_schema.sql</filename> </para> 
<para> An SQL script to create the tables and sequences required at the start of the test.</para> </listitem>
<listitem><para> <filename>init_data.sql</filename> </para> 
<para> An SQL script to initialize the schema with whatever state is required for the <quote>master</quote> node.  </para> </listitem>
<listitem><para> <filename>init_subscribe_set.ik</filename> </para> 
<para> A <xref linkend="slonik"> script to set up subscriptions.</para> </listitem>
<listitem><para> <filename>settings.ik</filename> </para> 
<para> A shell script that is used to control the size of the cluster, how many nodes are to be created, and where the origin is.</para> </listitem>
<listitem><para> <filename>schema.diff</filename> </para> 
<para> A series of SQL queries, one per line, that are to be used to validate that the data matches across all the nodes.  Note that in order to avoid spurious failures, the queries must use unambiguous <command>ORDER BY</command> clauses.</para> </listitem>
</itemizedlist>

<para> If there are additional test steps, such as
running <xref linkend="stmtddlscript">,
additional <xref linkend="slonik"> and SQL scripts may be necessary.</para>

</sect1>
<!-- Keep this comment at the end of the file
Local variables:
mode:sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document:"slony.sgml"
sgml-exposed-tags:nil
sgml-local-catalogs:("/usr/lib/sgml/catalog")
sgml-local-ecat-files:nil
End:
-->
