BEQUIESCE is a test harness for JavaScript libraries, with a bias towards proving that functions either fail correctly or succeed for the right reasons.
Existing tools for testing JavaScript suffer from three problems: 1) they emphasize DOM testing over module testing, 2) they are verbose to the point of being hard to maintain, and 3) they mask the true expressiveness of JavaScript.
The Bequiesce test harness begins with a solid basis as a straightforward library regression test tool, while following the guideline that test cases must be easy to read and maintain. Bequiesce does this by using JavaScript's ability to evaluate strings that contain JavaScript code. With this simple approach, the full power of JavaScript remains within the hands of the test developer.
Bequiesce is itself a library of JavaScript classes with one entry point, main.js. The library uses node.js as its execution platform. The only dependency is the joezone library, which shares a common license and is provided together with Bequiesce. There are no npm module dependencies. Running a test is done from Bash using the node command with the --use_strict option, like this:
bequiesce case.test.js
Bequiesce test packages are composed entirely of JavaScript which are parsed by the test harness. JavaScript statements within a test package are parsed line-by-line and shunted to one of four collections for subsequent evaluation: 1) common sections, 2) situation sections, 3) propositions, or 4) proofs. Test authors develop their test cases in a single source file, organized into groups separated by pragmas. The destination for each parsed line is determined by the presence of these three pragmas: @common, @using, and @testing.
Parsed lines that occur immediately after the common pragma are shunted to the common section: these JavaScript statements become part of the evaluation stream for every test case defined later in the test package.
Parsed lines that occur immediately after a using pragma are shunted to the situation section: these JavaScript statements become part of the evaluation stream for test cases defined within the next testing pragma.
Parsed lines that occur immediately after a testing pragma contain proposition-proof test cases. These lines are split into two halves by the presence of the double-semicolon ( ;; ) signal. Everything to the left of the signal is added to the collection of propositions. Everything to the right of the signal is added to the collection of proofs.
Here's what the simplest possible test package might look like.
//@using
var z = x + y;
//@testing
x = 1; y = 2; ;; z == 3
When parsed and evaluated by the test harness, this test package is effectively executed as:
x = 1; y = 2;
var z = x + y;
assert(z == 3);
We can improve our test coverage by keeping the situation the same, and by adding a few more proposition/proof test cases.
//@using
var z = x + y;
//@testing
var x = 1; var y = 2; ;; z == 3
var x = 1; var y = 'A'; ;; z == '1A'
var x = 1; var y = NaN; ;; isNaN(z)
var x = 1; var y; ;; isNaN(z)
When parsed and evaluated by the test harness, this second test package is effectively executed as:
x = 1; y = 2;
var z = x + y;
assert(z == 3);
x = 1; y = 'A';
var z = x + y;
assert(z == '1A');
x = 1; y = NaN;
var z = x + y;
assert(isNaN(z));
x = 1; var y;
var z = x + y;
assert(isNaN(z));
Notice that each proposition/proof test case is represented on a single line. This is an important design choice. It facilitates the easy assembly of a large number of simple test cases for each situation. When designing test cases, the author must keep this single-line rule in mind, and strive to set up simple test cases that can be proved or disproved with this approach.
Sometimes a proof will need to evaluate two or more assertions for a single proposition. This can be done by separating each proof with a double-ampersand signal ( && ). Here's an example that has two propositions each with three proofs, creating a total of six test cases.
//@using
var z = x + y;
//@testing
var x = 1; var y = 2; ;; z == 3 && Number.isInteger(z) && !isNaN(z)
var x = 1; var y = 'A'; ;; z == '1A' && typeof z == 'string' && isNaN(z)
The results of the test run are printed to the console. Here's what 003.test.js would produce:
[003.test: 4] Pass 6 Fail 0 [unnamed section] --> [unnamed group]
-------- --------
[003.test: 7] Pass 6 Fail 0
======== ========
Bequiesce Pass 6 Fail 0
When more than one @testing pragma is used, the results of each test group are tabulated separately. 004.test.js is a test package with three test groups. Notice that we can improve the readability of the results by naming sections and groups: simply provide a description immediately after the @using or @testing pragma. (Here we're using "sum of two values", "invalid inputs", etc.)
//@using sum of two values
var z = x + y;
//@testing invalid inputs
var x = 1; var y = NaN; ;; isNaN(z)
var x = NaN; var y; ;; isNaN(z)
var x = 1; var y; ;; isNaN(z)
var x; var y = 1; ;; isNaN(z)
var x = 1; var y = []; ;; isNaN(z)
var x = 1; var y = {}; ;; isNaN(z)
//@testing integers
var x = 1; var y = 2; ;; z == 3
var x = -1; var y = -2; ;; z == -3
var x = 1; var y = -2; ;; z == -1
var x = -1; var y = 2; ;; z == 1
//@testing floats
var x = 1.5; var y = 2; ;; z == 3.5
var x = 1; var y = 2.5; ;; z == 3.5
The results for each of the three groups are provide on their own line, and the total results for the file are provided afterwards.
This sample also demonstrates how a failure is reported. The proposition on line 9 sets y to be an empty array, and the proof expects the result to be NaN, but surprisingly it results in a valid number: an integer plus an empty array equals an integer! (1 + [] = 1)
==== Test Case =====================
[004.test: 1] Section: sum of two values
[004.test: 5] Group: invalid inputs
[004.test: 9] Proposition: var x = 1; var y = [];
[004.test: 9] Proof: isNaN(z) <-- FAILED
[004.test: 1] Situation:
var z = x + y;
[004.test: 5] Pass 5 Fail 1 sum of two values --> invalid inputs
[004.test: 12] Pass 4 Fail 0 sum of two values --> integers
[004.test: 18] Pass 2 Fail 0 sum of two values --> floats
-------- --------
[004.test: 21] Pass 11 Fail 1
======== ========
Bequiesce Pass 11 Fail 1
The simple examples demonstrated so far have used the two most important pragmas, @using and @testing. Example 004.test.js shows how more than one @testing group can facilitate the logical organization of test cases that apply to the same situation. A similar pattern exists whereby multiple @using pragmas can appear in a single test package. When this pattern occurs, each subsequent @using fully replaces the collection of JavaScript statements defined in any previous @using situation. Consider this scenario:
//@using situation1
var z = x + y;
//@testing proposition/proof1
var x = 1; var y = 2; ;; z == 3
//@using situation2
var z = y + x;
//@testing proposition/proof2
var x = 1; var y = 2; ;; z == 3
//@using situation3
var z = x + x;
//@testing proposition/proof3
var x = 1; ;; z == 2
//@using situation4
var z = y + y;
//@testing proposition/proof4
var y = 2; ;; z == 4
When test package 005.test.js is parsed, it effectively becomes this execution stream:
var x = 1; var y = 2;
var z = x + y;
assert(z == 3);
var x = 1; var y = 2;
var z = y + x;
assert(z == 3);
var x = 1;
var z = x + x;
assert(z == 2);
var y = 2;
var z = y + y;
assert(y == 4);
The example test packages so far have been unimaginative one-line situations. Realistically, a test situation might import class definitions, instantitate new objects, execute methods and test the state of the objects. In order to facilitate this pattern, the @common pragma is often useful. It behaves like a @using pragma, with the exception that it's "sticky" and applies to every situation declared in the test package. There can only be one @common pragma per test package.
Here is a more realistic scenario, where a JavaScript class is imported once in a @common section and applied to multiple @using/@testing situations.
When real regression tests are developed for libraries of any substantial size, it quickly become necessary to run an entire test suite -- a full collection of separate test packages. When running a test suite, everything that has been discussed so far still applies. A test suite is a collection of test packages within a single file system directory. (Only files following the naming convention will be included.) Execute the test suite from the CLI by passing a directory instead of a filename like this:
node --use_strict /path-to-bequiesce/src/main.js path-to-test-suite
