ack/doc/val.doc

.\" $Header$
.ll 72
.wh 0 hd
.wh 60 fo
.de hd
'sp 5
..
.de fo
'bp
..
.tr ~
.               PARAGRAPH
.de PP
.sp
..
.               CHAPTER
.de CH
.br
.ne 15
.sp 3
.in 0
\\fB\\$1\\fR
.in 5
.PP
..
.               SUBCHAPTER
.de SH
.br
.ne 10
.sp
.in 5
\\fB\\$1\\fR
.in 10
.PP
..
.               INDENT START
.de IS
.sp
.in +5
..
.               INDENT END
.de IE
.in -5
.sp
..
.               DOUBLE INDENT START
.de DS
.sp
.in +5
.ll -5
..
.               DOUBLE INDENT END
.de DE
.ll +5
.in -5
.sp
..
.               EQUATION START
.de EQ
.sp
.nf
..
.               EQUATION END
.de EN
.fi
.sp
..
.               TEST
.de TT
.ti -5
Test~\\$1:~
.br
..
.               IMPLEMENTATION 1
.de I1
.br
Implementation~1:
..
.               IMPLEMENTATION 2
.de I2
.br
Implementation~2:
..
.de CS
.br
~-~\\
..
.br
.fi
.sp 5
.ce
\fBPascal Validation Suite Report\fR
.CH "Pascal processor identification"
The ACK-Pascal compiler produces code for an EM machine
as defined in [1].
It is up to the implementor of the EM machine whether errors like
integer overflow, undefined operand and range bound error are recognized or not.
Therefore it depends on the EM machine implementation whether these errors
are recognized in Pascal programs or not.
The validation suite results of all known implementations are given.
.PP
There does not (yet) exist a hardware EM machine.
Therefore, EM programs must be interpreted, or translated into
instructions for a target machine.
The following implementations currently exist:
.IS
.I1
an interpreter running on a PDP-11 (using UNIX).
The normal mode of operation for this interpreter is to check
for undefined integers, overflow, range errors etc.
.sp
.I2
a translator into PDP-11 instructions (using UNIX).
Less checks are performed than in the interpreter, because the translator
is intended to speed up the execution of well-debugged programs.
.IE
.CH "Test Conditions"
Tester: E.G. Keizer
.br
Date: October 1983
.br
Validation Suite version: 3.0
.PP
The final test run is made with a slightly
modified validation suite.
.SH "Erroneous programs"
Some test did not conform to the standard proposal of February 1979.
It is this version of the standard proposal that is used
by the authors of the validation suite.
.IS
.TT 6.6.3.7-4
The semicolon between high and integer on line 17 is replaced
by a colon.
.sp
.TT 6.7.2.2-13
The div operator on line 14 replaced by mod.
.CH "Conformance tests"
Number of tests passed = 150
.br
Number of tests failed = 6
.SH "Details of failed tests"
.IS
.TT 6.1.2-1
Character sequences starting with the 8 characters 'procedur'
or 'function' are
erroneously classified as the word-symbols 'procedure' and 'function'.
.sp
.TT 6.1.3-2
Identifiers identical in the first eight characters, but
differing in ninth or higher numbered characters are treated as
identical.
.sp
.TT 6.5.1-1
ACK-Pascal requires all formal program parameters to be
declared with type \fIfile\fP.
.sp
.TT 6.6.6.5-1
Gives run-time error eof seen at call to eoln.
A have a hunch that this is a error in the suit.
.sp
.TT 6.6.4.1-1
Redefining the names of some standard procedures leads to incorrect
behaviour of the runtime system.
In this case it crashes without a sensible error message.
.sp
.TT 6.9.3.5.1-1
This test can not be translated by our compiler because two
non-identical variables are used in the same block with the same first eight
characters.
The test passed after replacement of one of those names.
.IE
.CH "Deviance tests"
Number of deviations correctly detected = 120
.br
Number of tests not detecting deviations = 20
.SH "Details of deviations"
The following tests are compiled without a proper error
indication although they do
not conform to the standard.
.IS
.TT 6.1.6-5
ACK-Pascal allows labels in the range 0..32767.
A warning is produced when testing for deviations from the
standard.
.sp
.TT 6.1.8-5
A missing space between a number and a word symbol is not
detected.
.sp
.TT 6.2.2-8
.TT 6.3-6
.TT 6.4.1-3
.TT 6.6.1-3
.TT 6.6.1-4
Undetected scope error. The scope of an identifier should start at the
beginning of the block in which it is declared.
In the ACK-Pascal compiler the scope starts just after the declaration,
however.
.sp
.TT 6.4.3.3-7
The values of fields from one variant are accessible from
another variant.
The correlation is exact.
.sp
.TT 6.6.3.3-4
The passing as a variable parameter of the selector of a
variant part is not detected.
A runtime error is produced because the variant selector is not
initialized.
.sp
.TT 6.8.2.4-2
.TT 6.8.2.4-3
.TT 6.8.2.4-4
.TT 6.8.2.4-5
.TT 6.8.2.4-6
The ACK-Pascal compiler does not restrict the places from where
you may jump to a label by means of a goto-statement.
.sp
.TT 6.8.3.9-5
.TT 6.8.3.9-6
.TT 6.8.3.9-7
.TT 6.8.3.9-16
There are no errors produced for assignments to a variable
in use as control-variable of a for-statement.
.TT 6.8.3.9-8
.TT 6.8.3.9-9
Use of a controlled variable after leaving the loop without
intervening initialization is not detected.
.IE
.CH "Error handling"
The results depend on the EM implementation.
.sp
Number of errors correctly detected =
.in +5
.I1
32
.I2
17
.in -5
Number of errors not detected =
.in +5
.I1
21
.I2
36
.in -5
Number of errors incorrectly detected =
.in +5
.I1
2
.I2
2
.in -5
.SH "Details of errors not detected"
The following test fails because the ACK-Pascal compiler only
generates a warning that does not prevent to run the tests.
.IS
.TT 6.6.2-8
A warning is produced if there is no assignment to a function-identifier.
.IE
With this test the ACK-Pascal compiler issues an error message for a legal
construct not directly related to the error to be detected.
.IS
.TT 6.5.5-2
Program does not compile.
Buffer variable of text file is not allowed as variable
parameter.
.IE
The following errors are not detected at all.
.IS
.TT 6.2.1-11
.I2
The use of an undefined integer is not caught as an error.
.sp
.TT 6.4.3.3-10
.TT 6.4.3.3-11
.TT 6.4.3.3-12
.TT 6.4.3.3-13
The notion of 'current variant' is not implemented, not even if a tagfield
is present.
.sp
.TT 6.4.5-15
.TT 6.4.6-9
.TT 6.4.6-10
.TT 6.4.6-11
.TT 6.5.3.2-2
.I2
Subrange bounds are not checked.
.sp
.TT 6.4.6-12
.TT 6.4.6-13
.TT 6.7.2.4-4
If the base-type of a set is a subrange, then the set elements are not checked
against the bounds of the subrange.
Only the host-type of this subrange-type is relevant for ACK-Pascal.
.sp
.TT 6.5.4-1
.I2
Nil pointers are not detected.
.sp
.TT 6.5.4-2
.I2
Undefined pointers are not detected.
.sp
.TT 6.5.5-3
Changing the file position while the window is in use as actual variable
parameter or as an element of the record variable list of a with-statement
is not detected.
.sp
.TT 6.6.2-9
An undefined function result is not detected,
because it is never used in an expression.
.sp
.TT 6.6.5.3-6
.TT 6.6.5.3-7
Disposing a variable while it is in use as actual variable parameter or
as an element of the record variable list of a with-statement is not detected.
.sp
.TT 6.6.5.3-8
.TT 6.6.5.3-9
.TT 6.6.5.3-10
It is not detected that a record variable, created with the variant form
of new, is used as an operand in an expression or as the variable in an
assignment or as an actual value parameter.
.sp
.TT 6.6.5.3-11
Use of a variable that is not reinitialized after a dispose is
not detected.
.sp
.TT 6.6.6.4-4
.TT 6.6.6.4-5
.TT 6.6.6.4-7
.I2
There are no range checks for pred, succ and chr.
.sp
.TT 6.6.6.5-6
ACK-Pascal considers a rewrite of a file as a defining
occurence.
.sp
.TT 6.7.2.2-8
.TT 6.7.2.2-9
.TT 6.7.2.2-10
.TT 6.7.2.2-12
.I2
Division by 0 or integer overflow is not detected.
.sp
.TT 6.8.3.9-18
The use of the some control variable in two nested for
statements in not detected.
.sp
.TT 6.8.3.9-19
Access of a control variable after leaving the loop results in
the final-value, although an error should be produced.
.sp
.TT 6.9.3.2-3
The program stops with a file not open error.
The rewrite before the write is missing in the program.
.sp
.TT 6.9.3.2-4
.TT 6.9.3.2-5
Illegal FracDigits values are not detected.
.CH "Implementation dependence"
Number of tests run = 14
.br
Number of tests incorrectly handled = 0
.SH "Details of implementation dependence"
.IS
.TT 6.1.9-5
Alternate comment delimiters are implemented
.sp
.TT 6.1.9-6
The equivalent symbols @ for ^, (. for [ and .) for ] are not
implemented.
.sp
.TT 6.4.2.2-10
Maxint = 32767
.sp
.TT 6.4.3.4-5
Only elements with non-negative ordinal value are allowed in sets.
.sp
.TT 6.6.6.1-1
Standard procedures and functions are not allowed as parameters.
.sp
.TT 6.6.6.2-11
Details of the machine characteristics regarding real numbers:
.IS
.nf
beta =       2
t =         56
rnd =        1
ngrd =       0
machep =   -56
negep =    -56
iexp =       8
minexp =  -128
maxexp =   127
eps =     1.387779e-17
epsneg =  1.387779e-17
xmin =    2.938736e-39
xmax =    1.701412e+38
.fi
.IE
.sp
.TT 6.7.2.3-3
.TT 6.7.2.3-4
All operands of boolean expressions are evaluated.
.sp
.TT 6.8.2.2-1
.TT 6.8.2.2-2
The expression in an assignment statement is evaluated
before the variable selection if this involves pointer
dereferencing or array indexing.
.sp
.TT 6.8.2.3-2
Actual parameters are evaluated in reverse order.
.sp
.TT 6.9.3.2-6
The default width for integer, Boolean and real are 6, 5 and 13.
.sp
.TT 6.9.3.5.1-2
The number of digits written in an exponent is 2.
.sp
.TT 6.9.3.6-1
The representations of true and false are (~true) and (false).
The parenthesis serve to indicate width.
.IE
.CH "Quality measurement"
Number of tests run = 60
.br
Number of tests handled incorrectly = 1
.SH "Results of tests"
Several test perform operations on reals on indicate the error
introduced by these operations.
For each of these tests the following two quality measures are extracted:
.sp
.in +5
maxRE:~~maximum relative error
.br
rmsRE:~~root-mean-square relative error
.in -5
.sp 2
.IS
.TT 1.2-1
.I1
25 thousand Whetstone instructions per second.
.I2
169 thousand Whetstone instructions per second.
.sp
.TT 1.2-2
The value of (TRUEACC-ACC)*2^56/100000 is 1.4 .
This is well within the bounds specified in [3].
.br
The GAMM measure is:
.I1
238 microseconds
.I2
26.3 microseconds.
.sp
.TT 1.2-3
The number of procedure calls calculated in this test exceeds
the maximum integer value.
The program stops indicating overflow.
.sp
.TT 6.1.3-3
The number of significant characters for identifiers is 8.
.sp
.TT 6.1.5-8
There is no maximum to the line length.
.sp
.TT 6.1.5-9
The error message "too many digits" is given for numbers larger
than maxint.
.sp
.TT 6.1.5-10
.TT 6.1.5-11
.TT 6.1.5-12
Normal values are allowed for real constants and variables.
.sp
.TT 6.1.7-14
A reasonably large number of strings is allowed.
.sp
.TT 6.1.8-6
No warning is given for possibly unclosed comments.
.sp
.TT 6.2.1-12
.TT 6.2.1-13
.TT 6.2.1-14
.TT 6.2.1-15
.TT 6.5.1-2
Large lists of declarations are possible in each block.
.sp
.TT 6.4.3.2-6
An 'array[integer] of' is not allowed.
.sp
.TT 6.4.3.2-7
.TT 6.4.3.2-8
Large values are allowed for arrays and indices.
.sp
.TT 6.4.3.3-14
Large amounts of case-constant values are allowed in variants.
.sp
.TT 6.4.3.3-15
Large amounts of record sections can appear in the fixed part of
a record.
.sp
.TT 6.4.3.3-16
Large amounts of variants are allowed in a record.
.TT 6.4.3.4-4
Size and speed of Warshall's algorithm depend on the
implementation of EM:
.IS
.I1
.br
size: 122 bytes
.br
speed: 5.2 seconds
.sp
.I2
.br
size: 196 bytes
.br
speed: 0.7 seconds
.IE
.TT 6.5.3.2-3
Deep nesting of array indices is allowed.
.sp
.TT 6.5.3.2-4
.TT 6.5.3.2-5
Arrays can have at least 8 dimensions.
.sp
.TT 6.6.1-8
Deep static nesting of procedure is allowed.
.sp
.TT 6.6.3.1-6
Large amounts of formal parameters are allowed.
.sp
.TT 6.6.5.3-12
Dispose is fully implemented.
.sp
.TT 6.6.6.2-6
Test sqrt(x): no errors.
The error is within acceptable bounds.
.in +5
maxRE:~~2~**~-55.50
.br
rmsRE:~~2~**~-57.53
.in -5
.sp
.TT 6.6.6.2-7
Test arctan(x): may cause underflow or overflow errors.
The error is within acceptable bounds.
.in +5
.br
maxRE:~~2~**~-55.00
.br
rmsRE:~~2~**~-56.36
.in -5
.sp
.TT 6.6.6.2-8
Test exp(x): may cause underflow or overflow errors.
The error is not within acceptable bounds.
.in +5
maxRE:~~2~**~-50.03
.br
rmsRE:~~2~**~-51.03
.in -5
.sp
.TT 6.6.6.2-9
Test sin(x): may cause underflow errors.
The error is not within acceptable bounds.
.in +5
maxRE:~~2~**~-38.20
.br
rmsRE:~~2~**~-43.68
.in -5
.sp
Test cos(x): may cause underflow errors.
The error is not within acceptable bounds.
.in +5
maxRE:~~2~**~-41.33
.br
rmsRE:~~2~**~-46.62
.in -5
.sp
.TT 6.6.6.2-10
Test ln(x):
The error is not within acceptable bounds.
.in +5
maxRE:~~2~**~-54.05
.br
rmsRE:~~2~**~-55.77
.in -5
.sp
.TT 6.7.1-3
.TT 6.7.1-4
.TT 6.7.1-5
Complex nested expressions are allowed.
.sp
.TT 6.7.2.2-14
Test real division:
The error is within acceptable bounds.
.in +5
maxRE:~~0
.br
rmsRE:~~0
.in -5
.sp
.TT 6.7.2.2-15
Operations of reals in the integer range are exact.
.sp
.TT 6.7.3-1
.TT 6.8.3.2-1
.TT 6.8.3.4-2
.TT 6.8.3.5-15
.TT 6.8.3.7-4
.TT 6.8.3.8-3
.TT 6.8.3.9-20
.TT 6.8.3.10-7
Static deep nesting of function calls,
compound statements, if statements, case statements, repeat
loops, while loops, for loops and with statements is possible.
.sp
.TT 6.8.3.2-2
Large amounts of statements are allowed in a compound
statement.
.sp
.TT 6.8.3.5-12
The compiler requires case constants to be compatible with
the case selector.
.sp
.TT 6.8.3.5-13
.TT 6.8.3.5-14
Large case statements are possible.
.sp
.TT 6.9-2
Recursive IO on the same file is well-behaved.
.sp
.TT 6.9.1-6
The reading of real values from a text file is done with
sufficient accuracy.
.in +5
maxRE:~~2~**~-54.61
.br
rmsRE:~~2~**~-56.32
.in -5
.sp
.TT 6.9.1-7
.TT 6.9.2-2
.TT 6.9.3-3
.TT 6.9.4-2
Read, readln, write and writeln may have large amounts of
parameters.
.sp
.TT 6.9.1-8
The loss of precision for reals written on a text file and read
back is:
.in +5
maxRE:~~2~**~-53.95
.br
rmsRE:~~2~**~-55.90
.in -5
.sp
.TT 6.9.3-2
File IO buffers without trailing marker are correctly flushed.
.sp
.TT 6.9.3.5.2-2
Reals are written with sufficient accuracy.
.in +5
maxRE:~~0
.br
rmsRE:~~0
.in -5
.IE
.CH "Level 1 conformance tests"
Number of test passed = 4
.br
Number of tests failed = 1
.SH "Details of failed tests"
.IS
.TT 6.6.3.7-4
An expression indicated by parenthesis whose
value is a conformant array is not allowed.
.IE
.CH "Level 1 deviance tests"
Number of deviations correctly detected = 4
.br
Number of tests not detecting deviations = 0
.IE
.CH "Level 1 error handling"
The results depend on the EM implementation.
.sp
Number of errors correctly detected =
.in +5
.I1
1
.I2
0
.in -5
Number of errors not detected =
.in +5
.I1
0
.I2
1
.in -5
.SH "Details of errors not detected"
.IS
.TT 6.6.3.7-9
.I2
Subrange bounds are not checked.
.IE
.CH "Level 1 quality measurement"
Number of tests run = 1
.SH "Results of test"
.IS
.TT 6.6.3.7-10
Large conformant arrays are allowed.
.IE
.CH "Extensions"
Number of tests run = 3
.SH Details of test failed
.IS
.TT 6.1.9-7
The alternative relational operators are not allowed.
.sp
.TT 6.1.9-8
The alternative symbols for colon, semicolon and assignment are
not allowed.
.sp
.TT 6.8.3.5-16
The otherwise selector in case statements is not allowed.
.IE
.CH "References"
.ti -5
[1]~~\
A.S.Tanenbaum, E.G.Keizer, J.W.Stevenson, Hans van Staveren,
"Description of a machine architecture for use with block structured
languages",
Informatica rapport IR-81.
.ti -5
[2]~~\
ISO standard proposal ISO/TC97/SC5-N462, dated February 1979.
The same proposal, in slightly modified form, can be found in:
A.M.Addyman e.a., "A draft description of Pascal",
Software, practice and experience, May 1979.
An improved version, received March 1980,
is followed as much as possible for the
current ACK-Pascal.
.ti -5
[3]~~\
B. A. Wichman and J du Croz,
A program to calculate the GAMM measure, Computer Journal,
November 1979.