ack/doc/LLgen/LLgen_NCER.n

.RP
.TL



Top-down Non-Correcting Error Recovery 
 in LLgen
.AU
Arthur van Deudekom
Peter Kooiman
.AI
Department of Mathematics and Computer Science
Vrije Universiteit 
Amsterdam





Supervised by
.AU
dr. D. Grune
.AI
Department of Mathematics and Computer Science
Vrije Universiteit
Amsterdam

.AB
This paper describes the design and implementation of a parser
generator with non-correcting error recovery based on the extended LL(1)
parser generator LLgen. It describes a top-down algorithm for implementing 
this error recovery technique that can handle left-recursive grammars. 
The parser generator has been tested with several existing ACK-compilers, 
among which C and Modula-2. Various optimizations have been tried and are
discussed in this paper. 
.AE
.LP
.nr PS 12
.nr VS 14

.NH
Introduction
.EQ
delim $$
.EN

.nr PS 10
.nr VS 12
.RS
.LP
One of the trickier problems in constructing parser-generators is what
to do when the input to the generated parser is not well formed. Several
approaches are known, most of which are `correcting', meaning that they
modify the input to make it correct. However, in most cases there are
several possible corrections, and often the one chosen will turn out
to be the wrong one. As a result of such an incorrect choice, spurious error 
messages can occur. Every programmer knows from experience how the omission 
of a single `)' can on occasion lead to pages of error messages. 

.LP
A radically different approach is to just discard all the input up to
and including the offending token, and start with a clean slate at the
token following the offending one. [RICHTER] describes how
this idea can be used to construct a non-correcting error recovery system
that will never introduce spurious error messages. It is, however,
possible that errors are overlooked.

.LP
In this paper we describe the incorporation of this non-correcting error
recovery into LLgen, an existing LL(1) parser generator.
In this introduction, we will describe in detail this non-correcting error
recovery technique, give an overview of LLgen and how it handles
errors, and finally describe how we have incorporated noncorrecting
error recovery in LLgen.
.RE

.NH 2
Non-correcting syntax error recovery

.LP
Richter describes how syntax error recovery can be done
without making any corrections to the input text. Richter gives three
reasons why recovery without correction is desirable:

.IP 1
In most cases there are many possible corrections, the choice among which
will severely influence the further processing of the input. Thus, the
probability of selecting the right correction is not high.

.IP 2
The harm done by selecting the wrong correction is often unlimited.

.IP 3
The loss of information to the user of a non-correcting recovery technique
need not be grave.

.LP
The non-correcting technique described by Richter can be summarized as
follows: When a syntax-error has occurred, the input up to and including the
erroneous symbol is discarded; the remainder of the 
input is processed by a substring parser of the input
language, that is a parser that recognizes any substring of a string in the input 
language. When the substring parser detects a syntax error, the offending 
symbol is reported as another error, and the input up to and including the 
erroneous symbol is discarded. The process is then repeated with the remaining input, possibly
finding other syntax errors, until all the input is scanned.
This process yields what Richter calls a 
.I 
suffix analysis 
.R
of an input string. Formally, given an input string 
.I x
, suffix analysis produces a set of strings $w sub k$ and a set of symbols
$ a sub k$ such that
.br

.IP
$x~ =~ w sub 0 a sub 0 w sub 1 a sub 1~...w sub n-1 a sub n-1 w sub n$
.LP
and such that:
.br
.IP
 $w sub 0$ is the longest prefix of $x$ that is  a prefix of
a string in the input language L, formally: there is a string $y$ such that 
$w sub 0 y$ is in  L, but there is no string $z$ such that $w sub 0 a sub 0 z$
is in L;
.IP
For $0 < k < n$, $w sub k$ is a longest substring of $x$ that is also a
substring of a string in L, formally there are strings $u$ and $v$ such that 
$u w sub k v$ is in L, but there are no strings $y$ en $z$ such that 
$y w sub k a sub k z$ is in L; 
.IP
$w sub n$ is a substring of $x$ 
that is a substring of a string in L, formally:
there exist $u$ and $v$, such that $u w sub n v$ is in L. Note that
$w sub n$ need not be a suffix of a string in L, if $x$ represents incomplete
input $w sub n$ is not a suffix of a string in L.

.LP
Now, the $a sub k$ indicate points at which an error is detected. The
"real" error need not be at $a sub k$, it can have occurred anywhere
within $w sub k a sub k$.
In his paper, Richter shows that, although this method may miss errors, it 
will never introduce spurious errors.

.LP
For implementing the technique, a parser that recognizes any
substring of the input language is needed. If we confine ourselves to
syntactical analysis, it is sufficient to construct a substring
recognizer. Richter himself does not give a practical construction, but
[CORMACK] describes how a LR substring parser can be constructed
that handles BC-LR(1,1) grammars. In this paper, we describe the construction
of a LL substring recognizer that can handle any grammar. Furthermore,
our recognizer is actually a suffix-recognizer, that is, a recognizer that 
recognizes any suffix of a string in the input language. Our suffix recognizer has the
correct-prefix property, 
meaning that it detects the first syntax error as early as possible
in a left-to-right scan of the input. Specifically, if the input language
is L and the invalid input is $x$ , it finds a string $w$ and an input symbol 
$a$ such that $x = way$  , there is a string $z$ such that $wz$
is in L, and there is no string $z$ such that $waz$ is in L.
Because the suffix parser has this correct-prefix property, it can be
used as a substring parser, because it will detect the first input symbol that
is not part of a substring of the language. Because it is a suffix-recognizer,
it additionally will detect incomplete input, because in that case 
at the end of the input the parser will not be in an accepting state.

.NH 2
Overview of LLgen

.LP
LLgen is an extended LL(1) parser generator. For a complete description, 
see [GRUNE].
LLgen can actually handle grammars that are not LL(1), because it allows
the use of conflict-resolvers. In case of an LL(1) conflict, these resolvers
are used to statically or dynamically decide which rule to use. As we will see
later, this feature makes it necessary for the suffix-recognizer to 
handle grammars that are not LL(1). Semantic actions can occur anywhere
in the grammar rules, and they are executed when their position is 
reached during parsing. A typical LLgen rule looks like
.br
.IP
S:	A {
.I action
} B
.LP
where the action is a piece of C-code, that will be executed
when the parser is using the rule for S and has recognized A.

.LP
LLgen-generated parsers use correcting syntax error recovery, based on a
scheme designed by R\*:ohrich [ROEHRICH], inserting or deleting symbols at the point of error detection
until correct input results. This means that actions in the parser will
always be executed in an order that could also have resulted from
syntactically correct input, and most importantly, once a grammar-rule
is started it is guaranteed to be completed. This means that syntactic
errors can never result in inconsistencies for the actions. Actions
only have to deal with syntactically correct input. In a nutshell, the
error recovery in LLgen-parsers works as follows: Suppose the parser is
presented with correct input that breaks off before the end. The error
recovery mechanism now provides a continuation path, chosen in such a
way that all active rules are left as soon as possible. Effectively, the
continuation path is the `shortest way out'. The symbols on this path are
called `acceptable', and end-of-file is also `acceptable'. Furthermore, at
each point along this `shortest path' there can be other terminals that
would be correct; these are `acceptable' as well. Now, when an
error occurs, all symbols that are not acceptable are discarded, until 
an acceptable symbol appears in the input. The tokens on the path up to 
but not including the acceptable input symbol are inserted. 
From then on, normal parsing resumes.

.NH 2
Incorporation of non-correcting error recovery in LLgen

.LP
An important consideration in incorporating the non-correcting recovery
in LLgen was that correct programs should suffer as little as possible
in what regards compilation speed. Furthermore, the existing error
recovery method has the highly desirable property that rules that are
started will be finished too, thus ensuring that errors in the
input text will not cause inconsistencies in the semantic actions. We have
implemented the non-correcting error recovery in such a way that this 
property is preserved.

.LP
The way we have achieved these goals is by actually including
the suffix recognizer as a `second recognizer' in the generated parser. 
Correct programs are handled in the usual way by the parser, but if an error
occurs the following happens: instead of going to the standard error
recovery routine, the parser starts executing the non-correcting error
handler. This process continues, reporting all errors, until the
end of the input text is reached. Then, control is handed back to
the standard error recovery routine. This routine will now think
there is no more input, and thus start inserting tokens so as to construct
a `shortest way out'. This ensures that all rules that were started are
also finished, and no inconsistencies can occur in the semantic actions. 
However, this method does require some modifications to the error reporting 
routine. Normally, if the generated parser inserts a token, it reports 
this to the user, but in this case this is undesirable. The insertions only 
serve to maintain consistency in the semantic actions 
and do not signify errors, so reporting of insertions should be suppressed. 
.bp
.nr PS 12
.nr VS 14
.PS
boxwid = boxwid / 1.5
boxht = boxht / 1.5
arcrad = arcrad / 1.5
movewid = movewid / 1.5
moveht = moveht / 1.5
arrowwid = arrowwid / 1.5
arrowht = arrowht / 1.5
arrowhead = arrowhead / 1.5
linewid = linewid / 1.5
lineht = lineht / 1.5
.PE
.NH
The LL suffix parser

.nr PS 10
.nr VS 12
.RS
.LP
In this chapter, we describe the construction of the LL suffix parser.
The described parser is not restricted to LL(1) grammars, because the
presence of conflict resolvers in LLgen allows for more general grammars,
that may even be left-recursive. We start this chapter with a discussion
of the implications of conflict resolvers, and continue with descriptions
of the parser algorithm, the used data-structures,
the handling of left- and right recursion, and some possible optimizations.
.RE

.NH 2
LLgen conflict resolvers and their implications

.LP
In grammars that are nearly but not completely LL(1), conflicts
will arise in the two places where parsing decisions are made: the choice
of which alternative to start (`alternation conflicts') and the decision
to stop or continue a repeated item (`repetition conflicts'). In order to
allow LLgen to handle this type of grammar, the user can 
specify conflict resolvers in those places where conflicts arise.
These resolvers are Boolean expressions labeling an alternative,
and are evaluated when a conflict arises during parsing. If the 
expression evaluates to `true' the labeled alternative will be taken.
The Boolean expressions are expressions in C, and can consult
any information available at the point they occur.
However, if a syntactic error has occurred in the input, and the non-correcting
error recovery starts, we can no longer rely on the conflict resolvers to 
guide parsing decisions. The suffix recognizer is only concerned with
syntax, and will not execute any semantic actions. It recognizes suffices
of correct input, but does not know or care what prefix would make
the suffix a correct program; as a result, the information that conflict
resolvers could use is not available, because the semantic actions
that would build this information have not been executed.
Therefore, the information used by the conflict resolvers is no longer 
reliable, and the suffix parser needs to be able to handle the underlying
grammar without their help. In particular, it has to be able to handle
left-recursive grammars.  

.NH 2
The suffix parser algorithm

.LP
Our algorithm needs easy access to the grammar rules; in the description
we assume there is an efficient way to access the grammar rules. In 
the next chapter we will describe the details of the actual implementation.
For the moment, we will only consider grammars that are not left- or 
right-recursive. In the next section, we will discuss how the algorithm has to be adapted
to handle left- and right recursion. 

.LP
Suppose the grammar is G, and the input to the suffix recognizer is 
$a sub 0 a sub 1 ... a sub n-1 a sub n$. Remember that parsing is
always started by the `normal' LLgen generated parser. It's only after
a syntactic error has occurred that the suffix recognizer will be started.
The input to the suffix recognizer thus is the `tail' of the input, starting
at the first symbol after the position where the first syntax error was
found.

.LP
Now, in order to get parsing going again, the parser scans the grammar
for rules which contain symbol $a sub 0$ in the right hand side:
.br

	A:	$alpha ~ a sub 0 ~ beta$
.br

.LP
where $alpha$ and $beta$ represent a string of terminals and non-terminals,
possible empty. Now, for each of these rules found, and for any string  
$b sub 0 b sub 1$...$ b sub m$ that can be generated by $beta$ it holds that
$a sub 0 b sub o b sub 1$...$b sub m$ is a substring of some string in L.
This can be shown as follows, supposing that the start symbol is S and
S $-> sup * gamma$  A $delta$:
.br

S $-> sup * gamma$ A $delta$ $-> sup * gamma ~ alpha ~ a sub 0 beta ~ delta
-> sup * gamma ~ alpha ~ a sub 0 b sub 0 b sub 1$...$b sub m delta$

.br
Of course, there may very well be more than one such string
$b sub 1 b sub 2$..$b sub m$, and one of these strings can be empty as well, if
$beta$ can produce empty. Now, in what we will call the 
.I 
predicting phase
.R
the algorithm will
produce all possible symbols $b sub 0$. Then, in what we will call the
.I 
accepting phase
.R
these symbols  are matched against
the input, and those not matching are discarded. Then, entering the next
predicting phase, the algorithm will produce
all symbols $b sub 1$, and match them against the next input symbol in
the subsequent accepting phase,
etc. In case one of the strings $b sub 0$...$b sub m$ is empty, or
the end of one of the strings is reached, some way to continue is
needed; we will discuss this later. First let's see how the
algorithm produces the strings $b sub 0$...$b sub m$ .

.LP
For each rule in the grammar of the form
.br

	A:	$alpha a sub 0 W sub 1 W sub 2$...$W sub p$
.br

with each $W sub k$ a terminal or nonterminal, a 
.I
prediction graph 
.R
is created that looks like this:

.PS
down; box "$a sub 0$"; arrow; box "$W sub 1$"; arrow
box "$W sub 2$"; arrow dashed; box "$W sub p$"
arrow; box "END" "$[A]$"
.PE

.LP
The bottom element of these prediction graphs is an end-marker containing the
left-hand side of the rule used. All these graphs have $a sub 0$ on top, and
this $a sub 0$ is matched against the $a sub 0$ in the input in the
accepting phase that follows, removing the
$a sub 0$ from the graph. If the prediction graph is now empty, we have to find a way 
to continue;  this case is treated later. First we will consider what to do if
the prediction graph is not empty. There are two possibilities: either $W sub 1$ is a 
terminal, or it is a nonterminal. If it is a terminal, we are finished for 
the moment; if not, the algorithm scans for rules of the form
.br

	$W sub 1$:	$U sub 1 U sub 2$...$U sub i$
.br

.LP
with each $U sub k$ a terminal or nonterminal. Now, the algorithm substitutes 
the top of the prediction graph with the right-hand sides 
of all the rules found. Because there can be more than one rule, the
prediction graph can now become a DAG (Directed Acyclic Graph).
Supposing there are two rules with $W sub 1$ in the LHS:

.br

	$W sub 1$:	$U sub 1 U sub 2$...$U sub i$
.br
	$W sub 1$:	$V sub 1 V sub 2$...$V sub j$

.LP
the prediction graph will now look like this:

.PS
B1: box "$U sub 1$"
move 
B2: box "$V sub 1$"
arrow dashed down from bottom of B1
B3: box "$U sub i$"
arrow dashed down from bottom of B2
B4:box "$V sub j$"
move to 0.5 <B3.se, B4.sw>
down;move
B5:box "$[W sub 1 ]$"
arrow dashed;
box "$W sub p$"
arrow;
box "END" "$[A]$"
arrow from B3.bottom to B5.top
arrow from B4.bottom to B5.top
.PE

.LP
The graph element representing $W sub 1$ is left in the stack, the
notation $[W sub 1 ]$ indicates it has been substituted. These substituted
element will from now on be ignored by the algorithm. The elements 
$U sub 1$ and $V sub 1$ are now `on top' of the prediction graph.

.LP
If $W sub 1$ can also produce empty, its successor in the prediction graph 
has to be processed
as well; the algorithm walks down the graph to this successor, and
there the process is repeated; if it is a terminal we are finished, else we 
substitute it with the right hand sides of its grammar rule. 
However, the element that we want to substitute now, say $W sub k$, cannot
be marked `substituted' just like that, because it can be on another
path, on which it cannot be substituted yet. Therefore, a copy of element
$W sub k$ is made, it is marked $[W sub k ]$, and an edge is created
from $[W sub k ]$ to the successor of $W sub k$. This produces graphs like
this:
.br
.PS
B1: box "$U sub 1$"
move
B2: box "$V sub 1$"
move
X1:box "$X sub 1$"
arrow dashed down from bottom of B1
B3: box "$U sub m$"
arrow dashed down from bottom of B2
B4:box "$V sub j$"
arrow dashed down from bottom of X1
Xj: box "$X sub j$"
move to 0.5 <B3.se, B4.sw>
down;move
B5:box "$[W sub 1 ]$"
arrow dashed;
B6: box "$W sub k$"
arrow
Wk1:box "$W sub k+1$"
arrow dashed
box "$W sub n$"
arrow;
box "END" "$[A]$"
arrow from B3.bottom to B5.top
arrow from B4.bottom to B5.top
move down from Xj.top;move;move;move
Wk: box "$[W sub k ]$"
arrow from Xj.bottom to Wk.top
arrow from Wk.bottom to Wk1.top
.PE

.LP
This process of substituting is repeated with all nonterminals that are
now on top of the prediction graph, until there are only terminals on top of 
the graph.
This completes the prediction phase of the algorithm, not taking into account
what to do if an END marker appears on top of the graph.
Now, the algorithm enters its accepting phase, in which
the terminals on top are compared with the next symbol in the input.
If a terminal in the graph matches the input, its element is deleted
from the graph, and the substitution process will continue with its
successors, in the next prediction phase.
If a terminal on top of the graph does not
match the input, the path it is on represents a `dead-end', which
does not need to be processed any further. The terminal is no longer
a `top', and the algorithm will not visit it again.

.LP
There is one tricky situation: consider again this graph:

.PS
B1: box "$U$"
move
B2: box "$a$"
move to 0.5 <B1.se, B2.sw>
down;move
B5:box "$W sub 1 $"
arrow dashed;
box "$W sub n$"
arrow;
box "END" "$[A]$"
arrow from B1.bottom to B5.top
arrow from B2.bottom to B5.top
.PE

.LP
Here, the algorithm is processing $W sub 1$ in the predicting phase, and
using some rule it has produced $a$ on top; there is another rule with
$W sub 1$ in its LHS which has produced nonterminal $U$ on top.
Now, suppose $U$ is a  nonterminal that can 
produce empty. Now, the algorithm starts substituting $U$, and walks
down $W sub 1$. What we definitely do not want 
is the algorithm to start substituting $W sub 1$ again, because then we
would loop forever. Therefore, if the algorithm starts processing 
element $W sub 1$ it should make it $[W sub 1 ]$ before it does
anything else. On entering the element 
for the second time in the prediction phase , it sees that it is already substituted, 
so there is nothing to do.
It then just walks to the successor of $W sub 1$ and
starts substituting it. This is correct, since the fact that the algorithm
enters an element for the second time in a prediction phase  means that the element
indirectly can produce the empty string, and thus its successor must
be substituted as well in the prediction phase.

.LP
It is easy to see that the substitution process will stop: the algorithm can 
only loop if  it starts processing an element for the second time in a
prediction phase, 
or if the  processing of an element eventually yields a graph with that 
same element on top. 
The first case cannot occur because the algorithm marks elements it is 
processing as `substituted' before it does anything else, meaning that those elements will not
be processed again; the second case can only occur if the grammar is 
left-recursive, which we assumed it was not. 

.LP
The algorithm simulates
left-most derivations of strings $a sub 0 b sub 0 b sub 1$..$b sub n$
starting from $a sub 0 W sub 1$..$W sub p$; as we showed before, if
the algorithm recognizes a string $a sub 0 b sub 0$..$b sub n$ that
string is a substring of some string in L. Conversely, because the
algorithm start out by using all rules of the form 
A:	$alpha a sub 0 beta$, and then proceeds to simulate all
possible left-most derivations, it will recognize all input
$a sub 0 b sub 0$... $b sub n$ that can be produced starting from
$a sub 0 beta$.

.LP
Now we will discuss what has to be done if an END marker appears as
top of the prediction graph. 
When this happens, it means that starting from some rule 
.br

	A:	$alpha a sub 0 beta$

.br
the algorithm has produced a leftmost-derivation of a string 
$a sub 0 b sub 1 .. b sub n$ starting from $a sub 0 beta$, or that $beta$ can produce
empty and the string so far is just $a sub 0$. The next step is to assume
that the have recognized A and that that some string produced by $alpha$
is part of the prefix that makes the suffix we are recognizing a 
correct string in L. Remember that in the END marker we kept record of
the LHS of the rule that has started the graph, and we will now use this
LHS to continue recognizing. What the algorithm does is scan for all
rules of the form:
.br

	B:	$gamma$ A $delta$
.br

with $gamma$ and $delta$ possibly empty strings of terminals and nonterminals.
The algorithm now starts a new component in the prediction graph, and if $delta$ is
$W sub 1 W sub 2$...$W sub n$ it looks like this:

.PS
down;box "$W sub 1$"; arrow
box "$W sub 2$"; line dashed; box "$W sub n$"
arrow; box "END" "$[B]$"
.PE

.LP
Note that the END marker now contains B, because we have started to match
a rule for B. If the $delta$ in the rule for B was empty, this just produces
and END marker with B in it; in this case, the process is just repeated
with all rules of the form:
.br

	C:	$zeta$ B $eta$
.br

.LP
etc, until we have a prediction graph with a nonterminal or terminal on top.
Now, the substitution algorithm is again applied over all nonterminals on
top, until every top contains a terminal. It is possible that during
substitution again an END marker will turn up; if this happens
we again scan for rules to continue with etc. 
This `continuation algorithm' can only loop if, when
trying to build a new prediction graph for matched symbol A, it produces an empty
graph with again matched symbol A. If this happens, the grammar was
(directly or indirectly) right-recursive, and we assumed that it was not.
Therefore, the algorithm will terminate. The terminals on top of the
new graph after applying this `continuation' algorithm are exactly those
that could follow the string $A sub 0 b sub 0$..$b sub n$ in a substring
of a string in L.
To see this, suppose we have `recognized' the rule
.br

	A:	$alpha a sub 0 beta$

.br
and $a sub 0 b sub 0 b sub 1$...$b sub n$ is the string produced from 
$a sub 0 beta$ by the algorithm. Now, using rule:
.br

	B:	$gamma$ A $delta$

.br
and supposing that S $->$ $zeta$ B $eta$ we get
.br

	S $->$ $zeta$ B $eta$ $->$ $zeta gamma$ A $delta$ $eta$ $->$ $zeta gamma a sub 0 b sub 0 b sub 1$ ... $b sub n$ $delta$ $eta$

.br
.LP
and thus any string produced by a derivation starting from
$delta$ can come right after $a sub 0 b sub 0$...$b sub n$ in a substring 
of some string in L. The algorithm will proceed to generate all these
strings starting from $delta$. If $delta$ produces empty, the above
is just repeated. Because in the `continuation' part
all possible rules are considered, the whole algorithm will recognize
all substrings of any string in L. In order to determine if we 
have actually recognized a suffix of some string in L, we need to
remember if within a predicting phase the `continuation' part of the algorithm has been run
on an END marker containing the start-symbol S;
if this is the case, then the input seen until now is a suffix of some string in L.
Formally, it means that there is a derivation starting from start symbol
$S$ such that if the
input seen until now is $a sub 0 a sub 1$..$a sub n$, then:
.br

	S $-> sup * alpha beta$ $-> sup * alpha a sub 0 a sub 1$..$a sub n$
.br

.LP
where $alpha$ can be empty, $beta$ is not empty.

.NH 2
The prediction graph data structure

.LP
The graphs that are produced by the suffix recognizer may grow extremely
large; to facilitate an efficient
implementation we have devised a way of keeping the size of the
data structure under control, in a way that is very similar to
the way described in [TOMITA].

.LP 
The basic idea is, that in a prediction phase of the algorithm, it is not
necessary to explicitly substitute each nonterminal every time it
turns up as a `top'; it is sufficient to do it once, because the
second substitution will produce exactly the same subgraph starting at
the substituted nonterminal. Here is an example:

.PS
down;box "$a$";arrow;box "A";arrow dashed;box "[B]";arrow
box "C";arrow dashed;box "END" "[X]"
move right from last box.e;
box "END" "[Y]";
arrow <- dashed up from last box.top;
box "D";arrow <- up from last box.top
box "B"
.PE

.LP
Here, in the left component of the graph, nonterminal B has been
substituted. Now, in the same prediction phase, the algorithm again runs into
B, now in the right component. There is no need to compute again
what the substitution will produce, it is exactly the part on top
of B in the left component. Therefore, all that is needed is:

.PS
down;box "$a$";arrow;box "A";arrow dashed;
B1: box "[B]";arrow 
box "C";arrow dashed;box "END" "[X]"
move right from last box.e;
box "END" "[Y]";
arrow <- dashed up from last box.top;
box "D"
arrow from B1.bottom to last box.top
.PE

So, when, in a prediction phase of the algorithm, a nonterminal is substituted,
the nonterminal is placed on a list, together with a pointer to
the substituted nonterminal. If in the same prediction phase a nonterminal that
is on the list becomes a top, all we need to do is place an edge
between the already substituted one and the successor of the top we are currently
processing. When a prediction phase is finished, the list is cleared.
There is one catch: if we consider again the last picture,
note that if nonterminal B can (directly or indirectly) produce empty,
it is also necessary to substitute D. However, it is not difficult to
determine if a nonterminal can produce empty. LLgen already computes
this information for each nonterminal.

.LP
Without this `joining together' of graph components, each
element in the graph has exactly one successor, except the END marker,
which has none.
Now that components get joined as described, an element can have any
number of successors. The recognizer algorithm now has to consider all
successors of a graph element instead of one.

.NH 2
Handling right recursion

.LP
The only problem right-recursive grammars cause in the algorithm is in the
`continuation' part; they can cause this part of the algorithm to loop
forever. As an example, consider:
.br

	A:	$alpha$ B
.br
	B:      $beta$ C
.br
	C:	$gamma$ A

.LP
Now suppose the `substitution' part of the algorithm has turned up
an END marker with nonterminal A in it. The continuation algorithm will
now produce:

.PS
box "END" "[A]";move;box "END" "[C]";move;box "END" "[B]";move
box "END" "[A]";move;box "END" "[C]"
.PE

.LP
etc. etc. However, a slight modification to the algorithm suffices
to eliminate this problem; within each prediction phase of the algorithm, we
simply maintain a list of nonterminals that have turned up in an
END marker. As soon as an END marker turns up whose nonterminal is
already in the list, we stop the `continuation' algorithm; the part
of the graph that would be produced by it already has been generated
by an earlier invocation of the algorithm in the same prediction phase.
At the end
of a prediction phase, when all heads are terminals, we clear the list.
This way, no looping can occur; even if the right recursion is
indirect, for instance if in the above example the rule for A had been
.br

	A:	$alpha$ B $delta$
.br
.LP
where $delta$ can produce empty, the algorithm still works; the substitution
of $delta$ will yield an END marker on top, and when trying to find
a continuation for LHS A the algorithm notices A is already on the list.


.NH 2
Handling left recursion

.LP
Left-recursion is, unfortunately, a much tougher problem than
right-recursion. The result of left-recursive grammar rules is that
the substitution algorithm never stops, because it can keep on building
the graph with the same set of rules without ever turning up a terminal.
One course of action would be to pre-process the grammar rules to
eliminate left-recursion; there are algorithms that eliminate direct
and indirect left-recursion. However, we have taken another course; by
allowing the produced graphs to contain loops, we can handle left
recursion without any modifications to the grammar. As soon as 
we come to the point that we want to substitute a nonterminal
which was already substituted earlier on the same path and in
the same prediction phase, we can 
make a link from the `older' nonterminal to the successor of
the `new' nonterminal. In this way we have constructed a loop
in the graph. As an example, suppose we have the following rules:
.br

D: A

A: B a

B: A | x

.br
Suppose also that we have nonterminal `D' on top of a stack. We 
now start substituting `D':

.PS
A: box "A"
move
X: box "x"
move to 0.5 <A.se, X.sw>
down
move
B: box "[B]"
arrow
box "a"
arrow
box "[A]"
arrow
box "[D]"
arrow dashed
box "END" "[S]"

arrow from A.s to B.n
arrow from X.s to B.n
 
.PE 

.LP
We now have an `A' on top of of the stack which was already 
substituted on the same path and also in the same prediction phase. To avoid
never ending substitution we make a loop as follows:

.PS
A: box "A" dashed
move
X: box "x"
move to 0.5 <A.se, X.sw>
down
move
B: box "[B]"
arrow
box "a"
arrow
A2: box "[A]"
arrow
box "[D]"
arrow dashed
box "END" "[S]"

arrow dashed from A.s to B.n
arrow from X.s to B.n
arc <- from B.w to A2.w
.PE

.LP
The dashed box with `A' in it means that it can be deleted, because
there is already an occurrence of it in the loop.

.LP
The most beautiful result of loops in graphs is 
that the original parsing algorithm needs only one minor change.
When the algorithm visits an element which has more than one 
outgoing edge the algorithm starts tracking down both paths,
just like before, only now there may be one or more backedges among
these edges, but the algorithm needs not to be aware of this fact.
The only difficulty with loops is that the algorithm might go into
a loop; it continues searching for terminals but it might happen
that there are no valid terminals in the loop. The solution to this
problem is not very difficult; just set a flag at all elements we
visit. When we reach an element which has this flag turned on, we
don't have to search any further.  At the end of the prediction phase, when we
have found all possible new heads, all flags are cleared.
Even if there are no loops in the
prediction graph, setting flags may be used as an optimization: 
it is possible that two paths come together at one point. In that situation
it is useless to scan for the second time the part of the graph which 
both paths have in common.

.NH 2
Some optimizations using reference counts

.LP
As explained in section 2.2, it is sometimes necessary to copy a
prediction graph element before substituting it. In order to determine
if a certain element has to be copied, it is convenient to maintain
a reference count in each graph element. This reference count keeps
track of the number of edges that enter an element. Now, when we want
to substitute an element with reference count not 0, we need to
copy it, because there is another path in the prediction graph that
contains the element we want to substitute, and on this other path
the element cannot be substituted yet.

.LP
Maintaining reference counts also enables us to perform another
optimization: remember that if, in a prediction phase, a terminal
is predicted that does not match the current inputsymbol, we from
then on just ignore the path in the graph starting at the terminal.
However, we can safely delete the terminal from the graph; furthermore,
all its successors in the prediction graph that have reference count
0 can be deleted as well, as can their successors with reference
count 0, etc. This way, we delete from the prediction graph
most elements that are no longer accessible, but not all of them; as will 
be explained in the next section, loops in the prediction graph
can cause problems. 

.NH 2
The algorithm to delete inaccessible loops

.LP
Deleting graph elements which are no longer reachable is not as easy
as it looks when there are loops in the graph, introduced by
the extension to the algorithm that handles left recursive grammars.
Suppose for example that we have a very simple loop as in the left 
picture below:

.PS
down
X: box "x" "(0)"
arrow 
box "[B]" "(2)"
arrow
box "a" "(1)"
arrow
box "[A]" "(1)"
arrow 
box "[D]" "(1)"
arc <- from 2nd box.w to 2nd last box.w

move right from X.ne
move 
move
move
move
move
move
down 
box "x" "(0)" dashed
arrow dashed
B: box "[B]" "(1)"
arrow
box "a" "(1)"
arrow
box "[A]" "(1)"
arrow
box "[D]" "(1)"
arc <- from B.w to 2nd last box.w
.PE

.LP
The number below each symbol indicates the reference count of that element.
Suppose now that we delete `x', then we have the situation depicted in the 
picture on the right. The loop consisting of `[B]', `a' and `[A]' is now
unreachable, so all these elements can be deallocated.
The reference count of `[B]' is 1, so it will not be deleted. To be precise
all elements in the loop have their reference counts on 1, and
consequently none of these will be deleted. But we stated earlier
that all elements of the loop cannot be reached anymore and that the
loop had to be deleted! In this example the reference counts of the
loop elements are all 1, but in more complex situations it is also 
possible that some of the elements have a reference count of more
than 1.

.LP
To solve this problem we present an algorithm, devised by E. Wattel, that
determines whether a loop can be deleted or not.
The algorithm consists of two parts. The first part of the algorithm goes as
follows: it presumes that all elements of the loop will indeed be
deleted. Every time it deletes an element it decreases the reference
count of all the successors of the element that are also member of the same 
loop.  How the algorithm knows which elements belong to the loop and which
do not will be explained later. The situation of the example above will now 
look like this:
 
.PS
down
box "[B]" "(0)" 
arrow 
box "a" "(0)"
arrow 
box "[A]" "(0)"
arrow
box "[D]" "(1)"
arc <- from 1st box.w to 2nd last box.w
.PE

.LP
The number below each symbol indicates again the reference count 
after we have applied the first part of the algorithm.

.LP
The second part of the algorithm checks and restores the 
reference counts of all members of the loop . When it finds 
out that one or more reference counts are not 0, it concludes 
that it is still possible to enter the loop in some way, and 
that it cannot be 
deleted yet. In the other case it reports that the loop can be 
deleted, which is also true in our example.

.LP
We will now formally describe the first part of the algorithm 
that finds all directed circuits from a given vertex, and determines if 
the vertices on those circuits can be deleted.
The algorithm works on prediction-graphs in which every edge that
is in a circuit is marked. Note that a marked edge may be in more than one circuit.
We will call this mark `C'.
The input to the algorithm is such a prediction graph, and a start vertex,
say A. The first part of the algorithm is:

.IP 1
Put the start vertex A on a list L; mark all edges `unused'
.IP 2
If L is empty, stop
.IP 3
For each vertex in list L, check if there are edges marked both C' and
`unused'. For each edge found, mark it `used', and traverse it to its
other endpoint; put this endpoint on a new list M, initially empty
.IP 4
Decrease the reference count of all vertices on M by 1
.IP 5
L := M; go to 2

.LP
It is clear that the algorithm will terminate: each edge is only traversed once,
and the number of edges is finite. We will now prove some properties of this
part of the algorithm.

.LP
.I
An edge is traversed by the algorithm if and only if it is on some
directed circuit $A ->$...$->A$.
.R
.br

The if-part is easy; if an edge $e$ connecting vertices $W$ and $V$ is on some directed circuit starting in
$A$, then there is a path $A ->$...$-> W -> V$; let $A ->$...$-> W -> V$ be a path
of minimum length from $A$ to $V$. If the length of the path from $A$ to
$W$ is $k$, then after turn $k$ of the algorithm $W$ will be on list L. To see
that this is the case, suppose that $W$ is not on list L after turn $k$;
this means that the edge entering $W$ was already marked used in a
previous turn, but then there would be a shorter path from $A$
to $W$, contradicting the assumption that the path is of
minimum length. The edge
$e$ is marked `C', because it is in a circuit; it is marked `unused', for if
it were marked used, there would be a shorter path from $A$ to $V$. So,
in turn $k + 1$, the edge $e$ will be traversed. 

.LP
On the other hand, suppose that an edge $e$ is traversed by the algorithm;
we will show by induction on the number of turns the algorithm has made
that $e$ is on a directed circuit $A->$..$->A$. In the first turn, all
edges from $A$ that are marked `C' are traversed, and clearly, if an edge
from $A$ is part of a circuit then that edge is part of a circuit from $A$ to $A$.
Now suppose that in turn $n+1$ an edge $e$ connecting vertices $W$ and
$V$ is traversed. This means the edge is 
marked `C', so it is part of some circuit. If there is a path from $V$ to $A$,
we can simply trace a circuit
$A->$...$-> W -> V -> $...$-> A$, and clearly $e$ is on a circuit from
$A$ to $A$. Now, suppose there is no path from $V$ to
$A$. We can always trace a circuit $W -> V ->$...$-> W$ because the
edge from $W$ to $V$ is part of a circuit; and by the
induction hypothesis there is a circuit $A ->$...$-> W ->$...$-> A$. We can
now make a `detour' at  $W$, yielding a circuit $A->$...$-> W -> V$...
$-> W ->$...$-> A$. This case is shown in the picture below.
So in either case $e$ is on a circuit from $A$ to $A$.

.PS
down; 
B1: box "A"; 
arrow dashed; 
B3: box dashed; 
arrow dashed;
B2: box "W";
arrow dashed; box dashed;
arc <- from B1.w to last box.w
arrow right "$e$" "C" from B2.e
box "V"; arrow dashed; box dashed;
arrow dashed  -> from last box.n to B3.e
.PE
  
.LP
.I
A vertex appears on list L if and only if it is on some directed
circuit from $A$ to $A$.
.R
.br

.LP
If a vertex is in such a circuit, there is an edge that enters it, which
is part of a circuit form $A$ to $A$; we already showed that this edge
is traversed by the algorithm, and thus the vertex will appear on list
L. Conversely, if a vertex appears on list L, then an edge entering
that vertex has been traversed by the algorithm; we showed that this
edge is part of a circuit from $A$ to $A$, and thus the vertex is
part of a circuit from $A$ to $A$.

.LP
.I
When the algorithm is finished, each vertex that is part of some
directed circuit from $A$ to $A$ has its reference count decreased by exactly
the number of edges entering it that are part of a directed circuit from $A$ to $A$.
.R
.br

.LP
Each edge that is part of some circuit from $A$ to $A$ is traversed
exactly once; the reference count of the endpoint is decreased
by one after an edge has been traversed. Thus, if a vertex is endpoint
of $k$ such vertices, its reference count is decreased by $k$.

.LP
.I
If the reference count of each of the vertices visited by the algorithm
is 0 after the algorithm has finised, all these vertices can be deleted; 
if the reference count is not zero for one or more of the visited
vertices, then none of them can be deleted.
.R
.br

.LP
Suppose all visited vertices have reference count 0; this means that
each of the vertices is only entered by edges that are on a circuit
from $A$ to $A$. Therefore, it holds that any path leading to any
of the visited vertices has to start in one of the visited vertices; there
is no path starting in an unvisited vertex to a visited one. Thus,
all the visited vertices are unreachable.
Conversely, if one of the visited vertices has reference count not zero,
then there is a path from an unvisited vertex to this vertex. Because from
the vertex with reference count non zero, we can get to $A$, and from $A$
we can get to any of the other vertices, all visited vertices are 
reachable.

.LP
The second part of the algorithm now checks if all reference counts are
zero, and if they are, it deletes all visited vertices. 


.NH 2
Marking loop elements

.LP
One point we have omitted so far is how the edges in the prediction
graph that are part of a loop get marked.
Basically, a loop can be detected:

	a. when it is made;
.br
	b. when we want to know about it.

.LP
The first approach checks if a loop is constructed
as soon as we join two paths in the graph, and if so, marks all
edges of the loop. The other approach does not do any checking when two
paths are joined together; it starts looking for loops when we want
to delete an element with reference count not 0, marking all edges
belonging to the loops it discovers. In practice it turns out that
we very often encounter elements that we would like to delete, but that have
reference count not 0, whereas the joining of paths occurs relatively 
infrequently. We therefore have chosen to check if a loop is created
when two paths in a prediction graph are joined. 

.LP
Now the question arises how to find and mark all edges of
the loop. For this problem we devised also an algorithm. 
Because we already know that there is an edge from the element on which 
the new path is connected to the successor of the joined element, the
algorithm only has to find a path from this last element back to the first one.
This can be done by a backtracking depth first search; to find a path from
one element to another we have to find a possible empty path 
from one of the successors of the first element to the last element. As
soon as we have found a path, we can mark all the edges on the path and also
the backedge as loop edges. In case that there is more than one path
back to the first element it is necessary that the algorithm continues
searching after it has found one path.

.LP
To avoid looping of this algorithm we have to set a flag at the elements
which are on the path already. When the algorithm is backtracking it can 
clear the flags at the elements it is leaving.

.LP
To speed up the searching process we can set flags at the edges we have already 
visited but did not lead back to the first element. When the algorithm
encounters such an edge it already knows that this edge is not worth
searching again and can be skipped. At the end of the algorithm these
flags have to be cleared again.

.LP
One might propose another optimization: as soon as
we reach an edge that is already marked as a loop edge, we
can stop searching for other loop edges. There is, however, 
a case in which this can go wrong. Imagine the following situation:

.PS
down
E: box "[E]"
arrow " C" ljust
D: box "[D]"
arrow " C" ljust
C: box "c"
arrow " C" ljust
box "b"
arrow " C" ljust
A: box "[A]"
arrow 
box "a"

move right from D
move right
J: box "[J]"
down
arrow from J.s " C" ljust
I: box "i"
arrow " C" ljust
H: box "[H]"
arrow from H.s to A.e 

arc <- from E.w to A.w 
move left from C
move left
"C"
arc -> from H.e to J.e
move right from I
move right
"C"

arrow dashed from E.s to J.n


.PE

What we have here is a prediction graph with two loops; all edges that belong 
to a loop are again marked with an `C'. Note that the edge between `[H]'
and `[A]' is not a loop edge. Suppose that `[J]' is not yet
completely substituted, i.e. there is another production rule for
J:
.br

J:	E

.br
The `E' on top of the right path is now joined with the `[E]' 
on the left path, which is depicted by the dashed arrow
between `[E]' and `[J]'. When we take a good look at the graph
we see that the two loops are merged into one. But that is not
the most important observation we have to make: not only the 
edge between `[E]' and `[J]' must be marked as a loop edge, but
also the edge between `[H]' and `[A]'! So it is not possible
to stop searching for loop edges as soon as we have found an
edge which was already marked as a loop edge. We have to continue
until we reach the element at which we started: `[E]'. So the 
optimization proposed above is incorrect.


.NH 2
Optimizations using FIRST and FOLLOW sets

.LP
In the algorithm as we have described it, every nonterminal on top of the graph
is substituted until only terminals remain on top; these terminals are
then matched against the current input symbol. However, by using
FIRST sets, we can save considerably on the number of computations 
necessary. Suppose one of the top elements of the graph is nonterminal A,
and the current inputsymbol is $a$. Then, it is of no use to substitute
A if terminal $a$ is not in FIRST(A), because then substituting A will
never produce $a$ on top of the graph. So, before substituting a
nonterminal we check if the current inputsymbol is in its FIRST set; if
it is not, we can declare the path the nonterminal is on a dead end, and
delete it, without having to perform the actual substitution. Of course, if
A can produce empty, we still have to consider its successor in the graph. 

.LP
Similarly, when we have an END marker on top, with nonterminal B in
it, and we consider using rule 
.br

	D:	$alpha$ B C $gamma$

.br
We first check if the current inputsymbol is in FIRST(C); if this is
not the case, there is no need to start a graph component with this
rule, because it will never produce the next inputsymbol on top.
Again, if C produces empty, we still have to evaluate the part of the
rule following C.

.LP
To circumvent the problems caused in the FIRST set optimization by
nonterminal that produce empty, we can also make use of FOLLOW-sets.
When substituting, if we encounter a nonterminal whose FIRST set does
not contain the current inputsymbol but which can produce empty,
we check if the current inputsymbol is in its FOLLOW set. If it is not,
there is no need to process its successor. Similarly, in case we
are processing an END marker as explained above, there is no need
to process the part of the rule following C if FIRST(C) does not
contain the input symbol, or C produces empty but the inputsymbol
is not in FOLLOW(C).
.bp
.nr PS 12
.nr VS 14

.NH
Test results

.nr PS 10
.nr VS 12
.RS

.LP
In this chapter, we discuss some test results that were obtained
by recompiling existing ACK compilers with the modified LLgen.
We tried several combinations of possible optimizations, including
`dumb' ones, like no optimization at all, not even deleting unreachable
prediction graph elements.
The incorporation of LLgen with non-correcting error recovery went
smoothly; only minor modifications to the Make-files were necessary.
Specifically, these modifications consisted of passing an extra
flag to LLgen, and including the new generated C-file Lncor.c in
the list of generated C-files. Also, the LLmessage error reporting
routine had to be adapted. We successfully recompiled the C, Modula-2
and Occam compilers; in the next sections, we discuss some test results
that were obtained with the Modula-2 and C compilers.

.RE
.LP
.NH 2
Performance

.LP
We will now present and discuss, with the aid of some 
diagrams, time and space measurements on the non-correcting error
recovery. We have measured the effect of various optimizations. 
These optimizations include the first-set optimization and the follow-set 
optimization. We also measured the effect of leaving out the loop-deletion
algorithm, regarding both time and space. We performed out measurements using 
C- and Modula-2-programs of three different sizes; one of approximately 
750 tokens, one of appr. 5000 tokens and one of appr. 15000 tokens. We have
chosen to represent the sizes of programs in the number of tokens instead of 
number of lines, because the number of tokens more realistically 
reflects the load the programs put on the error recovery mechanism. Also we give 
our time measurements in usertime instead of realtime, because realtime 
depends heavily on the load of the system, which usertime does not.
Our space measurements are based on the size of the prediction graphs.
Note that all files are entirely recognized by the non-correcting error 
recovery technique. We achieved this by putting a `1' at the beginning
of each file; because then each file starts with a syntax error LLgen 
is forced to continue with the non-correcting error recovery. 

.NH 3
Time and space measurements on the effect of the first-set optimization 

.LP
In the diagram below we show our time measurements we got from recognizing 
the C-programs both with and without first-set optimization. 

.G1
coord x 0, 17000 y 0, 65
ticks bot out at 750, 5000, 15000
label bot "Number of tokens"
label left "User Time" "(sec)" left .3
draw no_opt dashed 
draw first_opt dashed 

copy thru X
	times size +2 at $1, $2 
	times size +2 at $1, $3 
	next no_opt at $1, $2
	next first_opt at $1, $3
X until "XXX"

742	2.5	.9	
5010	16.3	5.8
14308	54.2	16.8
XXX

copy thru X "$1 $2" size -2 at 11000, $3 X until "XXX"
No optimization 55 
First-set optimization 20
XXX
.G2

.I
.ce
Time measurements of three C-programs with and without first-set optimization
.R

.LP
Notice the considerable time savings we 
get when the first-set optimization is turned on; a factor of slightly more than
3. Obviously this is an extremely useful optimization. On the other hand
we found there were no measurable time savings when using the follow-set
optimization; for that reason we did not chart the result of this optimization.
It seems that the time savings gained by the optimization are 
waisted again by the extra processing time needed. We conclude that 
this optimization is of little or no use when we want to save on time.

.LP
In the following picture the time measurements of three Modula-2 programs 
are given, again with and without first-set optimization. 

.G1
coord x 0, 17000 y 0, 65
ticks bot out at 750, 5000, 15000
label bot "Number of tokens"
label left "User Time" "(sec)" left .3
draw no_opt dashed 
draw first_opt dashed 
copy thru X
	times size +2 at $1, $2 
	times size +2 at $1, $3 
	next no_opt at $1, $2
	next first_opt at $1, $3
X until "XXX"

823	1.3	.6	
4290	7.6	3.5	
16530	30.5	14.3
XXX

copy thru X "$1 $2" size -2 at 13000, $3 X until "XXX"
No optimization 30 
First-set optimization 15
XXX
.G2

.I
.ce
Time measurements of three Modula-2-programs with and without first-set optimization
.R

.LP
From this picture we can conclude mainly the same as above; considerable
time savings when we use the first-set optimization; 
the factor is somewhat less, but still more than 2. Again we have omitted
the results of the follow-set optimization, for the same reason as before.

.LP
There is however one remarkable difference between the two languages: parsing
C-programs needs almost twice the time as parsing programs of comparable
sizes written in Modula-2. This can be explained by the fact that the 
C-grammar is far more complicated than that of Modula-2, and also the 
production rules are longer in C, so building, deleting and definitely
traversing the graph will consume more time.

.LP
Now we come to the space measurements of both C- and Modula-2 programs.
In the picture below we present the maximum sizes of the prediction graphs,
during the recognition of the three C-programs.
 
.G1
coord x 0, 17000 y 0, 18000 
ticks bot out at 750, 5000, 15000
label bot "Number of tokens"
label left "Maximum size of" "the prediction graph" "(bytes)"left .3
draw no_opt dashed 
draw first_opt dashed 
copy thru X
	times size +2 at $1, $2 
	times size +2 at $1, $3 
	next no_opt at $1, $2
	next first_opt at $1, $3
X until "XXX"

742	5568	10444
5010	7668	12664
14308	13636	17308
XXX

copy thru X "$1 $2" size -2 at 8000, $3 X until "XXX"
No optimization 16000 
First-set optimization 7000 
XXX
.G2

.I
.ce
Maximum sizes of the prediction graphs when recognizing three C-programs
.R

.LP
From this diagram we see that, although the prediction graphs
are smaller when the first-set optimization is used, the space savings are
not as spectacular as the time savings achieved by this optimization.

.LP
In Modula-2 the first-set optimization also causes a decrease in memory
usage. The savings are less than in C, but still about 1.5 Kb. Again 
this can be explained by the fact that the rules of the Modula-2 grammar 
are shorter than that of C.

.G1
coord x 0, 17000 y 0, 12000 
ticks bot out at 750, 5000, 15000
label bot "Number of tokens"
label left "Maximum size of" "the prediction graph" "(bytes)" left .3
draw no_opt dashed 
draw first_opt dashed 
copy thru X
	times size +2 at $1, $2 
	times size +2 at $1, $3 
	next no_opt at $1, $2
	next first_opt at $1, $3
X until "XXX"

823	5056	3292
4290	6420	4664
16530	11388	9632
XXX

copy thru X "$1 $2" size -2 at 8000, $3 X until "XXX"
No optimization 10000 
First-set optimization 4000 
XXX
.G2

.I
.ce
Maximum sizes of the prediction graphs when recognizing three Modula-2-programs
.R

.NH 3
Input that is recognized in quadratic time

.LP
The measurements presented may suggest that the time required to
recognize input depends linearly on the length of the input; however,
this is not always the case. When there are recursive rules in the
grammar, the time needed to recognize input that is produced by this
rules can become proportional to the square of the input length.
Consider this set of grammar rules:
.br
.nf

	S:	'{' A '}'
	A:	'a' A | $epsilon$

.fi
.LP
When the input is `{aaa....', the algorithm will produce the following 
prediction graphs: 

.PS
up; B1: box "END" "S"; arrow <- ;box "}";arrow <- ;box "A";arrow <- ;box "{";
move right from B1.se; move
up; B2: box "END" "S"; arrow <-; box "}"; arrow <-; box "[A]"; 
arrow <-; box "A"; arrow <-; box "a";
move right from B2.se; move
up; B3: box "END" "S"; arrow <-; box "}"; arrow <-; box "[A]";
arrow <-; box "[A]"; arrow <-; box "A"; arrow <-; box "a";
move right from B3.se;move
up; B4: box "END" "S"; arrow <-; box "}"; arrow <-; box "[A]";
arrow <-; box "[A]"; arrow <-; box "[A]"; arrow <- ; box "A"; arrow <-;box "a";
.PE

.LP
In each prediction phase, a new [A] appears on the prediction graph. However,
since A also produces empty, the prediction algorithm has to traverse all the
elements [A] until it finds the element `}'. In the first prediction phase,
there is one element [A], in the second there are two, etc, so in all
1 + 2 + 3 + ... + k = $k(k+1) over 2$ elements have to be traversed if 
there are k prediction phases, making this proportional to the square
of the input length. We constructed a parser with this simple input grammar
and measured the processing time the error recovery mechanism used.
In the following diagram the dashed line shows the processing time needed;
the dotted line is the curve $t = 13 n sup 2$. Clearly the processing time
is proportional to the square of the number of tokens.

.G1
coord x 0, 2100 y 0, 60 
ticks bot out at 500, 1000, 1500, 2000
label bot "Number of tokens"
label left "User Time" "(sec)" left .3
draw quad dashed 

copy thru X
        times size +2 at $1, $2 
        next quad at $1, $2
X until "XXX"

500  3.0     
1000 12.4
1500 28.6
2000 51.4 
XXX

draw dotted
for i from 0 to 2100 by 25 do { next at i, 0.000013 * i * i }
.G2

.LP
In the grammar used for the C compiler, array initializations are handled by a recursive
rule, so we would expect that the error recovery mechanism needs quadratic
processing time to recognize such an initialization; we made measurements on 
the processing time and indeed, the
processing time needed grows proportionally to the square of the size of the input, as the
next figure shows. Here, the processing times are about half of those in
the previous example; this is so because the recursion appears after two
tokens are recognized. Note that the algorithm only takes quadratic time
when it is recognizing input that is generated by a recursive grammar rule.
Other input is still recognized in linear time, regardless of the fact that
there are recursive grammar rules.

.G1
coord x 0, 5000 y 0, 85 
ticks bot out at 1150, 2400, 3600, 4800
label bot "Number of tokens"
label left "User Time" "(sec)" left .3
draw quad dashed 

copy thru X
        times size +2 at $1, $2 
        next quad at $1, $2
X until "XXX"

1150 5.1      
2400 20.3
3600 43.7 
4800 78.6
XXX
.G2

.LP
Unfortunately, there is no easy way to speed up the recognition of these
recursively defined language elements; they are caused by the substituted
tokens that are left in the prediction graph, and we cannot just delete those
`dummies' from the graph during a prediction phase because the `join' part of the
prediction algorithm depends on them. One could traverse the graph after
a prediction phase to delete the dummies, but then the processing
time needed to recognize non-recursively defined language elements would 
increase dramatically. However, we feel that in practice things
like large array initializations will not occur in hand-made programs; when
they occur, it is probably in computer-generated programs, which normally
will be correct anyway, meaning that the error recovery never sees them.
When testing such generated programs, one is likely
to use small test-cases, which are handled well by the error recovery.

.NH 3
Time measurements on the effect of leaving out the loop-deletion algorithm

.LP
We now show what effect the loop-deletion algorithm has on processing time. 
To put it another way: how much time can be saved when we turn off the 
loop-deletion algorithm. In the diagram below we give the measurements of 
the three C-programs; note that we do use the first-set optimization.

.G1
coord x 0, 17000 y 0, 22
ticks bot out at 750, 5000, 15000
label bot "Number of tokens"
label left "User Time" "(sec)" left .3
draw no_loop dashed 
draw loop dashed 
copy thru X
	times size +2 at $1, $2 
	times size +2 at $1, $3 
	next no_loop at $1, $2
	next loop at $1, $3
X until "XXX"

742	.9	.4	
5010	5.8	6.8
14308	16.8	20.5
XXX

copy thru X "$1 $2" size -2 at 11300, $3 X until "XXX"
With loop-deletion 20 
Without loop-deletion 9 
XXX
.G2

.I
.ce
Time measurements on processing three C-programs with and without the loop-deletion algorithm
.R

The diagram shows that the loop-deletion algorithm 
does not dramatically slow down the recognizing process. There is, however, 
a measurable time loss of \(+-25%. As we will see later, the loop-deletion
algorithm will turn out to be extremely useful in efficient use of memory 
when there are many loops in the graph.

The effect of the loop-detecion algorithm on parsing Modula-2 programs
is even less than with C-programs; in fact there is no measurable
time loss:

.G1
coord x 0, 17000 y 0, 15
ticks bot out at 750, 5000, 15000
label bot "Number of tokens"
label left "User Time" "(sec)" left .3
draw no_loop dashed 
draw loop dashed 
copy thru X
	times size +2 at $1, $2 
	times size +2 at $1, $3 
	next no_loop at $1, $2
	next loop at $1, $3
X until "XXX"

823	.6	.6
4290	3.5	3.8
16530	14.3	14.3
XXX

copy thru X "$1 $2" size -2 at 11800, $3 X until "XXX"
With loop-deletion 13 
Without loop-deletion 7
XXX
.G2

.I
.ce
Time measurements on processing three Modula-2-programs with and without a loop-deletion algorithm
.R 

There are at least two reasons for this; both result from the relative
simplicity of the Modula-2 grammar. The distance from a head to an 
end of stack marker is shorter than in C, and secondly Modula-2
causes fewer joins to occur than C, meaning that the loop marking algorithm
is run less often and when it is run it has fewer paths to search.


.NH 3
Space measurements on the effect of leaving out the loop-deletion algorithm

.LP
Clearly, to make any measurements on the space-usage effects of leaving out
the loop-deletion algorithm we need a program that causes the prediction
graph to contain loops; however, we have not been able to devise a C
or Modula-2 program that does this. In order to be able to make measurements,
we added an extra alternative to a rule of the C compiler grammar, making
it directly left-recursive. To make LLgen accept this new grammar, we
put a `%if' directive in the rule.

.LP
We have input our standard C test program consisting of 800 tokens to
the error recovery routine for this `doctored' C compiler, 
and compared the storage needed for the prediction graphs with the
loop deletion algorithm enabled with the storage needed when the
algorithm is disabled. With the loop-deletion algorithm enabled, the
maximum size of the prediction graph was 5576 bytes. When the loop
algorithm was disabled, the maximum size of the prediction graph
grew to 12676 bytes; furthermore, 12676 bytes of heap were allocated
for the prediction graph, but not deallocated again, because they were
in use by graph elements that were in inaccessible loops. The user-time
the program needed decreased only slightly, from 0.9 to 1.0 seconds. Given the
relatively small input program, this data suggests that when loops
are actually being made, the loop deletion algorithm is definitely 
worth the extra overhead it costs, considering the space
that would otherwise be occupied by inaccessible loops. To verify this,
we input the C program consisting of 15000 tokens to the compiler;
execution time increased from 17.3 to 21.1 seconds after enabling
the loop deletion algorithm, while the maximum size of the prediction graph
shrunk from 328664 to 13664 bytes. With the loop-deletion algorithm
disabled, 326720 bytes allocated for the graph were not deallocated again.
Again, given the relatively small increase in execution time and the
large reduction of memory usage, we feel that the loop-deletion
algorithm is useful enough to justify the overhead it creates.

.NH 2
Problems encountered

.LP
In this section we describe some of the problems we encountered
while testing the non-correcting error recovery.

.NH 3
The LLgen error reporting mechanism.

.LP
The parsers generated by LLgen call a user-supplied error reporting
routine, usually called LLmessage. This routine is called with an
integer parameter that is positive, zero or negative. When the parameter
is positive the parser has just inserted a token, whose
number is equal to the parameter; if it is zero, the parser 
has deleted a token whose number is in a global variable called LLsymb; if
it is negative, it means that LLgen expected end-of-file, but did not
find it. The routine LLmessage is supposed to print an error message,
and when a token is inserted, it should set all necessary attributes.

.LP
However, when non-correcting error recovery is used, the situation becomes slightly
different; when the parser inserts a token, it is only to keep the
semantic actions consistent, and does no longer signify an error.
However, the LLmessage routine still has to be called because the
attributes of the inserted token need to be set. Therefore, when
non-correcting error recovery is used, the LLmessage routine should not
print an error message when the parameter is positive, or else it will
print highly confusing error messages indeed. Furthermore, the
LLmessage routine will usually print a message like `token ... deleted' when
it is called with parameter equal to zero; however, when the non-correcting
error recovery is used, it is more appropriate to report something
like `token ... illegal', as the non-correcting error recovery does
not delete tokens. Finally, when an unexpected end-of-file is encountered,
LLgen normally just inserts the missing tokens and calls
LLmessage with the parameter equal to the token number;
when non-correcting error recovery is used we need a way to
actually report we have encountered an unexpected end-of-file. The
way we achieved this is by calling LLgen with parameter 0 and the
global variable LLsymb set to EOFILE when this situation occurs; the
routine LLmessage should print something like `unexpected end of file'
when it is called with parameter 0 and LLsymb is EOFILE. To facilitate
switching between correcting and non-correcting error recovery, the
file Lpars.h contains a statement `#define LLNONCORR' if non-correcting
error recovery is used.


.NH 3
Parsers being started in semantic actions

.LP
LLgen allows the programmer to define more than one nonterminal as the
start symbol of the input grammar; it will generate a parsing routine
for each of the start symbols. However, the error recovery code
is generated only once; it is shared by all parsers.
The programmer is free to call any
of the generated parsers whenever he wants; for instance, in the C-compiler
a separate parser for expressions in #if and #elsif statements is used. Whenever
the lexical analyzer encounters such a statement, it calls the expression
parser. It is also possible to call a parser in a semantic action of
another parser; in the MODULA-2 compiler a separate parser for 
definition modules is used. When the main parser encounters a 
FROM defmod IMPORT statement a semantic
actions opens the definition module defmod and starts the parser for
definition modules. 

.LP
The fact that subparsers can be started just about anywhere causes
problems when non-correcting error recovery is used. 
Suppose a parser calls another parser in a semantic action
to parse a separate input file. In the Modula-2 compiler, after 
seeing the FROM defmod IMPORT statement a semantic action opens
defmod and parses it; now, if a syntax error occurred before the
FROM IMPORT statement, the non-correcting error recovery will not
execute the action that opens and parses the definition module, but
it will not report an error either, because the statement 
FROM defmod IMPORT is part of the input language of the main parser.
However, suppose that during the parsing of a definition module
an error occurs; then, some semantic actions that would normally
be executed during parsing of the definition module will not have
taken place. When normal parsing is now resumed by the main parser,
after the non-correcting error recovery has finished with the
definition module, a lot of spurious semantic errors are likely to be
reported, because the semantic actions that would normally have been
executed during the definition module parsing have not been executed
by the error recovery. Therefore, it is desirable that the main parser
does not resume normal parsing, but instead continues with the non-correcting
error recovery as well. Any syntactic errors in the main program will
still be reported, but no spurious semantic errors will be reported
that way.

.LP
When the lexical analyzer calls other parsers, as is the case in
the ACK C compiler, recursive invocations of the non-correcting error
recovery routine can occur. This will happen if a parser starts the
error recovery, the error recovery calls the lexical analyzer, which
starts another parser that finds a syntax error and calls the
error recovery again. This is not really a problem, but is has
consequences for the implementation of the error recovery routine.

.LP
The worst case
occurs when two parsers are involved in parsing one input file, and
the secondary parser (e.g. an inline assembly parser) is called in a semantic
action of the main parser. Suppose now that the input text contains
a syntax error; after detecting this error, the parser starts the
non-correcting error recovery. This recovery does not execute any
semantic actions; therefore it will not start the subparser at those points
where the original LLgen generated parser would. As a result, parts
of the program that would be accepted by the subparser will now probably
be rejected as illegal, because the error recovery does not know it
should use another grammar to check these parts. This is a serious
problem, and we have devised and implemented two ways to solve it.

.LP
The first solution is based on the assumption that whenever a semantic
action occurs in the grammar, another parser can be started at that
point. Obviously, we have no way of knowing which semantic actions start
a parser and which don't, so we assume the worst.
Now, assume that in the grammar there are k symbols defined as
start symbols, say $W sub 1 , W sub 2 , ..., W sub k$. Each of these symbols
will cause LLgen to generate a parser that can be called in any
of the semantic actions of the grammar. We now introduce a new
symbol $X$, and a new grammar rule $X -> W sub 1 X | W sub 2 X | ... | 
W sub k X |
epsilon$.
In the grammar the error recovery algorithm uses, we insert this symbol
X at all positions where there are semantic actions in the original grammar,
so a rule $A -> alpha$ { action } $beta$ becomes $A -> alpha X beta$. As a
result, at each position in a grammar rule where a semantic action 
occurs, we now accept any input that would be accepted by any of the
parsers. Clearly, this solution is somewhat of a kludge, as it will
accept a lot of input that is not accepted by the original parser.
However, it is guaranteed to never give spurious error messages, because
whenever a parser would be started by the original parser, there now
is an $X$ in the grammar that produces all the strings that would be
accepted by that parser. We have implemented this solution, and found
it to be extremely slow, which of course was to be expected given the
number of semantic actions in the average grammar. Furthermore,
because each time a semantic action occurs in the grammar
a string accepted by any of the generated parsers is accepted, including
strings recognized by the currently running parser, error messages
become hard to interpret. As an example, consider the following
C program:
.br
.nf


	main()
	{
		int i, j;

		while (i < j 
           		j++;

		i = 1;
		j = 2;

	}


.fi
.LP
Clearly, there is a `)' missing in the while-statement;
however, if this program is input to the error recovery it will complain 
"} illegal", since after recognizing the
expression controlling the while the original parser starts  a 
semantic action, so the non-correcting recovery will accept a valid
C program at that point; after recognizing the three statements
following the while-statement as a separate program the
recognizer expects the missing `)', but gets `}' instead.

.LP
Our second solution is based on the observation that if we knew
which semantic actions can start other parsers, we would only
have to introduce the new symbol $X$ at those places where parsers
can get started. We have therefore extended LLgen with a new directive
%substart, which is used to indicate to the parser generator that
another parser may be started. The %substart is followed by the
startsymbols that will produce the parsers that can be called,
so %substart A, B, C; indicates that in the semantic action
following the directive the parsers produced by startsymbols
A, B, en C can be started. In the grammar used by the error
recovery, a new symbol $X$ will be introduced at this point,
along with a new rule $X -> AX | BX | CX | epsilon$. Of course, this
solution can still accept input that would not have been accepted
by original parser, for instance if a parser is started
conditionally, based on other semantic information. However, it
is a big improvement over the first solution, both in performance
and the input it accepts.

.NH 3
Syntactic errors being handled in semantic actions

.LP
A programmer may decide to handle certain syntactic  errors
in semantic actions, for instance because he is not satisfied with 
the standard error recovery. However, since the non-correcting error
recovery does not execute semantic actions, this may cause errors
to remain undetected. We encountered the following example in the ACK
Modula-2 compiler, in the grammar rule for assignment statement:
.br
.nf


	Assignment_statement:	lvalue
				[
				 	'='
					{
					 error(":= expected");
					}

					|

					':='
				]
				expression
				;

.fi
.LP
This works well in the original LLgen; however, statements like
`j=9' are not treated as syntactic, but as semantic errors.
The original LLgen generated parser
will print the (semantic) error message, but the non-correcting recovery
will not execute the semantic action and therefore the erroneous
input will be accepted.

.LP
To facilitate the incorporation of non-correcting error recovery in parsers 
that use this kind of `trick', we extended LLgen with the %erroneous
directive. The directive indicates to the non-correcting recovery 
mechanism that the token following it is not really part of the grammar.
When recognizing input, the error recovery will ignore tokens in the
grammar that have %erroneous in front of them. If in the example above,
the '=' is replaced with %erroneous '=', the non-correcting mechanism will
report an error when it sees a statement like 'j = 9'. See appendix B
for details about the implementation of the %erroneous directive. 

.LP
Another example is in the ACK C compiler. For some reason, the
grammar accepts function definitions without `()', so according
to the syntax a function definition can look like:
.br
.nf

	int func
	{
	  ....
	}
.fi

.LP
The absence of the `()', however, causes `func' to be entered in the
symbol table as non-function, and when the parser encounters the body
a semantic action will complain with the error message "Making function body
for non-function". This again will cause the non-correcting error
recovery to miss errors. Consider this piece of code:
.br
.nf

int i int j = 1;
{}

.fi

.LP
where apparently there's a `;' missing between the declarations
of i and j. The original LLgen-generated parser only gives semantic errors:
.br
.nf
"Making function body for non-function"
"j is not in parameter list"
"Illegal initialization of formal parameter, ignored"
.fi
.LP
As a result, the non-correcting error recovery will not report
any errors in this piece of code, because it does not execute the
semantic actions that recognize and report the error. Unfortunately,
due to the way the C-grammar is written, it is not possible to solve
this problem using a %erroneous directive; the part of the grammar 
that deals with declaratons would have to be rewritten so as to
syntactically reject functions without `()'. 

.NH 3
Semantic actions that read input

.LP
There are no restrictions on what a semantic action can do;
there is nothing to stop the programmer from writing a parser in such
a way that some of the input to the parser is processed by semantic
actions. Obviously, because the non-correcting error recovery does not
execute semantic actions, this kind of parser will not work at all
with the new error recovery. Ironically, LLgen itself is written in
such a fashion; {}-enclosed C-code in its input is processed by
a semantic action in the LLgen grammar. We feel that it is bad
practice to write parsers this way; the `eating' of parts of
the input should be done in the lexical analyzer, not in the parser.
After all, in the case of LLgen, one can regard a semantic action
in the input as one token, and thus it should be handled by
the lexical analyzer as such.

.NH 2
Examples of error recovery

.LP
We will now give some examples that compare non-correcting error
recovery with the correcting error recovery used by parsers generated
by `standard' LLgen.

Consider the next C program, where there is a `)' missing in the
header of function `test'.
.br
.nf

	1 	int test(a,b
	2
	3	int a,b;
	4
	5	{
	6		if (a < b)
	7			return(1);
	8		else
	9			return(0);
	10	}
.fi

.LP
This small error derails the `standard' parser; it produces the
following error messages, where we have left out 7 messages reporting
semantic errors:
.br
.nf

	line 3: , missing before type_identifier
	line 3: , missing before identifier
	line 3: ) missing before ;
	line 5: { deleted
	line 6: if deleted
	line 6: < deleted
	line 6: ) missing before identifier
	line 6: ) deleted
	line 7: identifier missing before return
	line 7: ; missing before return
	line 7: { missing before return
	line 8: else deleted

.fi
.LP
In contrast, the parser using non-correcting error recovery produces
only one error message:
.br

	line 3: type_identifier illegal

This error message correctly pin-points the error: there should
have been a `)' at the position where type-identifier `int' is.

.LP
Now, an example with Modula-2; consider this program:
.br
.nf

	1	MODULE test;
	2
	3	TYPES
	4		ElementRecordType = RECORD
	5		Element: ElementType;
	6		Next,
	7		Prior: ElementPointerType;
	8	END;
	9
	10	VARS a,b,c: ElementRecordType;
	11
	12
	13	BEGIN
	14
	15		a := b;
	16
	17	END test.

.fi
.LP
There are two syntactic errors in this program; on line 3, TYPES should be TYPE, and
on line 10, VARS should be VAR. We have left out the type declarations of
ElementType and ElementPointerType; clearly this will generate semantic
errors, but we are only interested in syntactic errors anyway.
The correcting error recovery parser
again derails on this program; it produces the following syntactic error messages:
.br
.nf

	line 3: CONST missing before identifier
	line 4: '=' missing before identifier
	line 4: RECORD deleted
	line 5: ':' deleted
	line 5: ';' missing before identifier
	line 5: '=' missing before ';'
	line 5: number missing before ';'
	line 6: ',' deleted
	line 7: '=' missing before identifier
	line 7: ':' deleted
	line 7: ';' missing before identifier
	line 7: '=' missing before ';'
	line 7: number missing before ';'
	line 8: ';' deleted
	line 10: identifier deleted
	line 10: ',' deleted
	line 10: identifier deleted
	line 10: ',' deleted
	line 10: identifier deleted
	line 10: ':' deleted
	line 10: identifier deleted
	line 10: ';' deleted
	line 13: BEGIN deleted
	line 15: identifier deleted
	line 15: := deleted
	line 15: identifier deleted
	line 15: ';' deleted
	line 17: END deleted
	line 17: identifier deleted

.fi
.LP
The error correction mechanism clearly makes the wrong guess by inserting
CONST on line 3; as a result, all that follows is rejected as incorrect.
In contrast, the non-correcting error recovery mechanism only produces
two error messages:
.br
.nf

	line 3: identifier illegal
	line 10: identifier illegal

.fi
.LP
This again exactly pin-points the errors: the identifiers TYPES and
VARS constitute the only errors in the program. Note that the
presence of more than one error does not cause any problems to the
non-correcting recovery mechanism. 

.bp
.nr PS 12
.nr VS 14

.NH
Conclusion

.nr PS 10
.nr VS 12

.LP
After implementing and testing a non-correcting error recovery mechanism 
we have come to the conclusion that it indeed is superior to correcting
mechanisms in what regards the error messages it produces; 
the examples we have given clearly show this. However, there is a
clear loss of performance when errors are present in a program, 
although we have found this performance
degradation to be acceptable. We feel that the benefits of
better error messages outweigh the loss of performance. In any case,
correct programs do not suffer at all from the incorporation 
of a non-correcting recovery mechanism.
The error recovery mechanism we implemented does not make
unreasonable demands on resources; the size of the prediction
graphs stays within reasonable limits. 

.LP
The main problems we encountered had to do with recognizing
`languages within languages', and semantic actions that did
unreasonable things like eating input. The more `well-behaved' a
parser is, the better the results the non-correcting error recovery
mechanism gives. This is also true for the input grammars: with a
language like Modula-2, whose syntax has been designed with parser
generators in mind, the performance of the non-correcting mechanism
is better than with C, whose syntax is extremely hard, if not
impossible to describe with a LL(1) grammar.

.bp
.nr PS 12
.nr VS 14

.NH
Bibliography

.nr PS 10
.nr VS 12

.IP [CORMACK] 12
Gordon V. Cormack, `An LR substring parser for noncorrecting syntax error
recovery', ACM SIGPLAN Notices, vol. 24, no. 7, p. 161-169, July 1989

.IP [GRUNE] 12
Dick Grune, Ceriel J.H. Jacobs, `A programmer friendly LL(1) parser
generator', Softw. Pract. Exper., vol. 18, no. 1, p. 29-38, Jan 1988

.IP [RICHTER] 12
Helmut Richter, `Noncorrecting syntax error recovery', ACM Trans. Prog. Lang.
Sys., vol.7, no.3, p. 478-489, July 1985

.IP [ROEHRICH] 12
Johannes R\*:ohrich, `Methods for the automatic construction of error
correcting parsers', Acta Inform., vol. 13, no. 2, p. 115-139, Feb 1980

.IP [TOMITA] 12
Masaru Tomita, Efficient parsing for natural language, Kluwer Academic
Publishers, Boston, p.210, 1986
.bp
.SH
Appendix A: Implementation Issues

.nr PS 10
.nr VS 12
.RS
.LP
In this appendix we will describe some implementation issues;
the data structure used to store the grammar during non-correcting
error recovery, postponing deletions of graph elements until after 
the prediction phase, and the implementation of the %substart directive .
.RE

.SH
A.1 The grammar data structure

.LP
The grammar data structure used by the non-correcting error recovery technique has
to meet two conditions: easy access to a rule as a whole to make
substituting nonterminals efficient and easy access to each symbol in the RHS 
of a rule to make starting error recovery and finding continuations
efficient. To fulfill these conditions we decided to construct the 
storage of the grammar as follows.

.LP
A rule in the grammar is divided in two
parts: a LHS and a RHS. The LHS is represented by a struct `lhs' and
for each symbol in the RHS a struct 'symbol' is constructed. 
A struct `lhs' contains the number of the 
nonterminal forming the LHS of the rule, a pointer to the RHS, the 
first- and follow-sets of the nonterminal and a flag 'empty' which
indicates whether the nonterminal produces empty or not. A struct 
`symbol' contains a field indicating the type of the symbol, i.e.
a terminal or a nonterminal, the number of the symbol, a `link' pointer
to a struct `symbol' that represents the same symbol, a `next' pointer
to the rest of the RHS and a pointer back to the LHS.

.LP
A special struct `symbol' is added to the end of the RHS to indicate 
the end of a rule. The type of this struct is LLEORULE, the number
is set to -1 and the pointers 'link' and `next' are nil.

.LP
In case that there is more than one RHS for a LHS, all the RHS's 
are put after each other and separated by another special struct 
`symbol'. The type of this struct is LLALT, the number is set to 
-1 and the 'link' pointer is nil. After the last RHS a `LLEORULE'-struct 
marker is added. 

.LP
Finally, to make searching efficient there are two arrays: `terminals'
and `nonterminals'. `terminals' is indexed by the number of a terminal
and contains for each terminal a struct containing a 'link' pointer 
to a symbol, representing this terminal, in the RHS of a rule. Because
this symbol has again a 'link' pointer to another symbol representing
the terminal, it is possible by following this chain of pointers 
to find all rules containing such a terminal. In a similar way `nonterminals'
is indexed by the number of a nonterminal and contains for each 
nonterminal a struct. This struct not only contains a 'link' pointer
linking all rules with this nonterminal, but also contains a 'rule' 
pointer. This pointer points to the RHS or RHS's of the rules of which
the nonterminal forms the LHS.
 
.LP
As an example, consider the following grammar:

.br
A:	a B
.br
B:	a | $epsilon$
.br

This will result in the picture below. Note that `pointer' fields 
without an arrow indicate nil pointers.

.PS
dx = 0.05

down 
A_a: box ht boxht/2 "link"
box invis "a" ljust with .e at A_a.w

move to A_a.s
move
move

A: box "link" "rule"
B: box "link" "rule"
line dashed from A.w to A.e 
line dashed from B.w to B.e
box invis "A" ljust with .e at A.w
box invis "B" ljust with .e at B.w

move to A.ne
right
move
move
down

LHS_A: box wid 1.2 * boxwid ht 2.5 * boxht "`A'" "rhs" "first" "follow" "empty 0"
line dashed from 0.2 <LHS_A.nw, LHS_A.sw> to 0.2 <LHS_A.ne, LHS_A.se>
line dashed from 0.4 <LHS_A.nw, LHS_A.sw> to 0.4 <LHS_A.ne, LHS_A.se>
line dashed from 0.6 <LHS_A.nw, LHS_A.sw> to 0.6 <LHS_A.ne, LHS_A.se>
line dashed from 0.8 <LHS_A.nw, LHS_A.sw> to 0.8 <LHS_A.ne, LHS_A.se>

move to LHS_A.ne + (1,0)

RHS_a1: box wid 2.0 * boxwid ht 2.5 * boxht "LLTERM" "`a'" "link" "next" "lhs"
line dashed from 0.2 <RHS_a1.nw, RHS_a1.sw> to 0.2 <RHS_a1.ne, RHS_a1.se>
line dashed from 0.4 <RHS_a1.nw, RHS_a1.sw> to 0.4 <RHS_a1.ne, RHS_a1.se>
line dashed from 0.6 <RHS_a1.nw, RHS_a1.sw> to 0.6 <RHS_a1.ne, RHS_a1.se>
line dashed from 0.8 <RHS_a1.nw, RHS_a1.sw> to 0.8 <RHS_a1.ne, RHS_a1.se>

move to RHS_a1.ne + (1,0)

RHS_B: box wid 2.0 * boxwid ht 2.5 * boxht "LLNONTERM" "`B'" "link" "next" "lhs"
line dashed from 0.2 <RHS_B.nw, RHS_B.sw> to 0.2 <RHS_B.ne, RHS_B.se>
line dashed from 0.4 <RHS_B.nw, RHS_B.sw> to 0.4 <RHS_B.ne, RHS_B.se>
line dashed from 0.6 <RHS_B.nw, RHS_B.sw> to 0.6 <RHS_B.ne, RHS_B.se>
line dashed from 0.8 <RHS_B.nw, RHS_B.sw> to 0.8 <RHS_B.ne, RHS_B.se>

move to RHS_B.ne + (1,0)

RHS_END1: box wid 2.0 * boxwid ht 2.5 *boxht "LLEORULE" "-1" "link" "next" "lhs"
line dashed from 0.2 <RHS_END1.nw, RHS_END1.sw> to 0.2 <RHS_END1.ne,RHS_END1.se>
line dashed from 0.4 <RHS_END1.nw, RHS_END1.sw> to 0.4 <RHS_END1.ne,RHS_END1.se>
line dashed from 0.6 <RHS_END1.nw, RHS_END1.sw> to 0.6 <RHS_END1.ne,RHS_END1.se>
line dashed from 0.8 <RHS_END1.nw, RHS_END1.sw> to 0.8 <RHS_END1.ne,RHS_END1.se>


move to LHS_A.s - (0,1)

LHS_B: box wid 1.2 * boxwid ht 2.5 * boxht "`B'" "rhs" "first" "follow" "empty 1"
line dashed from 0.2 <LHS_B.nw, LHS_B.sw> to 0.2 <LHS_B.ne, LHS_B.se>
line dashed from 0.4 <LHS_B.nw, LHS_B.sw> to 0.4 <LHS_B.ne, LHS_B.se>
line dashed from 0.6 <LHS_B.nw, LHS_B.sw> to 0.6 <LHS_B.ne, LHS_B.se>
line dashed from 0.8 <LHS_B.nw, LHS_B.sw> to 0.8 <LHS_B.ne, LHS_B.se>

move to LHS_B.ne + (1,0)

RHS_a2: box wid 2.0 * boxwid ht 2.5 * boxht "LLTERM" "`a'" "link" "next" "lhs"
line dashed from 0.2 <RHS_a2.nw, RHS_a2.sw> to 0.2 <RHS_a2.ne, RHS_a2.se>
line dashed from 0.4 <RHS_a2.nw, RHS_a2.sw> to 0.4 <RHS_a2.ne, RHS_a2.se>
line dashed from 0.6 <RHS_a2.nw, RHS_a2.sw> to 0.6 <RHS_a2.ne, RHS_a2.se>
line dashed from 0.8 <RHS_a2.nw, RHS_a2.sw> to 0.8 <RHS_a2.ne, RHS_a2.se>

move to RHS_a2.ne + (1,0)

RHS_ALT: box wid 2.0 * boxwid ht 2.5 * boxht "LLALT" "-1" "link" "next" "lhs"
line dashed from 0.2 <RHS_ALT.nw, RHS_ALT.sw> to 0.2 <RHS_ALT.ne, RHS_ALT.se>
line dashed from 0.4 <RHS_ALT.nw, RHS_ALT.sw> to 0.4 <RHS_ALT.ne, RHS_ALT.se>
line dashed from 0.6 <RHS_ALT.nw, RHS_ALT.sw> to 0.6 <RHS_ALT.ne, RHS_ALT.se>
line dashed from 0.8 <RHS_ALT.nw, RHS_ALT.sw> to 0.8 <RHS_ALT.ne, RHS_ALT.se>

move to RHS_ALT.ne + (1,0)

RHS_END2: box wid 2.0 * boxwid ht 2.5 *boxht "LLEORULE" "-1" "link" "next" "lhs"
line dashed from 0.2 <RHS_END2.nw, RHS_END2.sw> to 0.2 <RHS_END2.ne,RHS_END2.se>
line dashed from 0.4 <RHS_END2.nw, RHS_END2.sw> to 0.4 <RHS_END2.ne,RHS_END2.se>
line dashed from 0.6 <RHS_END2.nw, RHS_END2.sw> to 0.6 <RHS_END2.ne,RHS_END2.se>
line dashed from 0.8 <RHS_END2.nw, RHS_END2.sw> to 0.8 <RHS_END2.ne,RHS_END2.se>

# Next pointers upper row
.ps 30
circle radius .01 at 0.75 <A.ne, A.se> - (dx, 0) 
circle radius .01 at 0.3 <LHS_A.ne, LHS_A.se> - (dx, 0) 
circle radius .01 at 0.7 <RHS_a1.ne, RHS_a1.se> - (dx, 0) 
circle radius .01 at 0.7 <RHS_B.ne, RHS_B.se> - (dx, 0) 
.ps 10 

arrow from 0.75 <A.ne, A.se> - (dx, 0) to 0.3 <LHS_A.nw, LHS_A.sw>
arrow from 0.3 <LHS_A.ne, LHS_A.se> - (dx, 0) to 0.3 <RHS_a1.nw,RHS_a1.sw>
arrow from 0.7 <RHS_a1.ne, RHS_a1.se> - (dx, 0) to 0.7 <RHS_B.nw,RHS_B.sw>
arrow from 0.7 <RHS_B.ne, RHS_B.se> - (dx, 0) to 0.7 <RHS_END1.nw, RHS_END1.sw>


# Next pointers lower row
.ps 30
circle radius .01 at 0.75 <B.ne, B.se> - (dx, 0) 
circle radius .01 at 0.3 <LHS_B.ne, LHS_B.se> - (dx, 0) 
circle radius .01 at 0.7 <RHS_a2.ne, RHS_a2.se> - (dx, 0) 
circle radius .01 at 0.7 <RHS_ALT.ne, RHS_ALT.se> - (dx, 0) 
.ps 10

arrow from 0.75 <B.ne, B.se> - (dx, 0) to 0.3 <LHS_B.nw, LHS_B.sw>
arrow from 0.3 <LHS_B.ne, LHS_B.se> - (dx, 0) to 0.3 <RHS_a2.nw,RHS_a2.sw>
arrow from 0.7 <RHS_a2.ne, RHS_a2.se> - (dx, 0) to 0.7 <RHS_ALT.nw,RHS_ALT.sw>
arrow from 0.7 <RHS_ALT.ne, RHS_ALT.se> - (dx, 0) to 0.7 <RHS_END2.nw, RHS_END2.sw>


# Link pointers
.ps 30
circle radius .01 at 0.5 <RHS_a1.ne, RHS_a1.se> - (2*dx, 0) 
circle radius .01 at 0.5 <A_a.ne, A_a.se> - (dx, 0) 
circle radius .01 at 0.25 <B.ne, B.se> - (dx, 0) 
.ps 10

arrow dashed from 0.5 <RHS_a1.ne, RHS_a1.se> - (2*dx, 0) to RHS_a2.ne - (2*dx,0)
line dashed from 0.5 <A_a.ne, A_a.se> - (dx, 0) right 4.0 * boxwid then to RHS_a1.ne - (2*dx, 0) ->
line dashed from 0.25 <B.ne, B.se> - (dx, 0) right then up .75 then right 7.0 * boxwid then to RHS_B.ne - (2*dx, 0) ->


# LHS pointers upper row
.ps 30
circle radius .01 at 0.9 <RHS_a1.ne, RHS_a1.se> - (3*dx, 0) 
circle radius .01 at 0.9 <RHS_B.ne, RHS_B.se> - (3*dx, 0) 
circle radius .01 at 0.9 <RHS_END1.ne, RHS_END1.se> - (3*dx, 0) 
.ps 10

line from 0.9 <RHS_a1.ne, RHS_a1.se> - (3*dx, 0) down ->
line from 0.9 <RHS_B.ne, RHS_B.se> - (3*dx, 0) down ->
line from 0.9 <RHS_END1.ne, RHS_END1.se> - (3*dx, 0) down then left 8.0 * boxwid then to LHS_A.se -> 


# LHS pointers lower row
.ps 30
circle radius .01 at 0.9 <RHS_a2.ne, RHS_a2.se> - (3*dx, 0)
circle radius .01 at 0.9 <RHS_ALT.ne, RHS_ALT.se> - (3*dx, 0)
circle radius .01 at 0.9 <RHS_END2.ne, RHS_END2.se> - (3*dx, 0)
.ps 10

line from 0.9 <RHS_a2.ne, RHS_a2.se> - (3*dx, 0) down ->
line from 0.9 <RHS_ALT.ne, RHS_ALT.se> - (3*dx, 0) down ->
line from 0.9 <RHS_END2.ne, RHS_END2.se> - (3*dx, 0) down then left 8.0 * boxwid then to LHS_B.se ->


# Text above structs
box invis ht boxht/2 "terminals" with .s at A_a.n
box invis ht boxht/2 "nonterminals" with .s at A.n
box invis ht boxht/2 "lhs" with .s at LHS_A.n
box invis ht boxht/2 "lhs" with .s at LHS_B.n
box invis ht boxht/2 "symbol" with .s at RHS_a1.n
box invis ht boxht/2 "symbol" with .s at RHS_B.n
box invis ht boxht/2 "symbol" with .s at RHS_END1.n
box invis ht boxht/2 "symbol" with .s at RHS_a2.n
box invis ht boxht/2 "symbol" with .s at RHS_ALT.n
box invis ht boxht/2 "symbol" with .s at RHS_END2.n
.PE

.LP
Note that the empty alternative for `B' is represented in the 
data structure by the `LLEORULE-struct' immediately following
the `LLALT'-struct. When there are still other alternatives 
the `LLEORULE'-struct is replaced by a `LLALT'-struct followed
by the other alternatives and a `LLEORULE'-struct. 
Finally, when the empty rule is the only rule for a 
nonterminal the RHS will consist only of a `LLEORULE'-struct.

.SH
A.2 Delayed deletes

.LP
We encountered a problem with deleting elements during the 
prediction phase. Imagine that we have a nonterminal `B' on top of 
the graph, and `B' has two alternatives. Now suppose that we  
apply the first alternative and we find out that this alternative leads 
to a `dead end', i.e. a head that does not match the input symbol, so we want
to get rid of it. When we delete it immediately the deletion algorithm
will also deallocate `[B]' and possibly some elements below `[B]'.
However, there was another alternative for `[B]' which was not yet 
developed and maybe this alternative leads to a head which is legal.
But `[B]' has already been deleted and thus cannot be used anymore. A similar 
situation can occur when we want to delete a joined element; 
the substitution of a nonterminal
that only produces empty and thus has no element above it in the graph 
can also lead to such a situation. We therefore decided to put `dead ends' 
on a list, `cleanup_arr[]', and after the prediction phase has 
finished we delete all elements on this list, and all their descendants
that become unreachable of course.

.SH
A.3 Clearing flags

.LP
We implemented two different ways to clear the flags set by the prediction 
phase of the algorithm; the first recursively tracks down the whole graph 
following the flags, the second puts all elements visited by 
the prediction phase
on a list; after the prediction phase has finished the algorithm walks 
through this list clearing the flags of all elements on it. We took measurements
on both algorithms and found out that with small programs the times
did not differ much but large programs were processed faster by the
second algorithm. Therefore we decided to use the second algorithm. 

.LP
To speed up the algorithm even more, we do not deallocate the list
after a prediction phase has finished. We just set the number of 
elements on the list to 0. This saves considerably on the number
of `Malloc'-calls.

.SH
A.4 Implementation of %erroneous directive

.LP
As explained in chapter 3, the user can put a %erroneous directive
in front of a terminal, making the non-correcting error recovery
mechanism ignore that terminal. However, implementing this directive
was not entirely straightforward; consider, for example, the rule
.br
.nf

	A:	'a' | %erroneous 'b' | 'c';

.fi
.LP
Just leaving out terminal 'b' will not do, because then nonterminal
A produces empty all of a sudden, which it did not before. 
The rule should become
.br
.nf

	A:	'a' | 'c';

.fi
but this is hard to implement in LLgen. We took a different approach:
we introduce a new terminal 'ERRONEOUS', and substitute it for all 
terminals with an %erroneous directive in front of them. Thus, the
example rule becomes
.br
.nf

	A:	'a' | ERRONEOUS | 'c';

.fi
.LP
Since the terminal ERRONEOUS will never be in the input to the parser,
this has exactly the desired effect; when a predicting phase produces
ERRONEOUS as head of a prediction graph this head will never match the
input. In particular, it will not match the terminal that was
originally there (in this case 'b') so that terminal is no longer
regarded as part of the input language at that point.
.bp
.SH
Appendix B: Using the non-correcting error recovery

.LP
To use the new non-correcting error recovery mechanism, LLgen has to
be called with the new flag -n. LLgen will then create an extra file
called `Lncor.c' which contains the code for the non-correcting recovery 
mechanism. This file has to be compiled and linked with the rest
of the program, just like the file `Lpars.c'. 

.LP
The user-supplied error reporting routine `LLmessage' will have to be
modified slightly; when it is called with a positive parameter, it
should only set the attributes of the inserted token, but not report an 
error. Note that the lexical analyzer still must return the same token
as it did the last time it was called. When LLmessage is called with 
parameter 0, it should report that the token in global variable LLsymb 
is illegal; if the value of LLsymb is `EOFILE', the routine should
report an unexpected End-of-file. When LLmessage is called with parameter
-1, it should report that end-of-file was expected. To facilitate
switching between correcting and non-correcting error recovery,
the file Lpars.h contains a statement `#define LLNONCORR' 
which indicates that the non-correcting
mechanism is enabled.
Here is a
skeleton for the modified LLmessage routine:
.nr PS 8
.nr VS 10
.LP
.br
.nf

	#include "Lpars.h"
	extern int LLsymb;

	LLmessage(flag) 
	int flag;
	{
		if (flag < 0)     
		{
			/* Error message "end-of-file expected" */;
		}
		else if (flag)    
		{
			/* flag equals the number of the inserted token */
#ifndef LLNONCORR 

			/* Error message "token inserted" */;	
#endif

			/* Code to set attributes for inserted token */
			/* Code to make lexical analyzer return same token as before */

		else    
		{
			/* The number of the illegal or deleted token is in LLsymb */
#ifndef LLNONCORR

			/* Error message "token deleted" */;
#else

			if (LLsymb == EOFILE)
			{
				/* Error message "unexpected end of file" */
			}
			else
			{
				/* Error message "token illegal" */;
			}
#endif
			
		}

	}

.fi
.nr PS 10
.nr VS 12

.LP
For best results, one should check if the parser calls other parsers
in semantic actions; if this is the case, and the called parser
processes the same input file as the calling parser, then a %substart
should be put in front of the semantic action that starts a parser.
If a semantic action calls parsers defined by startsymbols say
A and B, then `%substart A, B;' should be put in front of the action.
As an alternative, one can use the -s flag of LLgen; this has the
same effect as putting `%substart X, Y, ....;' in front of all
semantic actions, where X, Y, .... are the startsymbols of the grammar.
Clearly, it is preferable to analyze the grammar and put %substart
directives only where appropriate.

Finally, beware of syntactic errors being handled in semantic
actions; eg, one could have a rule like
.nr PS 8
.nr VS 10
.LP
.br
.nf

        Assignment_statement:   lvalue
                                [
                                        '='
                                        {
                                         error(":= expected");
                                        }

                                        |

                                        ':='
                                ]
                                expression
                                ;
.fi

.nr PS 10
.nr VS 12
.LP
To ensure that the non-correcting mechanism will recognize the
`=' as a syntactic error, a `%erroneous' directive should be
put in front of it.