ack/doc/ceg/ceg.tr

.nr PS 12
.nr VS 14
.nr LL 6i
.tr ~
.TL
The Code Expander Generator
.AU
Frans Kaashoek
Koen Langendoen
.AI
Dept. of Mathematics and Computer Science
Vrije Universiteit
Amsterdam, The Netherlands
.NH
Introduction
.PP
A \fBcode expander\fR (\fBce\fR for short) is a part of the 
Amsterdam Compiler Kit (\fBACK\fR) and provides the user with
high-speed generation of medium-quality code. Although conceptually
equivalent to the more usual \fBcode generator\fR, it differs in some
aspects.
.LP
Normally, a program to be compiled with \fBACK\fR
is first fed to the preprocessor. The output of the preprocessor goes 
into the appropriate front end, which produces EM 
.[~[
Tanenbaum
.]]
(a
machine independent low level intermediate code). The generated EM code is fed
into the peephole optimizer, which scans it with a window of a few instructions,
replacing certain inefficient code sequences by better ones. After the
peephole optimizer a back end follows, which produces high-quality assembly code.
The assembly code goes via the target optimizer into the assembler and the
object code then goes into the
linker/loader, the final component in the pipeline. 
.LP
For various applications 
this scheme is too slow. When debugging, for example, 
reducing compile time is more important than execution time of a program.
For this purpose a new scheme is introduced:
.IP \ \ 1:
The code generator and assembler are
replaced by a library, the \fBcode expander\fR, consisting of a set of routines
which directly expand
the EM-instructions into a relocatable object file.
The peephole and target optimizer are not used.
.IP \ \ 2:
These routines replace the usual EM-generating routines in the front end; this
eliminates the overhead of intermediate files.
.LP
This results in a fast compiler producing object file, ready to be
linked and loaded, at the cost of unoptimized object code.
.LP
Extra speedup is obtained by generating code for a single EM-instruction
at a time, instead of doing pattern-matching on EM, as the usual code generator
does.
.LP
Because of the
simple nature of the code expander, it is much easier to build, to debug and to
test. Experience has demonstrated that a code expander can be constructed,
debugged and tested in less than two weeks.
.LP
This document describes the tools for automatically generating a
\fBce\fR (a library of C files), from two tables and 
a few machine-dependent functions. 
A thorough knowledge of EM is necessary to understand this document.
.NH
The code expander generator
.PP
The code expander generator (\fBceg\fR) generates a code expander from 
two tables and a few machine-dependent functions. This section explains how 
\fBceg\fR works. The first half describes the transformations that are done on
the two tables. The 
second half tells how these transformations are done by the \fBceg\fR.
.PP
A code expander consists of a set of routines that convert EM-instructions
directly to relocatable object code. These routines are called by a front 
end through the EM_CODE(3ACK)
.[~[
EM_CODE
.]]
interface. To free the table writer of the burden of building
an object file, we supply a set of routines that build an object file
in the ACK_A.OUT(5L)
.[~[
aout
.]]
format (see appendix B). This set of routines is called
the
\fBback\fR-primitives (see appendix A). In short, a code expander consists of a
set of routines which map the EM_CODE interface on the 
\fBback\fR-primitives interface.
.PP
To avoid repetition of the same sequences of
\fBback\fR-primitives in different
EM-instructions
and to improve readability, the EM-to-object information must be supplied in
two
tables. The EM_table maps EM to an assembly language, and the as_table
maps
assembly to \fBback\fR-primitives. The assembly language is chosen by the
table writer. It can either be an actual assembly language or his ad-hoc 
designed language.
.LP
The following picture shows the dependencies between the different components:
.sp
.PS
linewid = 0.5i
A: line down 2i
B: line down 2i with .start at A.start + (1.5i, 0)
C: line down 2i with .start at B.start + (1.5i, 0)
D: arrow right with .start at A.center - (0.25i, 0)
E: arrow right with .start at B.center - (0.25i, 0)
F: arrow right with .start at C.center - (0.25i, 0)
"EM_CODE(3ACK)" at A.start above
"EM_table" at B.start above
"as_table" at C.start above
"source language  " at D.start rjust
"EM" at 0.5 of the way between D.end and E.start
G: "assembly" at 0.5 of the way between E.end and F.start
H: "  back primitives" at F.end ljust
"(user defined)" at G - (0, 0.2i)
"  (ACK_A.OUT)" at H - (0, 0.2i) ljust
.PE
.PP
Although the picture suggests that during compilation the EM instructions are
first transformed into assembly instructions and then the assembly instructions
are transformed into object-generating calls, this
is not what happens in practice, although the user is free to think it does.
Actually, however the EM_table and the as_table are combined during code
expander generation time, yielding an imaginary compound table that results in
routines from the EM_CODE interface that generate object code directly.
.PP
As already indicated, the compound table does not exist either. Instead, each
assembly instruction in the as_table is converted to a routine generating C
code
.[~[
Kernighan
.]]
to generate C code to call the \fBback\fR-primitives. The EM_table is
converted into a program that for each EM instruction generates a routine,
using the routines generated from the as_table. Execution of the latter program
will then generate the code expander.
.PP
This scheme allows great flexibility (e.g., when \fBceg\fR is called with a
special flag it generates assembly instead of object code)
in the table writing, while still
resulting in a very efficient code expander. One implication is that the
as_table is interpreted twice and the EM_table only once. This has consequences
for their structure.
.PP
To illustrate what happens, we give an example. The example is an entry in
the tables for the VAX-machine. The assembly language chosen is a subset of the 
VAX assembly language.
.PP
One of the most fundamental operations in EM is ``loc c", load the value of c
on the stack. To expand this instruction the 
tables contain the following information:
.DS
EM_table   : \f5
   C_loc   ==>   "pushl $$$1".
     /* $1 refers to the first argument of C_loc. 
      * $$ is a quoted $. */


\fRas_table   :\f5
   pushl  src : CONST   ==> 
                         @text1( 0xd0);
                         @text1( 0xef);
                         @text4( %$( src->num)).
\fR
.DE
.LP
The as_table is transformed in the following routine:
.DS
\f5
pushl_instr(src)
t_operand *src;    
/* "t_operand" is a struct defined by the table writer. */
{
   printf("swtxt();");
   printf("text1( 0xd0);");
   printf("text1( 0xef);");
   printf("text4( %s );", substitute_dollar( src->num) );
}
\fR
.DE
Using "pushl_instr()", the following routine is generated from the EM_table:
.DS
\f5
C_loc( c)
arith c;
{
    swtxt();
    text1( 0xd0);    /* text1(), text4() are library routines,  */
    text1( 0xef);    /* which fill the text segment             */
    text4( c);
}
\fR
.DE
.LP
A compiler call to "C_loc()" will cause the 1-byte numbers "0xd0" 
and "0xef"
and the 4-byte value of the variable "c" to be stored in the text segment.
.PP
The transformations on the tables are done automatically by the code expander
generator.
The code expander generator consists of two tools, one to handle the 
EM_table, \fBemg\fR, and one to handle the as_table, \fBasg\fR. \fBAsg\fR 
transforms 
each assembly instruction in a C routine. These C routines generate calls
to the \fBback\fR-primitives. Finally, the generated C routines are used
by \fBemg\fR to generate the actual code expander from the EM_table.
.PP
The link between \fBemg\fR and \fBasg\fR is an assembly language.
We did not enforce a specific syntax for the assembly language;
instead we have chosen to give the table writer the freedom
to make an ad-hoc assembly language or to use an actual assembly language 
suitable for his purpose. Apart from a greater flexibility this
has another advantage; if the table writer adopts the assembly language that
runs on the machine at hand, he can test the EM_table independently from the
as_table. Of course there is a price to pay: the table writer has to
do the decoding of the operands himself. See section 4 for more details.
.PP
Before we describe the structure of the tables in detail, we will give 
an overview of the four main phases.
.IP "phase 1:"
.br
The as_table is transformed by \fBasg\fR. This results in a set of C routines. 
Each assembly-opcode generates one C routine. Note that a call to such a
routine does not generate the corresponding object code; it generates C code,
which, when executed, generates the desired object code.
.IP "phase 2:"
.br
The C routines generated by \fBasg\fR are used by emg to expand the EM_table. 
This
results in a set of C routines, the code expander, which conform to the 
procedural interface EM_CODE(3ACK). A call to such a routine does indeed
generate the desired object code.
.IP "phase 3:"
.br
The front end that uses the procedural interface is linked/loaded with the
code expander generated in phase 2 and the \fBback\fR-primitives (a supplied
library). This results in a compiler.
.IP "phase 4:"
.br
Execution of the compiler. The routines in the code expander are
executed and produce object code.
.RE
.NH
Description of the EM_table
.PP
This section describes the EM_table. It contains four subsections:
the first 3 sections describe the syntax of the EM_table,
the
semantics of the EM_table, and an list of the functions and
constants that must be present in the EM_table, in the file "mach.c" or in
the file "mach.h"; and the last section deals with the case that the table
writer wants to generate assembly instead of object code. The section on
semantics contains many examples.
.NH 2
Grammar
.PP
The following grammar describes the syntax of the EM_table.
.VS +4
.TS
center tab(%);
l c l.
TABLE%::=%( RULE)*
RULE%::=%C_instr   ( CONDITIONALS | SIMPLE)
CONDITIONAL%::=%( condition   SIMPLE)+   "default"   SIMPLE
SIMPLE%::=%( "==>" | "::=")   ACTION_LIST
ACTION_LIST%::=%[ ACTION   ( ";" ACTION)* ]   "."
ACTION%::=%AS_INSTR
%|%function-call
AS_INSTR%::=%""" [ label ":"]   [ INSTR] """
INSTR%::=%mnemonic   [ operand   ( ","   operand)* ]
.TE
.VS -4
.PP
The "(" ")" brackets are used for grouping, "[" ... "]" means ... 0 or 1 time,
a "*" means zero or more times, a "+" means one or more times and a "|" means 
a choice between left or right. A \fBC_instr\fR is 
a name in the EM_CODE(3ACK) interface. \fBcondition\fR is a C expression. 
\fBfunction-call\fR is a call of a C function. \fBlabel\fR, \fBmnemonic\fR
and \fBoperand\fR are arbitrary strings. If an \fBoperand\fR 
contains brackets, the
brackets must match. In reality there is an upper bound on the number of
operands; the maximum number is defined by the constant MAX_OPERANDS in de
file "const.h" in the directory assemble.c. Comments in the table should be
placed between "/*" and "*/". Finally, before the table is parsed, the
C preprocessor runs.
.NH 2
Semantics
.PP
The EM_table is processed by \fBemg\fR. \fBEmg\fR generates a C function
for every instruction in the EM_CODE(3ACK). 
For every EM-instruction not mentioned in the EM_table, a
C function that prints an error message is generated.
It is possible to divide the EM_CODE(3ACK)-interface in four parts :
.IP \0\01: 
text instructions      (e.g., C_loc, C_adi, ..)
.IP \0\02: 
pseudo instructions    (e.g., C_open, C_df_ilb, ..)
.IP \0\03: 
storage instructions   (e.g., C_rom_icon,  ..)
.IP \0\04: 
message instructions   (e.g., C_mes_begin, ..)
.LP
This section starts with giving the semantics of the grammar. The examples
are text instructions. The section ends with remarks on the pseudo
instructions and the storage instructions. Since message instructions are not
useful for a code expander, they are ignored. 
.PP
.NH 3
Actions
.PP
The EM_table consists of rules which describe how to expand a \fBC_instr\fR
from the EM_CODE(3ACK)-interface (corresponding to an EM instruction) into actions. 
There are two kinds of actions: assembly instructions and C function calls. 
An assembly instruction is defined as a mnemonic followed by zero or more
operands, separated by commas. The semantics of an assembly instruction is
defined by the table writer. When the assembly language is not expressive 
enough, then, as an escape route, function calls can be made. However, this
reduces
the speed of the actual code expander. Finally, actions can be grouped into
a list of actions; actions are separated by a semicolon and terminated 
by a ".".
.DS
\f5
C_nop   ==> .            
       /* Empty action list : no operation. */

C_inc   ==> "incl (sp)". 
       /* Assembler instruction, which is evaluated 
        * during expansion of the EM_table */

C_slu   ==> C_sli( $1).  
       /* Function call, which is evaluated during
        *  execution of the compiler. */
\fR
.DE
.NH 3
Labels
.PP
Since an assembly language without instruction labels is a rather weak 
language, labels inside a contiguous block of assembly instructions are 
allowed. When using labels two rules must be observed:
.IP \0\01:
The name of a label should be unique inside an action list.
.IP \0\02:
The labels used in an assembler instruction should be defined in the same
action list.
.LP
The following example illustrates the usage of labels.
.DS
\f5
   /* Compare the two top elements on the stack. */
C_cmp      ==>     "pop bx";           
                   "pop cx";          
                   "xor ax, ax";
                   "cmp cx, bx";
                   "je 2f";  /* Forward jump to local label */
                   "jb 1f";
                   "inc ax";
                   "jmp 2f";
                   "1: dec ax";
                   "2: push ax".
\fR
.DE
We will come back to labels in the section on the as_table.
.NH 3
Arguments of an EM instruction
.PP
In most cases the translation of a \fBC_instr\fR depends on its arguments.
The arguments of a \fBC_instr\fR are numbered from 1 to \fIn\fR, where \fIn\fR
is the
total number of arguments of the current \fBC_instr\fR (there are a few
exceptions, see Implicit arguments). The table writer may
refer to an argument as $\fIi\fR. If a plain $-sign is needed in an
assembly instruction, it must be preceded by a extra $-sign.
.PP
There are two groups of \fBC_instr\fRs whose arguments are handled specially:
.RS
.IP "1: Instructions dealing with local offsets."
.br
The value of the $\fIi\fR argument referring to a parameter ($\fIi\fR >= 0),
is increased by "EM_BSIZE". "EM_BSIZE" is the size of the return status block
and must be defined in the file "mach.h", see section 3.3. For example :
.DS
\f5
C_lol   ==>     "push $1(bp)". 
       /* automatic conversion of $1 */
\fR
.DE
.IP "2: Instructions using global names or instruction labels"
.br
All the arguments referring to global names or instruction labels will be
transformed into a unique assembly name. To prevent name clashes with library
names the table writer has to provide the
conversions in the file "mach.h". For example :
.DS
\f5
C_bra   ==>     "jmp $1". 
        /* automatic conversion of $1 */
        /* type arith is converted to string */
\fR
.DE
.RE
.NH 3
Conditionals
.PP
The rules in the EM_table can be divided in two groups: simple rules and 
conditional rules. The simple rules consist of a \fBC_instr\fR followed by 
a list of actions, as described above. The conditional rules (CONDITIONAL)
allow the table writer to select an action list depending on the value of 
a condition. 
.PP
A CONDITIONAL is a list of a boolean expression with the corresponding
simple rule. If
the expression evaluates to true then the corresponding simple rule is carried
out. If more than one condition evaluates to true, the first one is chosen.
The last case of a CONDITIONAL of a \fBC_instr\fR must handle the default case.
The boolean expression in a CONDITIONAL must be an C expression. Besides the
ordinary C operators and constants, $\fIi\fR references can be used 
in an expression. 
.DS
\f5
    /* Load address of LB $1 levels back. */
C_lxl                                 
    $1 == 0    ==>    "pushl fp".
    $1 == 1    ==>    "pushl 4(ap)".
    default    ==>    "movl $$$1, r0";
                      "jsb .lxl";
                      "pushl r0".
\fR
.DE
.NH 3
Equivalence rule
.PP
Among the simple rules there is a special case rule:
the equivalence rule. This rule declares two \fBC_instr\fR equivalent. To
distinguish it from the usual simple rule "==>" is replaced by a "::=".
The advantage of an equivalence rule is that the arguments are not 
converted (see 3.2.3).
.DS
\f5
C_slu   ::=  C_sli( $1).
\fR
.DE
.NH 3
Abbreviations
.PP
EM instructions with an external as an argument come in three variants in
the EM_CODE(3ACK) interface. In most cases it will be possible to take 
these variants together. For this purpose the ".." notation is introduced. 
.DS
\f5
 /* For the code expander there is no difference between 
  * the following instructions.  */
C_loe_dlb    ==>    "pushl $1 + $2".
C_loe_dnam   ==>    "pushl $1 + $2".
C_loe        ==>    "pushl $1 + $2".

/* So it can be written in the following way.  */
C_loe..      ==>    "pushl $1 + $2".
\fR
.DE
.NH 3
Implicit arguments
.PP
In the last example "C_loe" has two arguments, but in the EM_CODE interface 
it has one argument. However, this argument depends on the current "hol"
block; in the EM_table this is made explicit. Every \fBC_instr\fR whose
argument depends on a "hol" block has one extra argument; argument 1 refers
to the "hol" block.
.NH 3
Pseudo instructions
.PP
Most pseudo instructions are machine independent and are provided
by \fBceg\fR. The table writer has only to supply the functions :
.DS
\f5
prolog()
/* Performs the prolog, for example save 
 * return address */

locals( n) 
arith n;
/* Allocate n bytes for locals on the stack */

jump( label)
char *label;
/* Generates code for a jump to "label" */
\fR
.DE
.LP
These functions can be defined in "mach.c" or in the EM_table.
.NH 3
Storage instructions
.PP
The storage instructions "C_bss_\fIcstp()\fR", "C_hol_\fIcstp()\fR",
"C_con_\fIcstp()\fR" and "C_rom_\fIcstp()\fR", except for the instructions
dealing with constants of type string ( C_..._icon, C_..._ucon, C_..._fcon), are
generated automatically. No information is needed in the table.
To generate the C_..._icon, C_..._ucon, C_..._fcon instructions 
\fBceg\fR only has to know how to convert a number of type string to bytes;
this can be defined with the constants ONE_BYTE, TWO_BYTES, and FOUR_BYTES.
C_rom_icon, C_con_icon, C_bss_icon, C_hol_icon can be abbreviated by ..icon.
This also holds for ..ucon and ..fcon.
For example :
.DS
\f5
\\.\\.icon
    $2 == 1   ==>  gen1( (ONE_BYTE) atoi( $1)).
    $2 == 2   ==>  gen2( (TWO_BYTES) atoi( $1)).
    $2 == 4   ==>  gen4( (FOUR_BYTES) atoi( $1)).
    default   ==>   arg_error( "..icon", $2).
\fR
.DE
Gen1(), gen2() and gen4() are \fBback\fR-primitives, see appendix A, and
generate one, two, or four byte constants. Atoi() is a C library function which
converts strings to integers.
The constants "ONE_BYTE", "TWO_BYTES" and "FOUR_BYTES" must be defined in
the file "mach.h".
.NH 2
User supplied definitions and functions
.PP
If the table writer uses all the default functions he has only to supply
the following constants and functions :
.TS
tab(#);
l c lw(10c).
prolog()#:#T{
Do prolog
T}
jump( l)#:#T{
Perform a jump to label l
T}
locals( n)#:#T{
Allocate n bytes on the stack
T}
#
NAME_FMT#:#T{
Print format describing name to a unique name conversion. The format must
contain %s.
T}
DNAM_FMT#:#T{
Print format describing data-label to a unique name conversion. The  format
must contain %s.
T}
DLB_FMT#:#T{
Print format describing numerical-data-label to a unique name conversion.
The format must contain a %ld.
T}
ILB_FMT#:#T{
Print format describing instruction-label to a unique name conversion.
The format must contain %d followed by %ld.
T}
HOL_FMT#:#T{
Print format describing hol-block-number to a unique name conversion.
The format must contain %d.
T}
#
EM_WSIZE#:#T{
Size of a word in bytes on the target machine
T}
EM_PSIZE#:#T{
Size of a pointer in bytes on the target machine
T}
EM_BSIZE#:#T{
Size of base block in bytes on the target machine
T}
#
ONE_BYTE#:#T{
\\C type which occupies one byte on the machine where the \fBce\fR runs
T}
TWO_BYTES#:#T{
\\C type which occupies two bytes on the machine where the \fBce\fR runs
T}
FOUR_BYTES#:#T{
\\C type which occupies four bytes on the machine where the \fBce\fR runs
T}
#
BSS_INIT#:#T{
The default value which the loader puts in the bss segment
T}
#
BYTES_REVERSED#:#T{
Must be defined if you want the byte order reversed.
By default the least significant byte is outputted first.
.FS
When both byte orders are used, for example NS 16032, the table writer has to
supply his own set of routines.
.FE
T}
WORD_REVERSED#:#T{
Must be defined if you want the word order reversed.
By default the least significant word is outputted first.
T}
.TE
.LP
An example of the file "mach.h" for the vax4 with 4.1 BSD - UNIX.
.TS
tab(:);
l l l.
#define : ONE_BYTE : char
#define : TWO_BYTES : short
#define : FOUR_BYTES : long
:
#define : EM_WSIZE : 4
#define : EM_PSIZE : 4
#define : EM_BSIZE : 0
:
#define : BSS_INIT : 0
:
#define : NAME_FMT : "_%s"
#define : DNAM_FMT : "_%s"
#define : DLB_FMT  : "_%ld"
#define : ILB_FMT  : "I%03d%ld"
#define : HOL_FMT  : "hol%d"
.TE
Notice that EM_BSIZE is zero. The vax4 takes care of this automatically.
.PP
There are three routines which have to be defined by the table writer. The
table writer can define them as ordinary C functions in the file "mach.c" or
define them in the EM_table. For example, for the 8086 they look like this:
.DS
\f5
jump       ==>       "jmp $1".

prolog     ==>       "push bp";
                     "mov bp, sp".

locals     
  $1  == 0   ::=      .
  $1  == 2   ==>     "push ax".
  $1  == 4   ==>     "push ax";
                     "push ax".
  default    ==>     "sub sp, $1".
\fR
.DE
.NH 2
Generating assembly code 
.PP
When the code expander generator is used for generating assembly instead of
object code, not all the above mentioned constants and functions have to 
be defined. In this case, the constants "BYTES_REVERSED" and "WORDS_REVERSED" 
are not used (see section 5).
.NH 1
Description of the as_table
.PP
This section describes the as_table. Like the previous section it is divided in
four parts: the first part describes the grammar of the as_table; the second 
part describes the semantics of the as_table; the third part gives an overview
of the functions and the constants that must be present in the as_table, in 
the file "as.h" or in the file "as.c"; the last part describes the case when
assembly is generated instead of object code.
The part on semantics contains examples which appear in the as_table for the
VAX or for the 8086. 
.NH 2
Grammar
.PP
The form of the as_table is given by the following grammar :
.VS +4
.TS
center tab(#);
l c l.
TABLE#::=#( RULE)*
RULE#::=#( mnemonic | "...")   DECL_LIST   "==>"   ACTION_LIST
DECL_LIST#::=#DECLARATION   ( ","   DECLARATION)*
DECLARATION#::=#operand   [ ":"   type]
ACTION_LIST#::=#ACTION   ( ";"   ACTION) "."
ACTION#::=#IF_STATEMENT
#|#function-call
#|#@function-call
IF_STATEMENT#::=#"@if"   "(" condition ")"   ACTION_LIST
##( "@elsif"   "(" condition ")"   ACTION_LIST)*
##[ "@else"   ACTION_LIST]
##"@fi"
.TE
.VS -4
.LP
\fBmnemonic\fR, \fBoperand\fR and \fBtype\fR are all C identifiers,
\fBcondition\fR is a normal C expression.
\fBfunction-call\fR must be a C function call. 
Since the as_table is
interpreted on two levels, during code expander generation and during code
expander execution, two levels of calls are present in it. A "function-call"
is done during code expander generation, a "@function-call" during code
expander execution.
.NH 2
Semantics
.PP
The as_table consists of rules which map assembly instructions onto
\fBback\fR-primitives, a set of functions that construct an object file. 
The table is processed by \fBasg\fR, and it generates a set of C functions,
one for each assembler mnemonic. (The names of
these functions are the assembler mnemonics postfixed with "_instr", e.g.
\"add" becomes "add_instr()".) These functions will be used by the function 
assemble() during the expansion of the EM_table. 
After explaining the semantics of the as_table the function
assemble() will be described.
.NH 3
Rules
.PP
A rule in the as_table consists of a left and right side; 
the left side describes an assembler instruction (mnemonic and operands); the
right side gives the corresponding actions as \fBback\fR-primitives or as
functions, defined by the table writer, that call \fBback-primitives\fR.
A simple example from the VAX as_table and the 8086 as_table:
.DS L
\f5
movl src, dst  ==> @text1( 0xd0);
                   gen_operand( src); 
                   gen_operand( dst). 
    /* "gen_operand" is a function that encodes 
     * operands by calling back-primitives. */

rep ens:MOVS   ==>  @text1( 0xf3);
                    @text1( 0xa5).  

\fR
.DE
.NH 3
Declaration of types.
.PP
In general a machine instruction is encoded as an opcode optionally followed by
the operands, but there are two methods for mapping assembler mnemonics
onto opcodes: the mnemonic determines the opcode, or mnemonic and operands 
determine the opcode. Both cases can be easily expressed in the as_table.
The first case is obvious. For the second case type fields for the operands
are introduced.
.LP
When mnemonic and operands together determine the opcode, the table writer has 
to give several rules for each combination of mnemonic and operands. The rules
differ in the type fields of the operands.
The table writer has to supply functions that check the type
of the operand. The name of such a function is the name of the type; it
has one argument: a pointer to a struct of type t_operand; it returns
non-zero when the operand is of this type, otherwise it returns 0.
.LP
This will usually lead to a list of rules per mnemonic. To reduce the amount of
work an abbreviation is supplied. Once the mnemonic is specified it can be
refered to in the following rules by "...".
One has to make sure
that each mnemonic is mentioned only once in the as_table, as otherwise 
\fBasg\fR will generate more than one function with the same name.
.LP
The following example shows the usage of type fields.
.DS L
\f5
 mov dst:REG, src:EADDR    ==>    @text1( 0x8b); /* opcode */
                                  mod_RM( %d(dst->reg), src). 
                                               /* operands */

 ... dst:EADDR, src:REG    ==>    @text1( 0x89); /* opcode */
                                  mod_RM( %d(src->reg), dst). 
                                               /* operands */
\fR
.DE
The table-writer must supply the restriction functions, \f5REG\fR and
\f5EADDR\fR in the previous example, in "as.c"/"as.h".
.NH 3 
The function of the @-sign and the if-statement.
.PP
The right hand side of a rule consists of function calls. 
Since the as_table is
interpreted on two levels, during code expander generation and during code
expander execution, two levels of calls are present in it. A function-call
without a "@"-sign
is called during code expander generation (e.g., the \f5gen_operand()\fR in the
first example). 
A function call with a "@"-sign is called during code expander execution (e.g.,
the \fBback\fR-primitives). So the last group is a part of the compiler.
.LP
The next example concerns the use of the "@"-sign in front of a table writer
written
function. The need for this construction arises, e.g., when you 
implement push/pop
optimization; flags need to be set/unset and tested during the execution of
the compiler:
.DS L
\f5
PUSH src  ==>   /* save in ax */
                mov_instr( AX_oper, src);  
                /* set flag */
                @assign( push_waiting, TRUE).         

POP dst   ==>   @if ( push_waiting)
                       /* "mov_instr" is asg-generated */
                       mov_instr( dst, AX_oper);      
                       @assign( push_waiting, FALSE).
                @else
                       /* "pop_instr" is asg-generated */
                       pop_instr( dst).               
                @fi.
\fR
.DE
.PP
A problem arises when information is needed that is not known until execution of
the compiler.  For example one needs to know if a "$\fIi\fR" argument fits in
one byte.
In this case one can use a special if-statement provided by \fBasg\fR:
@if, @elsif, @else, @fi. This means that the conditions will be evaluated at
run time of the \fBce\fR. In such a condition one may of course refer to the
"$\fIi\fR" arguments. For example, constants can be packed into one or two byte
arguments:
.DS L
\f5
mov dst:ACCU, src:DATA  ==>  @if ( fits_byte( %$(dst->expr)))
                                   @text1( 0xc0);
                                   @text1( %$(dst->expr)).
                             @else
                                   @text1( 0xc8);
                                   @text2( %$(dst->expr)).
                             @fi.
.DE
.NH 3
References to operands
.PP
As noted before, the operands of an assembler instruction may be used as
pointers, to the struct t_operand, in the right hand side of the table.
Because of the free format assembler, the types of the fields in the struct
t_operand are unknown to \fBasg\fR. Clearly, however, \fBasg\fR must know these types.
This section explains how these types must be specified.
.LP
References to operands come in three forms: ordinary operands, operands that
contain "$\fIi\fR" references, and operands that refer to names of local labels.
The "$\fIi\fR" in operands represent names or numbers of a \fBC_instr\fR and must
be given as arguments to the \fBback\fR-primitives. Labels in operands
must be converted to a number that tells the distance, the number of bytes, 
between the label and the current position in the text-segment. 
.LP
All these three cases are treated in an uniform way. When the table writer
makes a reference to an operand of an assembly instruction, he must describe
the type of the operand in the following way.
.DS
\f5
       reference  :=  "%" conversion 
                      "(" operand-name "->" field-name 
                      ")"
       conversion := printformat |
                     "$"         |
                     "dist"
       printformat := see PRINT(3ACK)
.[~[
PRINT
.]]
\fR
.DE
The three cases differ only in the conversion field. The first conversion
applies to ordinary operands. The second applies to operands that contain
a "$\fIi\fR". The expression between brackets must be of type char *. The
result of "%$" is of the type of "$\fIi\fR". The
third applies to operands that refer to a local label. The expression between
the brackets must be of type char *. The result of "%dist" is of type arith.
.LP
The following example illustrates the usage of "%$". (For an
example that illustrates the usage of ordinary fields see the example in
the section on "User supplied definitions and functions").
.DS L
\f5
jmp dst    ==>    @text1( 0xe9);
                  @reloc2( %$(dst->lab), %$(dst->off), PC_REL).
\fR
.DE
.LP
A useful function concerning $\fIi\fRs is arg_type(), which takes as input a
string starting with $\fIi\fR and returns the type of the \fIi\fR"th argument
of the current EM-instruction, which can be STRING, ARITH or INT. One may need
this function while decoding operands if the context of the $\fIi\fR does not
give enough information.
If the function arg_type() is used, the file
arg_type.h must contain the definition of STRING, ARITH and INT.
.LP
%dist is only guaranteed to work when called as a parameter of text1(), text2() or text4().
The goal of the %dist conversion is to reduce the number of reloc1(), reloc2()
and reloc4()
calls, saving space and time (no relocation at compiler run time). 
.LP
The following example illustrates the usage of "%dist".
.DS L
\f5
 jmp dst:ILB    ==>    /* label in an instruction list */
                 @text1( 0xeb);          
                 @text1( %dist( dst->lab)).

 ... dst:LABEL  ==>    /* global label */
                 @text1( 0xe9);       
                 @reloc2( %$(dst->lab), %$(dst->off), PC_REL).
\fR
.DE
.NH 3
The functions assemble() and block_assemble()
.PP
Assemble() and block_assemble() are two functions provided by \fBceg\fR.
However, if one is not satisfied with the way they work the table writer can
supply his own assemble() or block_assemble().
The default function assemble() splits an assembly string in a label, mnemonic,
and operands and performs the following actions on them:
.IP \0\01:
It processes the local label; it records the name and current position. Thereafter it calls the function process_label() with one argument of type string,
the label. The table writer has to define this function.
.IP \0\02:
Thereafter it calls the function process_mnemonic() with one argument of
type string, the mnemonic. The table writer has to define this function.
.IP \0\03:
It calls process_operand() for each operand. Process_operand() must be
written by the table-writer since no fixed representation for operands
is enforced. It has two arguments, a string (the operand to decode) 
and a pointer to the struct t_operand. The declaration of the struct 
t_operand must be given in the
file "as.h", and the table-writer can put in it all the information needed for
encoding the operand in machine format.
.IP \0\04:
It examines the mnemonic and calls the associated function, generated by
\fBasg\fR, with pointers to the decoded operands as arguments. This makes it
possible to use the decoded operands in the right hand side of a rule (see
below).
.PP
The default function block_assemble() is called with a sequence of assembly
instructions that belong to one action list. For every assembly instruction 
in
this block assemble() is called. But, if a special action is
required on block of assembly instructions, the table writer only has to
rewrite this function to get a new \fBceg\fR that obliges to his wishes.
.PP
Only four things have to be specified in "as.h" and "as.c". First the user must
give the declaration of struct t_operand in "as.h", and the functions
process_operand(), process_mnemonic() and process_label() must be given 
in "as.c". If the right side of the as_table
contains function calls other than the \fBback\fR-primitives, these functions
must also be present in "as.c". Note that both the "@"-sign (see 4.2.3) 
and "references"
(see 4.2.4) also work in
the functions defined in "as.c". 
.sp
The folowing example shows a part of 8086 "as.h" and "as.c" files:
.nr PS 10
.nr VS 12
.DS L
\f5

/* Constants and type definitions in as.h */

#define        UNKNOWN                0
#define        IS_REG                 0x1
#define        IS_ACCU                0x2
#define        IS_DATA                0x4
#define        IS_LABEL               0x8
#define        IS_MEM                 0x10
#define        IS_ADDR                0x20
#define        IS_ILB                 0x40

#define AX                0
#define BX                3
#define CL                1
#define SP                4
#define BP                5
#define SI                6
#define DI                7

#define REG( op)         ( op->type & IS_REG)
#define ACCU( op)        ( op->type & IS_REG  &&  op->reg == AX)
#define REG_CL( op)      ( op->type & IS_REG  &&  op->reg == CL)
#define DATA( op)        ( op->type & IS_DATA)
#define LABEL( op)       ( op->type & IS_LABEL)
#define ILB( op)         ( op->type & IS_ILB)
#define MEM( op)         ( op->type & IS_MEM)
#define ADDR( op)        ( op->type & IS_ADDR)
#define EADDR( op)       ( op->type & ( IS_ADDR | IS_MEM | IS_REG))
#define CONST1( op)      ( op->type & IS_DATA  && strcmp( "1", op->expr) == 0)
#define MOVS( op)        ( op->type & IS_LABEL&&strcmp("\"movs\"", op->lab) == 0)
#define IMMEDIATE( op)        ( op->type & ( IS_DATA | IS_LABEL))

struct t_operand {
        unsigned type;
        int reg;
        char *expr, *lab, *off;
       };

extern struct t_operand saved_op, *AX_oper;
\fR
.DE
.DS L
\f5

/* Some functions in as.c. */

#include "arg_type.h"
#include "as.h"

#define last( s)     ( s + strlen( s) - 1)
#define LEFT         '('
#define RIGHT        ')'
#define DOLLAR       '$'


process_label( l)
char *l;
{
}


process_mnemonic( m)
char *m;
{
}


process_operand( str, op)
char *str;
struct t_operand *op;

/*        expr            ->        IS_DATA en IS_LABEL
 *        reg             ->        IS_REG en IS_ACCU
 *        (expr)          ->        IS_ADDR
 *        expr(reg)       ->        IS_MEM
 */
{
        char *ptr, *index();

        op->type = UNKNOWN;
        if ( *last( str) == RIGHT) {
                ptr = index( str, LEFT);
                *last( str) = '\0';
                *ptr = '\0';
                if ( is_reg( ptr+1, op)) {
                        op->type = IS_MEM;
                        op->expr = ( *str == '\0' ? "0" : str);
                }
                else {
                        set_label( ptr+1, op);
                        op->type = IS_ADDR;
                }
        }
        else
                if ( is_reg( str, op))
                        op->type = IS_REG;
                else {
                        if ( contains_label( str))
                                set_label( str, op);
                        else {
                                op->type = IS_DATA;
                                op->expr = str;
                        }
                }
}

/*********************************************************************/

mod_RM( reg, op)
int reg;
struct t_operand *op;

/* This function helps to decode operands in machine format.
 * Note the $-operators
 */
{
        if ( REG( op))
                R233( 0x3, reg, op->reg);
        else if ( ADDR( op)) {
                R233( 0x0, reg, 0x6);
                @reloc2( %$(op->lab), %$(op->off), ABSOLUTE);
        }
        else if ( strcmp( op->expr, "0") == 0)
                switch( op->reg) {
                  case SI : R233( 0x0, reg, 0x4);
                            break;

                  case DI : R233( 0x0, reg, 0x5);
                            break;

                  case BP : R233( 0x1, reg, 0x6);        /* exception! */
                            @text1( 0);
                            break;

                  case BX : R233( 0x0, reg, 0x7);
                            break;

                  default : fprint( STDERR, "Wrong index register %d\n",
                                    op->reg);
                }
        else {
                @if ( fit_byte( %$(op->expr)))
                        switch( op->reg) {
                            case SI : R233( 0x1, reg, 0x4);
                                    break;
        
                          case DI : R233( 0x1, reg, 0x5);
                                    break;
        
                          case BP : R233( 0x1, reg, 0x6);
                                    break;
        
                          case BX : R233( 0x1, reg, 0x7);
                                    break;
        
                          default : fprint( STDERR, "Wrong index register %d\n",
                                            op->reg);
                        }
                        @text1( %$(op->expr));
                @else
                        switch( op->reg) {
                          case SI : R233( 0x2, reg, 0x4);
                                    break;
        
                          case DI : R233( 0x2, reg, 0x5);
                                    break;
        
                          case BP : R233( 0x2, reg, 0x6);
                                    break;
        
                          case BX : R233( 0x2, reg, 0x7);
                                    break;
        
                          default : fprint( STDERR, "Wrong index register %d\n",
                                            op->reg);
                        }
                        @text2( %$(op->expr));
                @fi
        }
}
\fR
.DE
.nr PS 12
.nr VS 14
.LP
If a different function assemble() is needed, it can be placed in 
the file "as.c"; assemble() has one argument of type char *.
.NH 2
Generating assembly
.PP
It is possible to generate assembly instead of object files (see section 5), in
which case there is no need to supply "as_table", "as.h" and "as.c". 
This option is useful for debugging the EM_table.
.NH 1
Building a ce
.PP
This section describes how to generate a code expander. The best way to
generate one is to build it in two phases. In phase one, the EM_table is
written and tested.  In the second phase, the as_table is written and tested.
.NH 2
Phase one
.PP
The following is a list of instructions that describe how to make a
code expander that generates assembly instructions.
.IP \0\01:
Create a new directory.
.IP \0\02:
Create the "EM_table", "mach.h" and "mach.c" files; there is no need 
for "as_table", "as.h" and "as.c" at this moment.
.IP \0\03:
type
.br
\f5
install_ceg -as
\fR
.br
install_ceg will create a Makefile, and three directories : ceg, ce and back.
Ceg will contain the program ceg; this program will be
used to turn "EM_table" into a set of C source files ( in the ce directory)
, one for each
EM-instruction. All these files will be compiled and put in a library called
\fBce.a\fR.
.br
The option \f5-as\fR means that a \fBback\fR-library will be generated (in the directory back) that
supports the generation of assembly language. The library is named "back.a".
.IP \0\04:
Link a front end, "ce.a" and "back.a" together resulting in a compiler.
.LP
Now, the EM_table can be tested; if an error occurs, change the table
and type
\f5
.DS
\f5update\fR \fBC_instr\fR
 ,where \fBC_instr\fR stands for the name of the erroneous EM-instruction.
.DE
\fR
.NH 2
Phase two
.PP
The next phase is to generate a \fBce\fR that produces relocatable object
code.
.IP \0\01:
Remove the "ce" and "ceg" directories.
.IP \0\02:
Write the "as_table", "as.h" and "as.c" files.
.IP \0\03:
type
.br
\f5
install_ceg -obj
\fR
.br
The option \f5-obj\fR means that "back.a" will contain a library for generating
ACK_A.OUT(5L) object files, see appendix B. If different "back.a" is used,
omit the \f5-obj\fR flag.
.IP \0\04:
Link a front end, "ce.a" and "back.a" together resulting in a compiler.
.LP
The as_table is ready to be tested. If an error occurs, change the table.
Then there are two ways to proceed: 
.IP \0\01:
recompile the whole EM_table,
.br
\f5
update ALL 
\fR
.IP \0\02:
recompile just the few EM-instructions that contained the error,
\f5  
.br
update \fBC_instr\fR
.FS
This has to be done for every EM-instruction that contained the erroneous
assembly instruction.
.FE
.br
,where \fBC_instr\fR is an erroneous EM-instruction.
\fR
.NH
Acknowledgements
.LP
We want to thank Henri Bal, Dick Grune, and Ceriel Jocobs for their 
valuable suggestions and the critical reading of this paper.
.NH
References
.LP
.[
$LIST$
.]
.bp
.SH 
Appendix A, \fRthe \fBback\fR-primitives
.PP
This appendix describes the routines available to generate relocatable
object code. If the default back.a is used, the object code is in 
ACK A.OUT(5L) format.
.nr PS 10
.nr VS 12
.PP
.IP A1.
Text and data generation; with ONE_BYTE b; TWO_BYTES w; FOUR_BYTES l; arith n;
.VS +4
.TS
tab(#);
l c lw(10c).
text1( b)#:#T{
Put one byte in text-segment.
T}
text2( w)#:#T{
Put word (two bytes) in text-segment, byte-order is defined by
BYTES_REVERSED in mach.h.
T}
text4( l)#:#T{
Put long ( two words) in text-segment, word-order is defined by
WORDS_REVERSED in mach.h.
T}
#
con1( b)#:#T{
Same for CON-segment.
T}
con2( w)#:
con4( l)#:
#
rom1( b)#:#T{
Same for ROM-segment.
T}
rom2( w)#:
rom4( l)#:
#
gen1( b)#:#T{
Same for the current segment, only to be used in the "..icon", "..ucon", etc.
pseudo EM-instructions.
T}
gen2( w)#:
gen4( l)#:
#
bss( n)#:#T{
Put n bytes in bss-segment, value is BSS_INIT.
T}
.TE
.VS -4
.IP A2.
Relocation; with char *s; arith o; int r;
.VS +4
.TS
tab(#);
l c lw(10c).
reloc1( s, o, r)#:#T{
Generates relocation-information for 1 byte in the current segment.
T}
##s\0:\0the string which must be relocated
##o\0:\0the offset in bytes from the string. 
##T{
r\0:\0relocation type. It can have the values ABSOLUTE or PC_REL. These
two constants are defined in the file "back.h"
T}
reloc2( s, o, r)#:#T{
Generates relocation-information for 1 word in the
current segment. Byte-order according to BYTES_REVERSED in mach.h.
T}
reloc4( s, o, r)#:#T{
Generates relocation-information for 1 long in the
current segment. Word-order according to WORDS_REVERSED in mach.h.
T}
.TE
.VS -4
.IP A3.
Symbol table interaction; with int seg; char *s;
.VS +4
.TS
tab(#);
l c lw(10c).
switch_segment( seg)#:#T{
sets current segment to "seg", and does alignment if necessary. "seg" 
can be one of the four constants defined in "back.h": SEGTXT, SEGROM,
SEGCON, SEGBSS.
T}
#
symbol_definition( s)#:#T{
Define s in symbol-table.
T}
set_local_visible( s)#:#T{
Record scope-information in symbol table.
T}
set_global_visible( s)#:
.TE
.VS -4
.IP A4.
Start/end actions; with char *f;
.VS +4
.TS
tab(#);
l c lw(10c).
do_open( f)#:#T{
Directs output to file "f", if f is the null pointer output must be given on
standard output.
T}
output()#:#T{
End of the job, flush output.
T}
do_close()#:#T{
close output stream.
T}
init_back()#:#T{
Only used with user-written back-library, gives the opportunity to initialize.
T}
end_back()#:#T{
Only used with user-written back-library.
T}
.TE
.VS -4
.nr PS 12
.nr VS 14
.bp
.SH 
Appendix B, description of ACK-a.out library
.PP 
The object file produced by \fBce\fR is by default in ACK ACK_A.OUT(5L)
format. The object file consists of one header, followed by
four segment headers, followed by text, data, relocation information, 
symbol table and the string area. The object file is tuned for the ACK-LED,
so there are some special things done just before the object file is dumped.
First, four relocation records are added which contain the names of the four
segments. Second, all the local relocation is resolved. This is done by the 
function do_relo(). If there is a record belonging to a local
name this address is relocated in the segment to which the record belongs.
Besides doing the local relocation, do_relo() changes the "nami"-field
of the local relocation records. This field receives the index of one of the
four
relocation records belonging to a segment. After the local
relocation has been resolved the routine output() dumps the ACK object file.
.LP
If a different a.out format is wanted, one can choose between three strategies:
.IP \ \1:
The most simple one is to use a conversion program, which converts the ACK
a.out format to the wanted a.out format. This program exists for all most
.FS
Not all conversion programs can generate relocation information.
.FE
all machines on which ACK runs. The disadvantage is that the compiler
will become slower.
.IP \ \2: 
A better solution is to change the function output(), do_relo(), do_open()
and do_close() in such a way
that it produces the wanted a.out format. This strategy saves a lot of I/O.
.IP \ \3:
If you still are not satisfied and have a lot of spare time change the
\fBback\fR-primitives in such a way that they produce the wanted a.out format.