And as in uffish though he stood,
The jabberwock, with eyes of flame,
Came whiffling through the tulgey wood,
And burbled as it came!

Introduction.

Both the debugger and the syntax analyzer of the editor must have a notion of the syntax LISP. The debugger need to know the syntax to figure out where he has to add instrumentation code and the syntax analyzer must known how to divide the source in the different elements needed for syntax coloring. Because the syntax of LISP can be extended via macros, a fixed syntax hardcoded in the debugger or the analyzer is out of the question. Jabberwocky uses a special language to define the syntax. This makes it possible to extend the debugger and syntax coloring (for example when you have added your own control structures). The basic principle is as follows :

The syntax is defined in a syntax file.
process-definition-file uses this syntax file to generate lisp code that does the instrumentation needed by the debugger.
process-definition-file-to-java uses this syntax file to generate java code that does the syntax analysis needed for the syntax coloring.
Once this LISP and JAVA code is generated it must be compiled to be used by Jabberwocky.

When Jabberwocky is installed a default syntax file is used to generate the necessary code, so that the standard LISP syntax is handled correctly. If you extend the syntax of LISP and you want to see these extensions reflected in the debugging and highlighting you must modify this syntax file and then generate the necessary code.

If you want to modify the debugger keep in mind the following points :

Function calls are handled correctly by the debugger, you don't have to do anything at all for this.
The standard LISP macros are also coverred with in the default syntax.
User macros are the problem.The debugger tries to do his best for the ones you define (first he expands the macro call and then tries to add debugging code to the expanded code) but the result is not always what you expect. It is here that you can make improvements for debugging. For syntax coloring a user defined macro call or a user defined function call gets the same color.
Errors in the syntax can ruin the debugger and the syntax analyzer (part of there code is actually generated on base of the syntax), so be careful. Nothing can go wrong if you make sure that you have made backup copies of the following files:

lispsyntax (this is the source file defining the working of the current debugger).
debugcode.x86f, debugcode.fas (this is the core of the debugger for CMUCL,CLISP loaded when a interaction buffer starts.
lispanalyzergen.java, lispanalyzergen.class (this is the core of the syntax analyzer for the editor.

Modifying the syntax

The steps to modify the debugger are as follows:

Make a workdirectory, make this your current directory, copy the source distribution file 'Jabberwocky.0.1.tgz' in this directory and use 'tar -xvzf Jabberwocky.x.y.tgz' to expand the source distribution of Jabberwocky in it.
Do a 'cd Jabberwocky' and make a backup of 'lispsyntax'. Modify 'lispsyntax' to include your own rules.
Run 'install' to regenerate Jabberwocky and install it.
Now start Jabberwocky again and test the debugger and the code highlighting. You can use the example files testclisp.lisp and testcmucl.lisp as a testing startpoint.

Language to define the syntax

The extension language is strongly based on the syntax descriptions used in ANSI LISP, so that writing extensions is as simple as writing a syntax diagram.

The default file used by the debugger

To give you an idea of how the extension language looks like, look at 'lispsyntax' the source used to generate the parser/transformer of the debugger/editor delivered with this package. Although the language used is not yet defined it should look familiar to you.

Syntax of the language.

The language is composed of the following elements :

[comments]
Everything on a line after a ';' is considered a comment and is neglected.
[white spaces]
Used as separators, the following is a white space :
- blank space
- new line
- tab
- return
[definitions]
Defines named syntax definitions which can be used in other syntax definitions. The syntax is :

symbol = expression
symbol is the name of the definition and expression is a syntax definition. Recursive definitions are possible. Use definitions to simplify syntax definitions or to use recursion (see the loop macro for a example).

[expressions]
A syntax definition of the form:

symbol
Represents a specific LISP symbol
_symbol
Represents a match with any LISP symbol
~symbol
Represents a lexical binding to a lisp variable
#symbol
The expression in this place will be instrumented for debugging, represents a LISP form
@
Used to indicate the instrumentation of the enclosing list
[ expression ... expression ]
List of optional expressions, occurring zero or one time.
[ expression ... expression ]*
List of optional expressions, occurring zero or more times
{ exp...exp | ... | exp ... exp }
Match must be done with one of the set of expressions in the list
{ exp ..exp | ... | exp ... exp }*
Match must be done with one of the set of expressions in the list, but more then one match is possible
( expression expresion ...)
Represents a LISP list of expressions.
(expression ... expresion . expression)
Represents a dotted LISP list.
"text"
Any string

[symbols]
The symbols used in LISP (case insensitivity is automatically)
- Any string of characters with the exception of white spaces , @ , _ ,~,#,(,),[,]

A source in our language is a text file containing definitions and expressions.

Semantics of the language

The syntax tells us what the wellformed expressions and definitions are in our language but it says nothing about their meaning, for this we need a little bit of semantics. The best way to understand the semantics of the language is the consumer/producent metaphor. When a expression in our language is applied on a lisp expression two things can happens :

The expression recognizes the lisp expression consuming part of it and producing another lisp expression.
The expression does not recognize the expression an it generates a throw, no lisp expression is produced.

Lets now put these ideas in practice on the different type of expressions of our language. Let P be the list produced, E a expression in our language and L the lisp expression on which we applies expressions in our language. The parser/generator in the debugger will apply each expression on a given lisp expression until it gets not a throw and the lisp expression is fully consumed, the produced list is then the lisp expression with debug code added. If this sounds inefficient you are right this is just a semantic explanation, our language is actually compiled to become the parser/generator of the debugger which has the same effect as our semantic explanation, but he does it in a more efficient way.
The syntax highlighter works in the same fashion, but instead of generating code, it notes which symbols are variables,functions or macros in a lispform.
Lets look now at the semantics of the different elements in our language.

Symbol
If we apply a 'symbol' on a lisp expression L=(e1 e2... en) or L=() we have a throw if e1 is not equal to our 'symbol' or if L is the empty list. If e1 is equal to our 'symbol' then we append P with the symbol and L becomes (e2 ... en).
Let P=() , L=(defun f (n) (princ n)) and E=defun then applying E on L gives L=(f (n) (princ n)) and P becomes (defun).

Let P=(), L=(defun f(n) (princ n)) and E=let then applying E on L gives a throw.
"text"
If we apply "..." on a lisp expression L=(e1 e2 ... en) or L=() we have a throw if L is empty or if e1 is not a string. In all other cases L becomes (e2 ... en) and the first element e1 is added to P. You can think of "text'' as standing for any string.
_symbol
If we apply '_symbol' on a lisp expression L=(e1 e2.... en) or L=() we have a throw if L is empty. In all other cases L becomes (e2 ... en) and the first element e1 is added to P. You can think of _symbol as standing for any list element which must not be changed.
Let P=() , L=(a b c d) and E=_sym then applying E on L gives L=(b c d) , P=(a).

Let P=(),L=((a b) c d) and E=_sym then applying E on L gives L=(c d) , P=((a b))

Let P=(),L=() and E=_sym then applying E on L gives a throw.
~symbol
If we apply '~symbol' on a lisp expression L=(e1 e2 .. en) or L=() we have a throw if L is empty or e1 is a list. In all other cases L becomes (e2 ... en) and the first element e1 is added to P. Also e1 is added to the lexical environment of the debugger. During executing of the debugged code the system test if the variable is defined in the lexical environment at the execution point and if it is so the binding of the variable is saved. This means that you can refer to the binding of this variable during debugging. You don't have to worry when this variable can be referred (the system keeps track of this) only indicate that this is defined to become a variable.
Let P=() , L=(a b c d) and E=^sym then applying E on L gives L=(b c d) , P=(a).

Let P=(),L=((a b) c d) and E=^sym then applying E on L gives a throw.

Let P=(),L=() and E=^sym then applying E on L gives a throw.
#symbol
If we apply '#symbol' on a lisp expression L=(e1 e2 ... en) or L=() we have a throw if L is empty. In all other cases L becomes (e2 .... en) and the system tries to add debugging code to e1 before it is added to P. The debugging code added makes that you can place a breakpoint on e1, and see where e1 is located in the source. In some cases no debugging code is added (if e1 is not a list or if e1 represents a macro not recognized by the debugger). You can use # to indicate that you should be able to set breakpoints.
Let P=() , L=(a b c d) and E=^sym then applying E on L gives L=(b c d) , P=((add-debug-code a)).

Let P=(),L=() and E=^sym then applying E on L gives a throw.
@symbol
We can always apply @ to L. It consumes nothing and it produces nothing , its solely purpose is for its side effect, it allows you to set a breakpoint on the whole expression where it is part of. Use this in (defun ...) (let ...) just before the body of these functions.
[expression1 ... expressionn]
If we apply [expression1 ... expressionn] on a lisp expression L . It will first try to match expression1 .... expressionn and then the next syntax expression. If this fails it will only applly the expressions after the [...]. Consider this as a kind of optional syntax.
E=[a b] c , L=(a b c d) and P=() then applying E on L gives L=(d) , P=(a b c)

E=[a b] c , L=(c d) and P=() then applying E on L gives L=(d) , P=(c)

E=[a b] d , L= (c d) and P=() then applying E on L gives a throw
{e11...e1n|...|em1....emk}
If we apply {e11...e1n|...|em1....emk} on a lisp expression L . It will first try to apply e11...e1n on L followed by all expressions after { ... } if this causes a throw, it will try e21...e2l followed by the rest ... . You can consider { ... } as a kind of or where you want to try different alternatives.
Example 1 E={a b} c , L=(a c d) and P=() then applying E on L gives L=(d),P=(a c)

E={a b} c, L=(b c d) and P=() then applying E on L gives L=(d),P=(a b)

E={a b} c, L=(c d) and P=() then applying E on L gives a throw
[expression1 | ... |expressionn]* , {e11 ...e1n | ... | em1...emk}*
If we apply {...}* or [...]* on a lisp expression. We first try to apply as many times as possible [...] or {...} and then use the syntax expressions after the {...}* ([...]*). Use this if you have more then occurrence of the same elements.
E=[a b]* c, L=(a b a b c) and P=() then applying E on L gives L=(c) , P=(a b a b)
(expression1 ... expressionn)
If we apply (expression1 ... expressionn) on L=(e1 ... en) or L=() we get a throw if L is empty or if e1 is not a list . The result in the other cases is the result of applying expression1 .... expressionn on e1 where we start with a empty result list, this result list is then added to the original list. If e1 is not consumed fully we have also a throw.
Example 2 E=(a b c) , L=((a b c) d) , P=() then applying E on L gives L=(d) , P=((a b c))

E=(a b c),L=((a b c d) d), P=() then applying E on L gives a throw.
definitions (name = expression)
A definition gives a name to a expression , applying this name has the same effect as applying the expression. You can use definitions to to define a expression ones and then use it many times. Definitions can be used recursively , one can refer in a definition to itself.

Cavecats

When using recursive definitions make sure that you do not introduce infinite looping.