XLP Release 2

A.J.Hurst

Version 2.4.0

20080822:162322

Table of Contents

1. Introduction

1.1 Background

Literate Programming is a technique developed by Donald Knuth, and described in his seminal paper in the Computer Journal, 1984. It is both a software process and a suite of tools for supporting that process. Programs written as literate programs go through two (conceptually) independent processes known as tangling and weaving. Tangling creates the actual source code for input to a compiler; weaving creates a documentation file suitable for processing by a document system such as TeX.

More modern versions of literate program tools have attempted to decouple the N x M combinatorial explosion created by the choice of N programming languages and M documentation systems, by restricting the markup and presentational mechanisms. The tool currently in use in the work described here is such a tool: the markup of source documents is done in XML, and the documentation device is handled by XML translators. This reduces the problem from an N x M to an N + M one, where N = 1.

XLP started life as a literate programming tool developed from nuweb, and modified to develop XML code rather than LaTeX or HTML. It was extensively rewritten during 2003 to use an XSLT script rather than a C program to do the tangle and weave processes, so the code is completely platform independent. To run the code, only an XSLT translator (any flavour) is needed. The price for this conversion (apart from time) has been the loss of a number of "syntactic sugar" features, but I don't miss them much. They are gradually being reintroduced, generally in an improved form.

This new version is currently used to define and maintain a range of software that I use on a regular basis.

1.2 Synopsis

This program, or rather, suite of programs, performs literate program analysis and construction over a literate program file. Programs in any language can be constructed, as can documentation in any form. There are two phases to the use of the suite, each of which is handled by an XSLT file. The first of these phases uses a common XSLT file, while the second phase uses an XSLT file that is specific to the form of documentation being generated. For naming purposes, the first phase is known as the warp phase, while the second phase is called the weft phase.

The first phase (warp phase) comprises two passes, corresponding to the tangling and weaving operations of all literate program systems. It is important to make the distinction between phase and pass in this context. This phase is performed by the litprog.xsl file. Each of the passes within this phase is handled by a mode (in the XSL technical sense). The first pass, or mode=tangle, generates the tangled program files, and the second pass mode=weave generates an intermediate file of the woven documentation file.

This documentation file is then processed in phase two of the suite (the weft phase) by a second XSLT file, which is documentation language specific. There is only one pass in the weft phase. Currently, there are two such XSLT files, lit2html.xsl for HTML documentation, and lit2tex.xsl for TeX documentation. The output of this phase is an HTML or TeX document, suitable for processing by a browser (HTML) or by the TeX system itself (TeX). There are a standard set of macros (based upon plain TeX) that are assumed by the TeX translator, and which are available separately.

All three XSLT files (litprog.xsl for the warp phase, and lit2html.xsl and lit2tex.xsl for the weft phase) are described in this document, which is itself a literate program document.

1.3 The source file structure (XLP files)

The warp (first) phase of XLP reads the input XLP file, a document type definition for which is given in section 5, The litprog Document Type Definition (see also the file <litprog.dtd 6.1>). In BNF, the grammar of this file is as follows (terminals are represented in CAPITALS, although the corresponding element names are lower case, except for TableOfContents):

litxmlfile       = TITLE AUTHOR VERSION DATE
                   ( section | misc ) *
                   DocumentHistory?
                 .
section          = TITLE (p | o | d | subsection | misc)* .
subsection       = TITLE (p | o | d | subsubsection | misc)* .
subsubsection    = TITLE (p | o | d | subsubsubsection | misc)* .
subsubsubsection = TITLE (p | o | d)* .
misc             = TABLEOFCONTENTS | FILELIST | MACROLIST | IDENTIFIERLIST 
                 .
o                = (CODE | u | v)* .
d                = (CODE | u | v)* .
u                = USE-REFERENCE .
v                = VARIABLE-REFERENCE .
p                = (TEXT | u | v)* .
DocumentHistory  = Modified* .
Modified         = DATE AUTHOR VERSION COMMENT .

See the User Manual (following) for explanations of these elements.

1.4 Some base definitions

Here we define some important system parameters.

The first one is the location for the library of XSL translations for various base templates. Update this to point at the directory used to store templates such as basic2html.xsl, basic2tex.xsl, and so on (see for example, <lit2html.xsl 4.1> and <lit2tex.xsl 4.33>).

The following list defines all documentation template names. All the names of documentation templates that are passed straight through phase one (the warp phase) should be added to this list, in order to avoid any "No warp-tangle template" or "No warp-weave template" errors.

2. User Manual

2.1 Overview of an XLP Document

An XLP document is a well-formed XML document. For readers unfamiliar with XML, an XML document consists of a set of elements, each flagged with a starting and ending tag, and containing other elements, text, or both. Tags are defined by writing the element name within angle brackets <>, and the ending or closing tag has a leading / before the name. The term "well-formed XML document" means that all elements are strictly nested, and there is one outer level or root element.

<p>An example of a <b>paragraph</b> element, with nested <b>bold</b> elements</p>

The structure of the XLP document is formally defined elsewhere, but an informal description will suffice here. The basic outline of an XLP literate program looks like this:

        <?xml version="1.0"?>
        <!DOCTYPE litprog SYSTEM "/home/ajh/lib/dtd/litprog.dtd">
        <litprog>
        ... literate program content goes in here ...
        </litprog>
      

The first line of this program is a declaration that the document is an XML document, constructed according to the rules of XML version 1.0. This line is optional. The second line is mandatory, and specifies that the rules of this particular XML document, an XLP (or XML Literate Program) document, are defined in the separate file /home/ajh/lib/dtd/litprog.dtd.

The top-level or root element is a litprog element. This element contains all the literate programming content. Such content comes in two basic forms: documentation and code. Each of these two forms have their own set of elements. There is limited mixing of these elements, but generally an element is either pure code or pure documentation. These forms are described in the next section.

The basic model is that the document has the structure of a typical descriptive article: opening components are followed by substantive components, which are then followed by closing components. Opening material are things like title, author, date, which are required (if present) to be in a particular order. The substantive content is encapsulated within sections, each contain possible subsections, subsubsections, and even subsubsubsections. Closing material includes things like tables of content and indices.

Interspersed within the document are the code chunks, which are file and code definitions. These may generally appear at any point in the document, but it is their document order that defines how they are assembled into the overall code files. Readers familiar with other literate programming systems may view this arrangement as a conventional alternation of code and documentation fragments, with a superimposed document structure.

2.2 Documentation in an XLP Document

Documentation in an XLP document is built from the set of elements:

author*

The author of this literate program. Can appear in two places: at the top level, after the title element, or within a Modified element, after a date element.

b

markup text in bold face

c

markup text in code (typewriter) face

col

The column specification element, used in tables. It is an empty element, with one attribute, width, defining the width of a column as a fraction (<=1.0) of the total page width.

comment*

The last element within a Modified element, describing the nature of this particular modification.

d

define a code chunk (see code)

date*

Can appear after version in the document header, or as the first element in a Modified element. It defines the date of the document, or modification, respectively.

DocumentHistory*

A sequence of Modified elements, each of which contains a date, author, version, comment sequence of nested elements, each of which documents the appropriate data about the various modifications to this literate program. If it is present, it must be the last element in a litprog element.

dq

place double quotes around text

filelist*

A place marker to indicate where the list of files generated should be inserted. It can appear at the top level, or within a (sub)section.

i

markup text in italic face

identifierlist*

A place marker to indicate where the table of identifier cross references should be inserted. It can appear at the top level, or within a (sub)section.

item*

Can only appear within a description, enumerate or itemize element, where it defines one of the list items.

le

The less than or equal to symbol, <=.

macrolist*

A place marker to indicate where the list of macro names should be inserted. It can appear at the top level, or within a (sub)section.

Modified*

contains date, author, version, comment elements, defining attributes of each modification of the literate program.

o

define an output file (see code)

p

markup a paragraph

section*

Can only appear as a top-level element within the litprog element.

subsection*

Can only appear within a section element.

subsubsection*

Can only appear within a subsection element.

subsubsubsection*

Can only appear within a subsubsection element.

TableOfContents*

Defines the place in the literate program where the table of contents should be inserted. Can only appear at the top level of the document, or within a section or sub(sub...)section.

term*

Can only appear within a description element, where it defines the (documentation) term being defined.

TeX

The TeX token.

td*

As for HTML, within a table row element (tr), defines a table data cell.

th*

As for HTML, within a table row element (tr), defines a table header cell.

title*

When this element appears immediately after the opening litprog tag, it is the title of this literate program. Within a section or sub..section, it is the title of the corresponding (sub..)section.

tr*

As for HTML, defines a table row.

u

A use reference to a code fragment. In documentation, during the weaving pass, the reference becomes a link to the code fragment, unless an optional attribute expand is present and non-empty, when the reference is expanded to its value. In code fragments, during the tangling pass, the reference is replaced by the expanded code fragment, as a (recursive) macro expansion.

uri

markup a Universal Resource Indicator

v

markup a variable (see code)

verb

markup verbatim text in-line

verbatim

markup verbatim text as a display (paragraph) item.

version*

Can appear in either the literate program header or a Modified element. In both cases, it follows an author element.

2.3 Code in an XLP Document

Code elements in an XLP literate program take two forms: file definitions, and code chunks. Code chunks are fragments of code used to build other code chunks or file definitions. For this reason, the distinction between code chunks and file definitions is often blurred, and they are referred to collectively as code chunks. It should be obvious from the context when it is important for the distinction to be made.

The code fragments are assembled through a process of expansion (in technical terms, a "macro expansion") that replaces references to other chunks by the code defined in that chunk.

Code within a chunk can be quite arbitrary: there is no interpretation or parsing of the code, except for XML elements. Remember that the document is one large XML file, and text within a code chunk is interpreted according to the rules of XML. In practice, this means that the characters "less than" (<) and "ampersand" (&) have special meanings, and must be escaped if they are used within code text. The escape sequences are &lt; and &amp; respectively.

Alternatively, the block quoting mechanism of XML can be used. This entails surrounding the entire text that needs to be quoted with the special sequences <![CDATA[ and ]]>. All text within these quote sequences will appear verbatim. It follows that if the sequence ]]> is to appear within the quoted text, it too must be escaped, for example with ]]>]<![CDATA[]]>]<![CDATA[>]]>><![CDATA[ (the three separate characters, each surrounded by close then open escape sequences). Strictly speaking, breaking the sequence once would suffice, but this is a more sesquipedalian approach !

There are three code elements that may appear within documentation (and already mentioned above):

o

An output element. This element defines a file built by the literate program. The name of the file is given by the value of a required attribute, the file attribute. Note that more than one file element may have the same name, when the file is considered to be the concatenation of all similarly named chunks, arranged document in order. All file definitions are collected into a list accessed by the element filelist.

d

A code chunk defining element. The name of the code chunk (by which it may be referenced) is given by the value of a required attribute name. Note that more than one code chunk may have the same name, when the code chunk is considered to be the concatenation of all similarly named chunks, arranged in document order. It is an error to have names that are common to both code and file chunks. All code definitions and uses are collected into a list accessed by the element macrolist.

v

A variable element. This may be either a defining occurrence, or a using occurrence, although within documentation it is unlikely to be the former. Within documentation, is usually used to highlight discussion about a particular variable within the code. See below for use within a code chunk. All variable definitions and uses are collected into a list accessed by the element identifierlist.

There are three elements that may appear within code chunks (i.e., within o or d elements). These are:

u

A code chunk using element. This element is normally empty, and defines the point at which a code chunk is expanded into the current code chunk or output file. It has a mandatory attribute name, which cross-references the corresponding code chunk(s).

It may also have the optional attribute include, which, when it has the value no, negates the inclusion of the reference code chunk. This may be used to "comment out" code, while still retaining the documentation surrounding the excluded code. The documentation is annotated accordingly.

A second optional attribute is expand, which normally has the default value no. When explicitly set to yes however, it forces the expansion of the chunk into the document at that point. When called within code chunks, this is the default behaviour, but when called within documentation chunks, this changes the representation of the macro from a link to the expanded text of the macro. This is useful for defining macros used within the documentation, rather than code generation (for example, the date at the head of this document was generated in this way). Warning: multiple chunks are not handled correctly, as of version 2.2.0.

v

A variable element. See also use within documentation, above. This may be a defining occurrence, or a using occurrence, depending upon the optional attribute var. If var has the (default) value use (var="use"), then the occurrence is marked up as a use of the variable. If the attribute has the value def (var="def"), then the occurrence is marked up as a defining occurrence. Note that there is no analysis of the content of the variable element, which allows arbitrary lexical forms of variable names to be used. (But be wary of languages such as perl, or bash shell scripts, which use \$ and other signs to flag usage of the variable. These annotations should be excluded from the variable name.)

com

A comment element. This element may be used to markup in-line comments within the code. How the comment is rendered depends upon the subsequent weft phase, but it will presumably capitalize upon the contextual position. (Aside: the tangling of the output file could conceivable include the comment as an in-line comment - the difficulty here is that the literate programming system has no knowledge of the syntactic form of such in-line comments.)

2.4 Running XLP

To use the XLP literate programming system, you need an XSLT translator. I use xsltproc (see LibXML), but other XSLT translators will work just as well with the files described herein. In what follows, we assume the xsltproc translator.

There are two phases to running the system, called the warp and weft phases. The warp phase performs tangling and weaving, and generates an intermediate file for the weft phase. To run the warp phase, use the call:

  xsltproc litprog.xsl yourXLPfile.xlp >yourXLPfile.xml

For the weft phase (document generation), use the call

  xsltproc lit2html.xsl yourXLPfile.xml >yourXLPfile.html

This is for HTML generation. For TeX documentation, replace the two occurrences of html in the above line with tex.

Note that if documentation is not required, the weft phase may be omitted.

3. The warp, or literate program phase

This phase is responsible for reading and translating the source xlp file. There are two passes to this phase, known as the tangle and weave passes. Each pass performs a sweep over the input file, applying templates to the elements as they are seen. In the first pass, the XSLT mode of mode="tangle" is used; in the second pass, mode="weave" is used.

In the tangle pass, all documentation chunks are effectively ignored, while file chunks cause a new document to be written, containing the code within the file chunk, with all code chunk references expanded out through a process of macro expansion. Each file chunk creates a new text document named with the given file name, which can then be used for further processing by compilers, etc..

In the weave pass, a new translation of the source XLP document is made. This new document is given a .xml extension, and its contents consist of much the same contents as are in the original XLP file, but with some elements renamed, and additional housekeeping information issued to assist the final weft phase.

3.1 Templates defined in litprog.xsl

3.2 litprog.xsl: the warp translator

This is the XSLT transformer script for phase one, the warp phase of the literate programming suite. It is responsible for tangling the output documents, and weaving the XML document for input to the weft phase.

3.2.1 litprog: root template

The root template is responsible for the two passes of tangling and weaving. Note that the tangle phase generates its own documents (in the XML technical sense), while all the documentation for the weft phase is generated as the "default" output of this phase by the tangle pass.

3.2.2 litprog: define variables

The variable crossrefs is set to a value built from a newline separated list of code and file chunks, with each entry (or line) in the form name@location, where name is the name of the chunk or file, and location is the chunk number defining that file or chunk.

The value stored as the value of a chunk number is multi-valued, in the form s.p, where s is the section number and p is the part number within the section.

The two variables premacro and postmacro define the escape sequences used respectively to start and end a macro text.

3.2.3 litprog: define script parameters

The script parameters give some control over the behaviour of the literate programming system. Currently two are defined: Verbose and machine.

Verbose can be set to a non-negative integer. When 0 (the default), no tracing information is given. Progressively higher values give more information about the progress of tangling and weaving.

machine is used to define the target of compilation. Not currently interpreted.

3.2.4 litprog: define procedure templates

Define all the callable templates for the warp phase.

3.2.5 litprog: do-replace template

do-replace is a procedure with three parameters, text, replace, and by. Multiple occurrences of replace in the string text are recursively replaced by the value of by, and this rebuilt value is returned as the value of the template call.

3.2.6 litprog: count-trailing-blanks template

count-trailing-blanks, as its name suggests, (recursively) counts the trailing blanks in the string parameter string and returns this value. The initial value of the parameter count should be set as 0. The actual value returned by the procedure is this initial value plus the number of trailing blanks, hence the need to initialize it correctly.

3.2.7 litprog: lookupchunkno template

lookupchunkno returns a list of the chunk numbers of a named chunk. The name of the required chunk is passed as the parameter chunkname, and the procedure calls an auxiliary recursive template lookupchunknoR to find the actual chunk numbers.

{Note 3.9.1}

The first key part of the lookup algorithm. Jump straight to the match in the table of the required chunk name. If it is there, the find variable is set to the remainder of the table starting with the match. If it is not there, the find variable will be empty.

{Note 3.9.2}

The second key part of the lookup algorithm. Return the first part of the find string (it is the desired chunk number, and use the (remainder of the) find string as the lookup table for the next recursion. The algorithm is order n, where n is the number of matches in the cross reference table for chunkname.

lookupchunknoR takes as parameters the required chunkname, a partially searched table of cross references, and the separator string used to separate matches for the chunk name. It returns a sep separated list of values that match the second half of all (chunkname, chunkno) pairs in the cross reference list.

The sep separator is passed as a parameter, since on the first call, it must be empty. Otherwise we would return lists of the form ,4,10,15 rather than 4,10,15.

This chunk is conditionally included to provide debugging information about the progress of the chunk number lookup procedure.

3.2.8 litprog: lookup template

3.2.9 litprog: substparms template

This callable template takes a text string representing a chunk definition, and substitutes for any formal parameters used within the chunk. Formal parameters are found by virtue of them being enclosed within 4 sets of square brackets, as defined by the global variables premacro and postmacro.

Note that there is a further complication in that formal parameters nested inside actual parameters have been escaped, by using 4 sets of curly brackets. These escapings must be restored before parameter expansion can begin. This is done by substituting square brackets for the curly ones.

This is a challenging routine, as evidenced by the number of debug components built in! Basically, we split the incoming text string into a pre-formal parameter part (the head), and a post-formal parameter part (the tail), as indicated by the premacro and postmacro strings of square brackets. The formal parameter itself is looked up in the parameter table, and a result string is built by concatenating the pre-string, the parameter value, and a recursively expanded post-string.

We build a new version of the text string parameter in the variable etext. Two nested calls on the do-replace template are made, one to replace the premacro escape, and one to replace the postmacro escape.

Build the result of macro expansion recursively. The (first) macro parameter is chopped out, leaving a head and tail string. The parameter is looked up to find its replacement value, and the result returned is the head string, the replacement text value, and the recursive expansion of the tail string.

3.3 litprog: the tangle pass

3.3.1 litprog: tangle all litprog files

{Note 3.16.1}

see point 1

{Note 3.16.2}

see point 2

{Note 3.16.3}

see point 3

{Note 3.16.4}

see point 4

{Note 3.16.5}

see point 5

{Note 3.16.6}

see point 6

To tangle a complete litprog, we scan through all file chunk definitions, expanding them into an appropriately named file. Issue a message as each one is processed.

(version 2.4.0) There was a bug here: multiple chunks of "o" elements did not get appended, since each one started a new document, thus overwriting the previous chunk. Here the new logic is this:

1. Do all tangling by searching for "o" elements.
2. For each one found, check if it is the first one with this particular file name. If it isn't, ignore it.
3. If it is, issue a message that we are tangling the file, then ....
4. ... start a new document with this file name ...
5. ... tangle this chunk and ...
6. ... tangle all following "o" chunks with the same filename.

3.3.2 litprog: tangle a single file definition chunk

This is done in a manner similar to code chunk tangling. Here, only the leading new line character is removed. Why? Good question. When I have an answer, I'll add it to this documentation.

3.3.3 litprog: tangle a code definition

A code chunk is tangled in three stages.

Firstly, the chunk has any nested elements resolved and expanded, as appropriate. This expansion is saved in the local variable chunki (initial chunk).

Secondly, if the chunk ends in a newline character and the trim attribute is set to yes, the trailing newline is removed. The result of this operation is saved in the variable chunke (end trimmed chunk).

Thirdly, a leading newline character is removed, if present. The result of this operation is saved in the variable chunkes (start and end trimmed chunk).

The resultant value is returned as the result of tangling this code fragment or chunk.

3.3.4 litprog: tangle variable declarations

In tangle mode, variable definitions and uses are basically passed straight through to the weft phase.

3.3.5 litprog: discard documentation in tangle mode

All documentation fragments are ignored in tangle mode.

3.3.6 litprog: tangle all chunks into a single use

A use call on a chunk name has been made, which must be expanded into the document-order concatenation of all code chunk definitions with the same name.

3.3.6.1 litprog: replace the machine parameter in chunk name

This bit is a hack. I really want a method to replace key strings with parameters, but there isn't anything I can think of other than using this "do-replace" with a hard-coded substitution string. Incidentally, do-replace is copied from Michael Kay's excellent book "XSLT Programmer's Reference".

3.3.6.2 litprog: check whether to include a use chunk

"include" is an attribute that allows the user to comment out chunks of code, simply by putting an 'include="no"' attribute into the chunk call. It defaults to "yes", so set up that default.

3.3.6.3 litprog: save use parameters

3.3.6.4 litprog: compute indentations for use expansion

Now some stuff to determine indentation. This is heavy XML. We first pull in the most recent text node in the tree. This will have the indentation blanks before this call on 'u', and so we count and extract those blanks.

Following that processing, the value of variable $blanks will be inserted after all new lines in the text generated by this call on 'u'. That, together with the indentation that appeared before the call itself, will mean that all lines of the replacement text will have the same indentation, assuming that they are themselves aligned - any additional indentation is preserved.

3.3.6.5 litprog: check definitions for non-use

Now check that we have some definitions of this chunk. Count them and issue a warning if the count is zero.

3.3.6.6 litprog: include raw text and expand indentation

Firstly, cross reference this call on the chunk.

Then the real code expansion takes place. We collect up all matching chunk definitions, and add them in document order.

If the chunk is to be included, build the raw text of it, and then expand all newlines with indentation as computed above.

debug this call on expanding a chunk

Show all the chunks that match this use instance.

Build the parameter table. This is a string where each actual parameter is surrounded by the pre- and post- macro strings (as defined by the variables premacro and postmacro respectively), then concatenated together in parameter document order.

Build the raw chunk corresponding to this use instance match.

Now expand the actual parameters into the raw chunk, replacing all formal parameters.

Now rebuild the replacement text by expanding all newlines with their appropriate indentation. This is to preserve the indentation established in the actual call on this chunk.

3.3.7 litprog: default tangle template

This is the default tangle template, invoked if there is no other template defined to handled this element. Provide an error message in the event of no tangle template.

3.4 litprog: the weave pass

3.4.1 litprog: weave a litprog

Not really much to this, since the whole litprog file is scanned in document order, generating the intermediate documentation file as we go.

3.4.2 litprog: weave a code or file definition

{Note 3.41.1}

Build the uses attribute list, by scanning all child u elements, and looking up their chunk numbers.

The d | o template, in the weave mode, builds a CODE element in the woven .xml file, CODE elements define code fragments. Such elements can have the following attributes:

type

for a d fragment, the type is define, indicating a chunk definition. For an o fragment, the type is file.

name

The name of the code chunk or fragment. For example, the name of the following code chunk is litprog: d | o-weave template. For a file fragment, the name attribute gives the name of the file.

number

code chunks are numbered in sequence from the beginning, for cross referencing purposes.

Generate a helpful debug message to show what other fragments this code chunk uses.

{Note 3.43.1}

Scan through all in-line comments (such as this one), and for each one, generate a NOTE element, cross referenced by the ordinal number of this comment within the chunk.

This template, d/com|o/com, handles comments with code fragments in the weave pass. The comment itself is handled later, but for the moment, we just insert a forward reference to it as a link in the current code fragment. The reference is identified by the ordinal position of the comment within this code fragment.

3.4.3 litprog: weave a variable markup

The elements 'u' (use) and 'v' (value) both refer to 'd' (define) elements elsewhere in the document. The difference is that in weave mode 'u' inserts a hot link to the actual definition, whereas 'v' inserts the actual value defined in the 'd' element. This is equivalent to macro expansion, and is what happens to 'u' elements in tangle mode. In tangle mode, 'v' elements are ignored. In other words, 'v' elements are only relevant to document fragments.

3.4.4 litprog: u-weave template

The u-weave template is responsible for inserting a link to the definition of the referenced code fragment, in-line in the documentation. Note that at this stage, the include parameter is simply passed on: it is not interpreted as to whether the code fragment is actally included until the documentation construction phase.

3.4.5 litprog: p-weave template

3.4.6 litprog: section-weave template

3.4.7 litprog: listing templates

3.4.8 litprog: default weave template

This template handles all elements not elsewhere handled in the weave pass. Note that a table of documentation related elements is kept, and if an element name is matched in this table, no warning message is issued. All elements, whether flagged or not, are passed straight through. Note that all the attributes must be copied as well.

In performing the table lookup, note that the separator must be present at both ends. This is to ensure that names that are substrings are not matched implicitly.

4. The weft, or documentation phase

This phase retranslates the output file from phase 1, the XML file. There is a translation script for each class of documentation processors. Two are defined here, lit2html.xsl, for translation to HTML, and lit2tex.xsl, for translation to TeX.

4.1 lit2html.xsl

4.1.1 lit2html: handle the root element

4.1.2 lit2html: handle the literate program translation

4.1.3 lit2html: handle documentation fragments

The abstract gets a title, and indentation

Centre the author in heading font

Ditto the date

The title is a bit more important

The subtitle is less important

Label the version

Build documentation paragraphs

4.1.4 lit2html: handle code fragment references

I think this one is out-of-date. Check if VALUE elements are ever created by < >

Not sure about this one either.

4.1.5 lit2html: handle a code fragment use

{Note 4.13.1}

I generate the I tag literally here because the use of xsl:element generates an I element with a leading newline - NOT what is wanted here!

4.1.6 lit2html: handle code chunk definitions

There are two basic choices for the code body. Short text only fragments containing no new lines are formatted directly onto the definition line, to emphasize the fact that no new lines are part of the code fragment. Code bodies containing more than one element <lit2html: body contains several children > (as when a macro call is expanded within the body), or containing at least one new line <lit2html: body contains new line characters >, are formatted into a separate paragraph.

Two examples for the last paragraph! Note that the second half of the condition can be explained the same way, even though it contains a new line character, since the new line is escaped.

Build a list of chunks in which this code chunk is referenced, and store it in the variable reflist.

If the variable reflist is not empty, display the list of chunk numbers for each of the chunk names in the list. Only unique numbers need be displayed, hence the nested if statement within the //CODE for-each, rather than another for-each which would itemize repeated uses.

Build a list of all the chunk numbers that also define this chunk (that is, they have the same name). All of these chunks are concatenated in document order to build the resultant code chunk.

4.1.7 lit2html: handle notes and comments in code fragments

4.1.8 lit2html: handle variable definitions and uses

Variable cross referencing generates links between the variable use and definition points. A definition generates an anchor of the form "VAR-variable name", and a use generates a link to the anchor similarly formed. These links and anchors are highlighted in red.

4.1.9 lit2html: handle maintaintable

4.1.10 lit2html: TableOfContents

TableOfContents builds a table of contents by scanning all sections, subsections and subsubsections, using their ordinal positions to construct a stepped list of section numbers and titles

4.1.11 lit2html: handle a code text element

4.1.12 lit2html: construct file definition list

4.1.13 lit2html: construct chunk definition list

4.1.14 lit2html: construct identifier definition list

4.1.15 lit2html: handle macro parameters

4.1.16 lit2html: handle document history

4.1.17 lit2html: special character templates

Define here all the special characters that cannot be represented exactly (either because there is no corresponding glyph, or because they require escaping in TeX

4.1.18 lit2html: default template

Three templates, which a) strip leading newlines from text nodes, b) handle non-text nodes in stripping leading new-lines, and c) a catch-all default template for all other cases, that prints a warning message as appropriate. The element and its attributes are passed through unchanged.

4.2 lit2tex.xsl

This file was originally constructed on 20030529:215331

The purpose of this file is to convert the XML output of the literate programming XML tangle and weave translator <litprog.xsl 3.1> to a tex document, suitable for processing by plain TeX.

It is assumed that the original litprog document doc.xlp has been processed by <litprog.xsl 3.1> to doc.xml. This script converts it to doc.tex

4.2.1 lit2tex: handle the root node

4.2.2 lit2tex: handle various documentation elements

4.2.3 lit2tex: handle a literate program translation

4.2.4 lit2tex: handle paragraphs

4.2.5 lit2tex: handle reference calls

4.2.6 lit2tex: handle code fragment definitions

This template formats a code fragment. There is special work to be done on the name, because of escaping TeX characters. Underlines are replaced with their escaped versions and the result saved in the variable escname. Note that the distinction between the escaped string $escname and the unescaped string @name must be preserved, depending upon whether we are using the string for TeX purposes, or for XML purposes.

If the variable reflist is not empty, display the list of chunk numbers for each of the chunk names in the list. Only unique numbers need be displayed, hence the nested if statement within the //CODE for-each, rather than another for-each which would itemize repeated uses.

4.2.7 lit2tex: handle code use calls

4.2.8 lit2tex: handle macro parameters

The space before the end verbatim seems to be required, as the end verbatim seems to consume the blank immediately before it. Don't know why.

4.2.9 lit2tex: handle variable definitions and uses

4.2.10 lit2tex: handle in-line comments in code

4.2.11 lit2tex: handle maintenance table documentation

4.2.12 lit2tex: build table of contents

The table of contents is constructed using an halign macro, set to the page width, and indenting each section heading according to the depth of the heading. The last template is right justified, and carries the page number for the start of the section (actually, the heading of the section). We define the table template, then enter loops to process headings at each level with a (nested) loop. Close the table of contents with a closing brace and a newline.

Construct a (one-level) section number, no indent, format with the title text and a page reference using the sanitised xref title text

Construct a two-level section number, indent it once, format with the title text and a page reference using the sanitised xref title text

Construct a three-level section number, indent it twice, format with the title text and a page reference using the sanitised xref title text

4.2.13 lit2tex: build file list

The file list macro builds a fairly simple TeX table of file identifiers, together with a list of the chunk numbers in which they are defined. Once the table parameters are setup, scan through all CODE elements looking for file definitions with unique file names. Escape underline characters from these names. Then generate a list of all chunk numbers defining this file. Note the need to keep the unescaped file name $thisname distinct from the escaped file name $escname.

4.2.14 lit2tex: build identifier list

4.2.15 lit2tex: handle chunk definition list

The challenge in tabulating a list of chunk definitions is in handling the variable widths of the fields, given that some of them may require multiple lines to fit within the page boundaries. This really requires a two pass algorithm, of greater complexity than we can afford here. Consequently, we settle for some manually determined widths, which can be overridden if necessary. The default widths are 50\% for the chunk name, and 25\% for each of the definition and use lists.

Scan through all CODE elements to find the definitions, and build these into a table of identifiers, showing both the point of definitions, and all uses of the identifier.

We define three variables to store the widths of the three chunk name list columns: chunk name, chunk definitions, and chunk uses. These values are either defined by the user with attributes to the macrolist element, or are set by default to the values 0.50, 0.25, 0.25 respectively.

This next bit is really messy, basically because of the horrendous TeX code being produced. This is what gets produced by this bit of code for the macrolist reference in this program itself (see later).

\halign{\tabskip=0em\advance\hsize by -1em
\vtop{\hsize=0.60\hsize#\hfil}\hskip0.5em&
\vtop{\hsize=0.20\hsize#\hfil}\hskip0.5em&
\vtop{\hsize=0.20\hsize#\hfil}\cr
\bf Identifier&\bf Defined in&\bf Used in\cr

Each of the table columns gets a width that can be set in the original XML call on macrolist via the optional attributes namewidth, definewidth and usewidth. Each of these parameters should be a fraction of 1.0, defining the fractional width of the page taken up by that column, and their sum should be <=1.0

4.2.16 lit2tex: handle document history

Generate a table to capture the document history.

Define the various column width parameters for the Document History table. Each of these is set to a default value, unless there is an explicit attribute setting, when they are set instead to the value passed in the attribute.

4.2.17 lit2tex: text node handling

These templates handle text nodes in the TeX weft phase. The callable template replacesubstring is used in a couple of the templates.

4.2.17.1 lit2tex: callable template replacesubstring

This callable template takes a string, and returns a new string with substrings replaced consistently.

4.2.17.2 lit2tex: replace dollar signs with escaped dollar signs

A straightforward replace of any dollar sign by their escaped versions in TeX. These otherwise would start mathematics mode.

4.2.17.3 lit2tex: filter at signs in text nodes

This template sanitises code text to double up any at signs. This is a hangover from the C version of the litprog, and will be removed at a future date. The offending code is the TeX routine \verb"codeverbatim" in file "almostverb".

4.2.17.4 lit2tex: filter underlines in non-verbatim text nodes

We need to make sure that text in text nodes gets appropriate markup from the TeX point of view. Particular cases are underscores and dollar signs. Munge through all text nodes to replace these with their sanitised text equivalents.

The exceptions are where the text occurs in various verbatim modes. The XPath expression in the match filters out these exceptions.

The following template filters text nodes to replace underlines with their TeX escaped equivalents. Note that the filtering is NOT applied when the containing node is a verb or CODE node, as these contexts have their own verbatim quoting mechanisms.

4.2.18 lit2tex: special character templates

Define here all the special characters that cannot be represented exactly (either because there is no corresponding glyph, or because they require escaping in TeX

4.2.19 lit2tex: default template

A catch-all for any undefined templates. Issue a warning message and pass the element through unchanged.

5. weft: Common weft fragments

6. The litprog Document Type Definition

{Note 6.1.1}

The text parameter entity defines the standard element choices that go into the various documentation chunks. Add any extra document element choices in here, and they will be automatically distributed into all doco fragments.

{Note 6.1.2}

The misc parameter entity defines the miscellaneous element choices that go into the various documentation chunks. These elements are usually only specified once per document, but there is no restriction on them being specified more often.

6.1 litprog.dtd: Literate Programming elements

o

o elements are file definitions. The file attribute defines the name of the file.

d

d elements are code chunk definitions. The name attribute defines the name of the code chunk, by which it can be referenced in u elements

u

u elements are uses of code chunks. They are normally empty elements, and may appear in either o or d elements, or in documentation. In tangling, they are replaced by the expanded form of the reference code chunk. In weaving, they are replaced by links to the referenced code definition. If a weave mode u element is accompanied by the attribute expand, which is non-empty, it is replaced by its expanded form.

v

v elements are variable instances. The variable name forms the element content. If accompanied by the attribute var set to def, the instance is taken to be a defining instance, otherwise if the var attribute is set to use, or does not appear, the instance is regarded as a use of the variable.

com

com elements are comments. They may be used within code or file chunks to annotate the code. Depending upon the context, these comments may be marked-up into the documentation.

6.2 litprog.dtd: Bibliographic elements

6.3 litprog.dtd: Heading and Sectioning elements

6.4 litprog.dtd: Miscellaneous elements

6.5 litprog.dtd: table elements

6.6 litprog.dtd: uri elements

6.7 litprog.dtd: DocumentHistory elements

6.8 litprog.dtd: Entities

7. The LitprogXML Document Type Definition

8. File Index

9. Macro Index

10. Identifier Index

Document History