Lessons learned from the translation of documentation from LaTeX to HTML

Xavier Leroy (INRIA)

Introduction

The initial document

An HTML version of this document was desirable for several reasons:

• Allow browsing of the documentation via the Web for those who wish to learn about the language, yet do not feel like printing the whole thing.
• Support word searching in the documentation (via CGI-scripts), which allows experienced programmers to locate specific library functions or command options much faster and much more precisely than a paper index.
• Serve as a starting point for providing on-line documentation for the microcomputer ports of Caml Light (through the Windows help system under MSDOS and through DocMaker on the Macintosh). Previous experiments with DocMaker showed that it is nearly impossible to go directly from the DVI file to a decent on-line document. The simplicity and popularity of HTML makes it a good intermediate format.

The Caml Light documentation is about 200 pages long. It was therefore out of the question to rewrite it manually in HTML. Automatic translation was not obvious either, due to the typographical gap between TeX and HTML. For instance, even though the documentation does not contain math formulas properly speaking (no integrals, no greek letters, ...), TeX's math mode is frequently used in BNF definitions and program samples to get metavariables, subscripts, superscripts, ... Typographical enrichments (italics, bold, monospaced fonts) are used extensively for disambiguation (e.g. distinguish terminal symbols from BNF operators) and must be rendered faithfully. The documentation also contains many tables and a few line drawings, which caused some difficulties.

Preparing the LaTeX source

• Word searching ignores the text contained in the bitmaps.
• The bitmaps translate poorly to the Macintosh or Windows online documentation systems.
• The bitmaps are generally hard to read, since their resolution usually does not match that of the HTML viewer.
• Transmission times are increased.

To avoid the recourse to bitmaps and allow the production of decent HTML, we introduced a number of TeX macros in the LaTeX source, to ``abstract over'' the typesetting details and make the semantics of the LaTeX source more explicit. For instance, we wrote \var{x} instead of $x$ to denote the metavariable named x. Similarly, the n-th element of a sequence v was typed \nth{v}{n} instead of $v_n$. The same technique was also used to eliminate low-level constructs and environments such as center and tabular.

For typesetting with TeX, the new constructs (\var, \nth, etc) were simply macro-defined as their old form, so the end result was the same. For translation to HTML, however, they were specially recognized and the intended meaning was rendered in the best possible HTML way: for instance, \nth{v}{n} becomes <i>v(n)</i>.

The overall approach is basically that of LaTeX for structuring TeX document (instead of typesetting a section header by hand, just write \section{...} to communicate your intentions to the typesetting program), but extended to lower-level elements of the text: metavariables, sequence elements, entries in precedence tables, ...

For the parts of the documentation automatically generated, we modified the programs that generated them in LaTeX to either output HTML directly (embedded in a \begin{rawhtml} ... \end{rawhtml} environment) or output simplified LaTeX that would look bad on paper but translates nicely to HTML.

Finally, a few nasty uses of math mode (arrays of formulas) had to be translated manually to HTML and embedded in the source documentation as \begin{rawhtml} ... \end{rawhtml} blocks, following immediately the original TeX code bracketed by \begin{latexonly} ... \end{latexonly}. The presence of the two versions (LaTeX and HTML) side by side in the source helps keeping them consistent during modifications. About 0.2% of the source had to be manually translated this way; the remaining 99.8% are automatically processed.

Generating HTML

• The definitions of many non-standard commands and environments we use can be communicated to latex2html using simple LaTeX definitions \newcommand and \newenvironment, but a few others (most notably the alltt environment) cannot be defined this way and require modification of latex2html itself. The innards of latex2html are undocumented, scarcely commented, and the Perl code is far from self-explanatory.
• The memory requirements of latex2html appear to be superlinear in the size of the input document. When we ran it on the full manual, the process died after having exhausted all the available virtual memory space (about 200M). We then tried to run latex2html independently on each chapter; it still used uncomfortably large amounts of memory (around 100M) on the biggest chapters; moreover, glueing together the HTML pages obtained for each chapter (inter-chapter links, main table of contents) is difficult to do automatically and painful to do by hand.

We therefore had to write our own translator from the extended subset of LaTeX used to HTML. The translator consists in two main parts:

• The translator proper reads the whole LaTeX document and outputs a single huge HTML document. It is written in Caml Light, using heavily the lexical analyzer generator camllex (the Caml equivalent to lex for C). Since we did not use Perl but a modern, type-safe high-level programming language with good memory management, our translator has negligible memory requirements, runs quickly, is easy to extend, and took little time to develop.
• The output of the translator is then split in smaller pages (at <H1> and <H2> headers, basically). Finally, the pages are ``threaded'' together by inserting ``Next'' and ``Previous'' buttons at the top and bottom of the pages. Two simple Perl scripts do the job adequately.

You may judge the final result and compare it with the DVI file obtained from the original.

The sources of the translator are available here. Some background about the Caml language is helpful to understand them.

Lessons learned

The HTML language

HTML viewers

HTML generators and transformers

• They are hard to find due to the lack of well-known repositories. A few catalogues of tools are available on the Web, but incomplete and often outdated.
• Many are quick hacks that have been put together in a hurry and made public after very little testing. ``It works for me; have fun!''.
• Others (such as latex2html) are more mature and better supported, but still not robust enough. In particular, latex2html and a few others bomb on large inputs.

The immaturity of the field probably explains many of these weaknesses. But we also believe that the widespread use of Perl to program these tools is also partly responsible for the situation. Perl is a fine generalization of the Unix line-oriented tools, and it works great for programs of the kind ``read a line, massage it, print it''. Many CGI scripts fall in this category, but not the more advanced tools we need to produce HTML from other high-level sources. Perl is inherently not suited to the parsing and transformation of structured languages such as LaTeX and HTML. Reading the whole input in a string and rewriting it over and over with s/.../.../eg just does not cut it in terms of clarity and performance; it's like playing the Well-tempered clavier on a Jew's harp. Languages with high-level parsing capabilities, real data structures and clean semantics are clearly needed here.

In which source language should we write documentation?