PEP 258 – Docutils Design Specification | peps.python.org
PEP 258 – Docutils Design Specification PEP 258 – Docutils Design Specification Author : David Goodger <goodger at python.org> Discussions-To : Doc-SIG list Status : Rejected Type : Standards Track Requires : 256 , 257 Created : 31-May-2001 Post-History : 13-Jun-2001 Table of Contents Rejection Notice Abstract Specification Docutils Project Model Publisher Readers Parsers Transformer Writers Input/Output Docutils Package Structure Front-End Tools Document Tree Error Handling Python Source Reader Processing Model AST Mining Docstring Extraction Rules Attribute Docstrings Additional Docstrings Choice of Docstring Format Identifier Cross-References Stylist Transforms Project Web Site Copyright Acknowledgements Rejection Notice While this may serve as an interesting design document for the now-independent docutils, it is no longer slated for inclusion in the standard library.
Abstract This PEP documents design issues and implementation details for Docutils, a Python Docstring Processing System (DPS). The rationale and high-level concepts of a DPS are documented in PEP 256 , “Docstring Processing System Framework”. Also see PEP 256 for a “Road Map to the Docstring PEPs”. Docutils is being designed modularly so that any of its components can be replaced easily. In addition, Docutils is not limited to the processing of Python docstrings; it processes standalone documents as well, in several contexts.
No changes to the core Python language are required by this PEP. Its deliverables consist of a package for the standard library and its documentation. Specification Docutils Project Model Project components and data flow: +---------------------------+ | Docutils : | | docutils . core . Publisher , | | docutils . core . publish_ * () | +---------------------------+ / | \ / | \ 1 , 3 , 5 / 6 | \ 7 +--------+ +-------------+ +--------+ | READER | ----> | TRANSFORMER | ====> | WRITER | +--------+ +-------------+ +--------+ / \\ | / \\ | 2 / 4 \\ 8 | +-------+ +--------+ +--------+ | INPUT | | PARSER | | OUTPUT | +-------+ +--------+ +--------+ The numbers above each component indicate the path a document’s data takes.
Double-width lines between Reader & Parser and between Transformer & Writer indicate that data sent along these paths should be standard (pure & unextended) Docutils doc trees. Single-width lines signify that internal tree extensions or completely unrelated representations are possible, but they must be supported at both ends. Publisher The docutils.core module contains a “Publisher” facade class and several convenience functions: “publish_cmdline()” (for command-line front ends), “publish_file()” (for programmatic use with file-like I/O), and “publish_string()” (for programmatic use with string I/O).
The Publisher class encapsulates the high-level logic of a Docutils system. The Publisher class has overall responsibility for processing, controlled by the Publisher.publish() method: Set up internal settings (may include config files & command-line options) and I/O objects. Call the Reader object to read data from the source Input object and parse the data with the Parser object. A document object is returned. Set up and apply transforms via the Transformer object attached to the document.
Call the Writer object which translates the document to the final output format and writes the formatted data to the destination Output object. Depending on the Output object, the output may be returned from the Writer, and then from the publish() method. Calling the “publish” function (or instantiating a “Publisher” object) with component names will result in default behavior. For custom behavior (customizing component settings), create custom component objects first, and pass them to the Publisher or publish_* convenience functions.
Readers Readers understand the input context (where the data is coming from), send the whole input or discrete “chunks” to the parser, and provide the context to bind the chunks together back into a cohesive whole. Each reader is a module or package exporting a “Reader” class with a “read” method. The base “Reader” class can be found in the docutils/readers/__init__.py module. Most Readers will have to be told what parser to use.
So far (see the list of examples below), only the Python Source Reader (“PySource”; still incomplete) will be able to determine the parser on its own. Responsibilities: Get input text from the source I/O. Pass the input text to the parser, along with a fresh document tree root. Examples: Standalone (Raw/Plain): Just read a text file and process it. The reader needs to be told which parser to use. The “Standalone Reader” has been implemented in module docutils.readers.standalone .
Python Source: See Python Source Reader below. This Reader is currently in development in the Docutils sandbox. Email: RFC 822 headers, quoted excerpts, signatures, MIME parts. PEP: RFC 822 headers, “PEP xxxx” and “RFC xxxx” conversion to URIs. The “PEP Reader” has been implemented in module docutils.readers.pep ; see PEP 287 and PEP 12 . Wiki: Global reference lookups of “wiki links” incorporated into transforms. (CamelCase only or unrestricted?) Lazy indentation?
Web Page: As standalone, but recognize meta fields as meta tags. Support for templates of some sort? (After <body> , before </body> ?) FAQ: Structured “question & answer(s)” constructs. Compound document: Merge chapters into a book. Master manifest file? Parsers Parsers analyze their input and produce a Docutils document tree . They don’t know or care anything about the source or destination of the data. Each input parser is a module or package exporting a “Parser” class with a “parse” method.
The base “Parser” class can be found in the docutils/parsers/__init__.py module. Responsibilities: Given raw input text and a doctree root node, populate the doctree by parsing the input text. Example: The only parser implemented so far is for the reStructuredText markup. It is implemented in the docutils/parsers/rst/ package. The development and integration of other parsers is possible and encouraged. Transformer The Transformer class, in docutils/transforms/__init__.py , stores transforms and applies them to documents.
A transformer object is attached to every new document tree. The Publisher calls Transformer.apply_transforms() to apply all stored transforms to the document tree. Transforms change the document tree from one form to another, add to the tree, or prune it. Transforms resolve references and footnote numbers, process interpreted text, and do other context-sensitive processing. Some transforms are specific to components (Readers, Parser, Writers, Input, Output).
Standard component-specific transforms are specified in the default_transforms attribute of component classes. After the Reader has finished processing, the Publisher calls Transformer.populate_from_components() with a list of components and all default transforms are stored. Each transform is a class in a module in the docutils/transforms/ package, a subclass of docutils.transforms.Transform . Transform classes each have a default_priority attribute which is used by the Transformer to apply transforms in order (low to high).
The default priority can be overridden when adding transforms to the Transformer object. Transformer responsibilities: Apply transforms to the document tree, in priority order. Store a mapping of component type name (‘reader’, ‘writer’, etc.) to component objects. These are used by certain transforms (such as “components.Filter”) to determine suitability. Transform responsibilities: Modify a doctree in-place, either purely transforming one structure into another, or adding new structures based on the doctree and/or external data.
Examples of transforms (in the docutils/transforms/ package): frontmatter.DocInfo: Conversion of document metadata (bibliographic information). references.An
The rest of this page is on the live site.