annotate Logo

annotate -- Annotations and Docstrings for Lua Values

Introduction · Basic Usage · The annotate Module · The annotate.check Module · The Module · The annotate.test Module · Changelog · Download · Installation · Contact · License


There are basically two ways for documenting code in a dynamically typed programming language like Lua: you can write static documentation like external readme files or comments that can be extracted by specialized documentation tools, or you can annotate Lua values with runtime information. The first approach enables you to extract useful information without running any code, a popular tool for this is LDoc. One well-known representative of the second approach is Python with its docstrings. One advantage of this approach is that you can easily process those runtime annotations and e.g. provide interactive help, or type checking.

This module uses the ideas presented here and here to provide a basis for docstring handling in Lua, and flexible argument and return value checking for Lua functions.

Basic Usage

To add a docstring annotation to a Lua value (like e.g. a function) you use the annotate base module:

$ cat > test1.lua
-- loading the module returns a callable table
local annotate = require( "annotate" )

-- annotated function definitions consist of a call to the
-- annotate function concatenated to a normal (anonymous)
-- function value
local func = annotate[=[
The `func` function takes a number and a string and prints them to
the standard output stream.
]=] ..
function( a, b )
  print( a, b )

func( 1, "hello" )

The annotate module itself doesn't do anything with the docstrings and Lua values, but hands both to modules like e.g. the annotate.check module:

$ cat > test2.lua
local annotate = require( "annotate" )
require( "annotate.check" )  -- we ignore the return value for now

local func = annotate[=[
The `func` function takes a number and a string and prints them to
the standard output stream.

    func( a, b )
        a: number
        b: string
]=] ..
function( a, b )
  print( a, b )

func( 1, "hello" )
func( 2, true )        --> line 17

When run, the above example will output:

$ lua test2.lua
1       hello
lua: func: string expected for argument no. 2 (got boolean).
stack traceback:
        [C]: in function 'error'
        [compiled_arg_check]:48: in function 'argc'
        ../src/annotate/check.lua:842: in function 'func'
        test2.lua:17: in main chunk
        [C]: in ?

The annotate Module

By itself the annotate module does nothing except providing syntax for associating a docstring with a Lua value. It does so using a __call metamethod that takes a string and returns an object with a __concat metamethod. So the general usage looks like this:

local annotate = require( "annotate" )
local annotated_v = annotate[[some string]] .. v

What you put into the docstrings is your business, but I suggest markdown, because it looks good as plain text, and you can convert it to many formats, e.g. using a converter like pandoc. There are also Lua libraries for converting markdown texts.

To actually do something with the annotations you need handler modules that get registered with the annotate module. For this the annotate module provides a register method:

annotate:register( function( v, docstring ) ... end [, replace] )

There are two kinds of callback functions, those that wrap or replace the original value, and those that don't. For the former kind, the replace argument must evaluate to a true value. Those callbacks are called in the order of registration, and they must return the replacement value. The non-replacing kind of callback is called after all modifying callbacks are handled, but the order in which they are called is unspecified (and shouldn't matter anyway). Their return values are ignored.

The annotate.check Module

The annotate.check module registers itself with the annotate module when require'd (see above). For every function that gets annotated, it parses the given docstring and extracts argument and return type information from a special function signature in the docstring. It then replaces the original function with a type checking version. Various fields in the annotate.check module table can be used to fine-tune the type checking (see below).

Function Signature Syntax

The annotate.check module scans paragraphs (sequences of characters delimited by \n\n) in the docstring and takes the first that looks like a function signature as used in the Lua reference manual. A function signature starts with a name or function designator (module names + function name, delimited by .), followed by a parameter list in parentheses, an optional return value specification, and if necessary a mapping of parameter names to types. You can put Lua-style single line comments at all places where whitespace is allowed.


pcall( f [, arg1, ...] ) ==> boolean, any*
    f   : function  -- the function to call in protected mode
    arg1: any       -- first argument to f
    ... : any*      -- remaining arguments to f

tonumber( any [, number] ) ==> nil/number

table.concat( list [, sep [, i [, j]]] ) ==> string
    list: table     -- an array of strings
    sep : string    -- a separator, defaults to ""
    i   : integer   -- starting index, defaults to 1
    j   : integer   -- end index, defaults to #list

table.insert( list, [pos,] value )
    list : table    -- an array
    pos  : integer  -- index where to insert (defaults to #list+1)
    value: any      -- value to insert filename [, mode] )
        ==> file               -- on success
        ==> nil,string,number  -- in case of error
    filename: string           -- the name of the file
    mode    : string           -- flags similar to fopen(3)

file:read( ... ) ==> (string/number/nil)*
    ...: (string/number)*      -- format specifiers

file:seek( [whence [, offset]] ) ==> number
                                 ==> nil, string
    self  : file               -- would default to `object`
    whence: string
    offset: number

os.execute( [string] )
        ==> boolean
        ==> boolean/nil, string, number

mod.obj:method( [a [, b] [, c],] [d,] ... )
        ==> boolean            -- when successful
        ==> nil, string        -- in case of error
      a: string/function       -- a string or a function
      b: userdata              -- a userdata
                               -- don't break the paragraph!
      c: boolean               -- a boolean flag
      d: number                -- a number
    ...: ((table, string/number) / boolean)*

Predefined Type Checking Functions

The table check.types (where check is the result of the require-call) comes with some predefined type checking functions. Those predefined type checking functions only cover basic Lua data types, see below for how to add your own application specific checking functions.

Some optional type checkers are defined if the necessary modules and functions are available:

Tuning the Type Checker

Checking for basic Lua types already helps, but typically support for application specific data types is needed. To register a new type simply add the type checking function to the types sub-table.

local annotate = require( "annotate" )
local check = require( "annotate.check" )
check.types.file = function( v )
  return io.type( v ) == "file"

local func2 = annotate[=[
    func2( [fh] )
        fh: file  -- a file handle
]=] ..
function( out )
  out = out or io.stdout
  out:write( "Hello World!\n" )

You can disable type checking for the following function definitions by setting the enabled field to false. In that case the annotate.check module doesn't replace the original function.

check.enabled = false

Previously defined functions are unaffected by this change.

You can selectively enable/disable type checking for arguments and return values using the arguments and return_values flags. Again, this only affects functions defined after this change.

check.arguments = true
check.return_values = false

By default the type checking module throws an error for undefined type checkers, or if a docstring for a function does not have a function signature. You can change that by providing a custom error function:

check.errorf = function( msg ) print( msg ) end -- print warning
-- check.errorf = function() end -- ignore completely

The Module

The module registers itself with the annotate module when require'd to provide interactive help for all Lua values with an annotation. It can also wrap other help modules (like e.g. ihelp) to delegate help requests for values not having a docstring. Assuming help is the return value of the require-call:

If the argument to the help module (or to the lookup function) is a string, tries to require the string (and suitable substrings) looking for a Lua value with an annotation using the string as a path.

The annotate.test Module

The annotate.test module is a simple unit testing module inspired by Python's doctest. The idea is to provide code examples in the docstrings using the syntax of the interactive Lua interpreter. The code examples can be executed and verified as working Lua code by this module. It registers itself with the annotate module when require'd and stores the test code it finds in the docstrings in an internal table for later execution. The tests are started by calling the result of the require( "annotate.test" ) call.

local annotate = require( "annotate" )
local test = require( "annotate.test" )
-- ... some function definitions with annotations
test( 1 ) -- parameter is output verbosity (0-3, default is 1)

The test output quotes the function name, if the docstring also contains a type signature (as for the annotate.check module, see there) before the test code section. Test results and statistics are written to the standard error channel.

If you want to take unit testing really seriously, the test code will become way too big to be included in the docstrings. In this case you should consider using a designated unit testing module for most of the tests, and only use this module to make sure the examples in the documentation stay correct.

Test Syntax

The beginning of the test code section is denoted by a simple header or a markdown header (in atx-style format).

After the header, any line indented 4 spaces is either a line containing Lua code, or a line containing output of the Lua code before. Lua code starts with ">" or ">>". Each line of Lua code is compiled as a separate chunk if possible (like in the interactive interpreter). All tests for a single annotation share a custom environment containing a modified print function, and the global variable F that refers to the value the annotation is for (useful for testing local functions), as well as __index access to the default global environment.

The output lines are matched against values returned from the Lua chunks (via return or =), against error messages, and against the output of the print function (but only if used directly in the test code, the output of Lua's standard print function cannot be matched). The string ... in an output line is equivalent to the string pattern .-, a group of one or more whitespace characters is equivalent to %s+. Additionally, whitespace at the end of the output is ignored.

An empty line (containing only whitespace) is skipped (unless it starts with 4 spaces in which case it is considered an output line). The test/example section ends with the first non-empty line that is not indented at least 4 spaces.


local annotate = require( "annotate" )
local test = require( "annotate.test" )

func = annotate[=[
This is function `func`.

   func( n ) ==> number
       n: number

    > return func( 1 )
    > function f( n )
    >> return func( n )
    >> end
    > = f( 2 )
    > = f( 2 ) -- this test will fail!
    > print( "hello\nworld" )
    > = 2+"x"
    ...attempt to perform arithmetic...

This is the end of the test code!
]=] ..
function( n )
  return n

test() -- run the tests

The result is:

### [++-++] function func( n )
### TOTAL: 4 ok, 1 failed, 5 total



The source code (with documentation and test scripts) is available on github.


There are two ways to install this module, either using luarocks (if this module already ended up in the main luarocks repository) or manually.

Using luarocks, simply type:

luarocks install annotate

To install the module manually just drop annotate.lua and annotate/*.lua somewhere into your Lua package.path. You will also need LPeg (at least for the type checker, and the test module).


Philipp Janda, siffiejoe(a)

Comments and feedback are always welcome.


annotate is copyrighted free software distributed under the MIT license (the same license as Lua 5.1). The full license text follows:

annotate (c) 2013 Philipp Janda

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.