Lua Programming Gems

edited by Luiz Henrique de Figueiredo, Waldemar Celes, Roberto Ierusalimschy.

ISBN 978-85-903798-4-3.

Copyright c

2008 by the editors and individual contributors. All rights reserved.

Book cover by Pedro de Mazza Cerqueira. Lua logo design by Alexandre Nako.

Typesetting by the editors using LATEX.

Book web site: http://www.lua.org/gems/

Feisty Duck Digital

Book Distribution

www.feistyduck.com

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii

Foreword, by Cameron Laird . . . . . . . . . . . . . . . . . . . . . . . . . . .

ix

Lua and Lightroom, by Mark Hamburg . . . . . . . . . . . . . . . . . . . . . .

xi

Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xiii

I

Programming Techniques

1

Lua Per-Thread Library Context . . . . . . . . . . . . . . . . . . . . . .

3

Doug Currie

2

Lua Performance Tips . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

Roberto Ierusalimschy

3

Vardump: The Power of Seeing What’s Behind . . . . . . . . . . . . .

29

Tobias S ülzenbr ück and Christoph Beckmann

4

Serialization with Pluto . . . . . . . . . . . . . . . . . . . . . . . . . . .

33

Ben Sunshine-Hill

5

Abstractions for LuaSQL . . . . . . . . . . . . . . . . . . . . . . . . . .

43

Tom ás Guisasola Gorham

6

Boostrapping a Forth in 40 Lines of Lua Code . . . . . . . . . . . . . .

57

Eduardo Ochs

7

Effecting Large-Scale Change (with little trauma) using Metatables . .

71

S érgio Alvares Maffra and Pedro Miller Rabinovitch

II

Design Techniques

8

MVC Web Development with Kepler . . . . . . . . . . . . . . . . . . .

85

Andr é Carregal and Yuri Takhteyev

9

Filters, Sources, Sinks, and Pumps . . . . . . . . . . . . . . . . . . . .

97

Diego Nehab

10

Lua as a Protocol Language . . . . . . . . . . . . . . . . . . . . . . . .

109

Patrick Rapin

11

Lua Script Packaging . . . . . . . . . . . . . . . . . . . . . . . . . . . .

119

Han Zhao

12

Objects, Lua-style . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

129

Reuben Thomas

13

Exceptions in Lua

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

135

John Belmonte

III Algorithms and Data Structures

14

Word Ladders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

149

Gavin Wraith

15

Building Data Structures and Iterators in Lua . . . . . . . . . . . . . .

155

Luis Carvalho

16

A Primer of Scientific Computing in Lua . . . . . . . . . . . . . . . . .

173

Luis Carvalho

17

Complex Structured Data Input . . . . . . . . . . . . . . . . . . . . . .

201

Julio M. Fern ández-D´ıaz

18

Lua Implementations of Common Data Structures . . . . . . . . . . .

211

Matthew M. Burke

19

Tic-Tac-Toe and the Minimax Decision Algorithm . . . . . . . . . . . .

239

Rafael Savelli and Roberto de Beauclair Seixas

IV Game Programming

20

Using Lua in Game and Tool Creation . . . . . . . . . . . . . . . . . .

249

Konstantin Sokharev and Vadim Groznov

21

A Dynamic and Flexible Event System for Script-Driven Games . . . .

259

Robert Oates

22

Lua for Game Programming . . . . . . . . . . . . . . . . . . . . . . . .

269

Steve Gargolinski

23

Designing an Efficient Lua Driven Game Scripting Engine . . . . . . .

281

Nicolas Peri

V

Embedding and Extending

24

Enhanced Coroutines in Lua

. . . . . . . . . . . . . . . . . . . . . . .

291

Patrick Rapin

25

Using Lua in Pascal . . . . . . . . . . . . . . . . . . . . . . . . . . . .

301

Jeremy Darling

26

Porting Lua to a Microcontroller . . . . . . . . . . . . . . . . . . . . . .

313

Ralph Hempel

27

Writing C/C++ Modules for Lua . . . . . . . . . . . . . . . . . . . . . .

325

Ralph Steggink and Wim Couwenberg

28

Interpreted C Modules . . . . . . . . . . . . . . . . . . . . . . . . . . .

337

J ér ôme Vuarand

Preface

It gives us great pleasure to publish this collection of Lua gems. Not only does

it record some of the existing wisdom and practice on how to program well in

Lua, but it also reflects the maturity of the Lua community. It is gratifying

to see that Lua has motivated other people to learn it well and to share their

knowledge with other users. In well-written articles that go much beyond the

brief informal exchange of tips in the mailing list or the wiki, the authors share

their mastery of all aspects of Lua programming, elementary and advanced.

Producing this book has required several steps. In response to a call for con-

tributions, we received over 70 abstracts, selected 43, and received full versions

for 28 of these. The authors received our comments and suggestions to prepare

the final version of their articles. The whole process took two years, much longer

than we had imagined. The selection of abstracts proved to be surprisingly dif-

ficult. Many potentially good submissions could not be accepted due to space

limitations. Despite the long time it took and the amount of work it required

(or because of it!), we are very happy to have this collection of articles on Lua

contributed by members of our community. We trust the book was worth waiting

for.

We thank all the authors for their hard work on the articles and everyone

that submitted abstracts in the first phase.

We also thank the whole Lua

community for its friendliness and expertise. The active participation of our

users has been to us a constant source of motivation for improving Lua. Finally,

we give our warm thanks to Cameron Laird and Mark Hamburg for writing

forewords to this book.

Additional material and errata will appear in the book web site:

http://www.lua.org/gems/

The Lua team

Rio de Janeiro, November 2008

Foreword

by Cameron Laird

When I need a programming language that’s as easy as possible to embed,

I choose Lua. Lua isn’t just supple, free, portable, and compact, though; it’s

also powerful — and to get the most out of it, I’m glad I have Lua Programming

Gems.

I need to explain that I mean something specific by that. Most of my read-

ing is on the ’Net: I look up references, I read tutorials for unfamiliar material,

I moderate a half-dozen Wikis, and I chat about specific techniques with col-

leagues on-line. My consumption of books has nose-dived. Lua Programming

Gems is a book worth reading, though: its individual chapters get across ideas

that simply aren’t explained anywhere else.

Lua Programming Gems emphasizes practicality in a way I like. While the

six authors in Part III certainly employ classroom concepts correctly in their

examinations of “Algorithms and Data Structures”, they do it all with working

Lua code. The same pattern is apparent throughout Lua Programming Gems:

it’s filled with ideas likely to help me in my next programming project.

If you’re new to Lua, you might be anxious about what you’ll find. You

can see that Lua offers definite advantages, but how hard is it to pick up

what’s undeniably a minority language? Lua Programming Gems will ease your

concerns: the authors write clearly, modestly, and even deftly. The very first

chapter, for example, tackles difficult material, including dynamically-allocated

per-thread storage. The tone is consistent throughout the book. Rather than

show off their expertise or indulge in private jokes, habits common for authors

from other domains, the Lua Programming Gems authors focus on the specific

details and examples that best teach their chosen topics. They make it inviting

to dig deeper in Lua than you might do on your own.

Among the highlights of Lua Programming Gems for me: Part IV gives

insight into “Game Programming”, an area where I’ll probably never work,

although many of the techniques apply more broadly; Part V on “Embedding

and Extending” is crucial for much of the programming I like; and Chapter 13,

“Exceptions in Lua”, is a particular interest of mine.

Do you want to “program well in Lua”? The Lua team set that as a goal when

it first announced its plans for Lua Programming Gems. The final result fulfills

that goal; you’ll like it.

Lua and Lightroom

Mark Hamburg

Founder, Adobe Photoshop Lightroom

When we started work on the project that would become Adobe Photoshop

Lightroom, we knew we wanted to make scriptability an important part of our

story, so early on we reviewed the usual suspects. What drew us to Lua was

its combination of simplicity, power, ease of embedding, and relatively high-

performance. Having a straightforward license helped too when it came time

to talk to Adobe’s lawyers. Personally, as an old Scheme fan, I was drawn to

its first-class closure support. I also found the coroutine system intriguing. The

relative minimalism also resonated with a back-to-basics attitude that had us

weaning ourselves away from intensive C++ usage and back toward C.

Still, it was hard to position Lua as anything other than an obscure choice.

We could cite heavy use in the games community and we had set out with a

mission of learning something from game developers, but if asked what mate-

rials one could turn to learn Lua or where we would find experienced Lua pro-

grammers, the answers were limited. For the former, we had the well-written

reference manual, some good material on the Lua users wiki, and an intelligent

forum on the Lua mailing list. This was good material, but there wasn’t a lot of

it. For the latter question, our answer was essentially “Any programmer worth

hiring ought to be able to learn Lua quickly.” This was a situation we were pre-

pared to deal with and the arrival of Programming in Lua certainly helped, but

it was easy to understand why it might be off putting to someone looking in from

the outside.

Why this matters is that along with Lua’s simplicity come some issues that

make people with backgrounds in other languages stumble. The beauty of a

small core is that there is a real opportunity for mastery. This is one of C’s great

strengths as well. That small core, however, comes at a price. For example, Lua

has no syntax for exception handling. C doesn’t either but having one seems

almost required in modern languages. Lua has a syntax for object-oriented

message sends, but the actual implementation of an object system or class

system is a roll-your-own affair in Lua. As a result, one sees such issues raised

repeatedly on the Lua users list as people new to the language start using it and

then ask “But what about

?”

Programming in Lua provides answers to some of these questions but those

answers are necessarily terse. Lua Programming Gems dives deeper on these

issues and many more. It shows ways to deal with threading — an issue we

went through a few iterations on in Lightroom — and gives extended examples

of how to hook Lua into your application. You may not always like the answers.

For example, the object system presented here is quite minimalist. In the spirit

of Lua, however, you remain free to roll your own using or not using the ideas

presented here. The value comes in seeing the well worked examples together

with a discussion of motivations and comparisons to other approaches. If this

book had been around during Lightroom’s development, we probably would

have happily adopted some of the techniques it presents while simply taking

inspiration from others. As it was, we largely had to find our own way and while

that was rewarding in itself, the Lua community and particularly new Lua users

can be happy to now have a field guide that maps out some of the trails.

The broader lesson from Lightroom that I would like to leave Lua users with

is that you should let it pervade your work. Lua is sometimes described as being

a language for gluing pieces together, but as we discovered that glue can extend

quite deep. We started out looking for a scripting language for a native code

application. Then we started thinking it would be nice to allow Lua to exist

as a peer to native code. In the end, we ended up with a system where native

code provides the foundation, but it is effectively a second-class citizen in the

application as a whole. Large portions of Lightroom ended up getting written

in Lua including the object-relational mapping layer for the database and the

layout system for views. Lua defines the structure of the application and its

extensibility mechanisms. As a result, we had an application that was smaller

by far than some of its competitors, easy to change, largely cross-platform in

its implementation, and suffered essentially no compile-link cycle. The reason

things work out this way is that Lua is both very expressive compared to most

native languages and sufficiently efficient that you can let it do a lot more of

the work than one might be tempted to in other scripting languages. At the

same time, the boundary between native code and Lua is sufficiently clean and

efficient that when we needed to do things in native code, it wasn’t a huge burden

to expose that functionality to Lua nor to access functionality written in Lua.

So, my advice to Lua users and potential users is to think seriously about

how widely you can let Lua spread through your work, be grateful for books like

this one and Programming in Lua and be even more grateful for the work that

the Lua team has done and their generosity in sharing it with the world.

Contributors

Following the Brazilian tradition, the contributors are listed in alphabetical

order of first name.

André Carregal was introduced to Lua in 1994 during his MSc in Computer

Science, which was supervised by Roberto Ierusalimschy. He has been working

with web development using Lua since 1996.

He currently coordinates the

Kepler project and the LuaForge site while working as a consultant for Lua-

related projects.

Ben Sunshine-Hill is a PhD student at the University of Pennsylvania,

studying computer graphics. He did his undergraduate studies at the University

of Southern California and received an MSc from the University of California,

Los Angeles. He has been a game developer at several mobile and mainstream

game development studios, and has previously published work on real-time

rendering methods.

Diego Nehab was introduced to Lua in 1996, while working for Tecgraf in

PUC-Rio. Over the years, he has been involved in a variety of Lua-related

projects, including the IupLua, CDLua, IMLua, and LuaSQL libraries. He is

best known as the author of the LuaThreads and LuaSocket libraries. Diego

received a BEng in Computer Engineering and an MSc in Programming Lan-

guages from PUC-Rio, under the supervision of Roberto Ierusalimschy. He later

received an MSc and a PhD in Computer Graphics from Princeton University.

His research now focuses on high-quality shape acquisition and on real-time

rendering techniques.

Doug Currie develops award-winning medical devices with Sunrise Labs,

Inc. in Auburn, New Hampshire, USA. Over a thirty-year career, Doug has

led electronics, mechanical, and software teams developing high-tech products

with particular emphasis on reliability and adaptability. Some of these products,

based on a massively parallel computing architecture Doug invented, are used

in national transportation and world-class manufacturing operations. With a

special interest in little languages, Doug has also contributed technically to open source projects such as Moscow ML, Hibernate, Gambit Scheme, and SICStus

Prolog. Doug holds an S.B. degree in Electrical Engineering and Computer

Science from the Massachusetts Institute of Technology.

Eduardo Ochs is a mathematician, or sort of; his interest on simplification

of proofs led him to Non-Standard Analysis, and from there he drifted to Logic,

Type Theory and Categorical Semantics. In parallel with his “normal” academic

life he has been a contributor to the GNU Project since 1999, and his main

focus areas in Free Software are little languages and programmable textual

interfaces. He keeps a big, messy homepage at http://angg.twu.net/; all the

html pages in it are generated with “BlogMe”, another extensible little language

built on top of Lua.

Gavin Wraith is Emeritus Reader in Mathematics, Sussex University, UK.

Joined Sussex University in 1963, retired in 1999. Founding chairman of the

Sussex University Computer Science department in 1985.

Jérôme Vuarand is a young software engineer specialized in AI and working

in the video games industry. He discovered Lua while looking for an embeddable

scripting language, just when Lua 5.1 was released, and he fell in love both

for the language itself and its new package system. His initial motivation was

to move away from legacy in-house script engines, but he’s now using it as a

general programming language in all his personal projects, from mobile robotics

to modern game engines entirely written in Lua.

John Belmonte is a software engineer currently residing in New York City.

He happened upon Lua as a video game developer in 2000 and was among the

first to embed it into a home console title. Since then he has been active in the

Lua community through chartering lua-users.org, participating in workshops,

and contributing to the language’s evolution.

Julio M. Fern ández-D´ıaz has a PhD degree in Mining Engineering (1989).

He is Professor of Applied Physics at the University of Oviedo in Spain and

researches mainly in the field of atmospheric aerosols.

His interests lie in

developing physical and mathematical simulations on computers.

His first

‘computer’ was an HP25 calculator in 1977. As a programmer, he uses Fortran,

C, Lua, Tcl/Tk and Postscript, usually as part of his research.

Konstantin Sokharev professionally develops video games since 2001, suc-

cessfully completed two RPG/RTS projects for PC, one for PocketPC/Palm. He

currently holds a post of technical director at IceHill llc. developing Action-RTS

title “Empire Above All” and several unannounced projects.

Luis Eduardo Ximenes Carvalho has a BSc (1997) in Civil Engineering

from the Federal University of Cear á (UFC), an MSc (2000) in Transportation

Engineering from the Federal University of Rio de Janeiro (UFRJ), and an

MSc (2002) in Computer Science from UFC, all in Brazil. He is currently a

PhD candidate in the Division of Applied Math at Brown University, where his

research comprises applications of Bayesian statistics to computational biology.

He also has interest in logistics and optimization, scientific computing, graph

theory, and programming languages, especially Lua.

Matthew Burke is an Assistant Professor of Computer Science at The George

Washington University in Washington, D.C.

Lua has replaced Forth as his

favorite language in which to program while riding the subway, and he does

so using whatever device is serving as his PDA du jour. He is also developing

a curriculum for introductory Computer Science which uses Lua. When not

programming, he likes to travel with his wife and son. He was the organizer of

the Lua Workshop 2008.

Nicolas Peri is co-founder and technical director of the French company Stone-

Trip, creator of the 3D game development platform ShiVa. He is in charge,

among other things, of the ShiVa scripting engine, which is based on Lua. Before

that, he worked as engine developer for other gaming companies, including

Kalisto Entertainment and UbiSoft Tiwak.

Patrick Rapin studied at the Swiss Federal Institute of Technology at Lau-

sanne (EPFL). He is now a software engineer working for Olivetti Engineer-

ing at Yverdon-les-Bains, developing printer firmware, image processing algo-

rithms, and printer test tools.

Pedro Miller Rabinovitch , a PUC-Rio graduate, has worked with Lua at

Tecgraf and Cipher Technology, and is currently a game developer at Jagex.

Rafael Moreira Savelli graduated in Computer Engineering at PUC-Rio.

He worked for Tecgraf in PUC-Rio for over four years. He is now studying for an

MSc at UFF and working in the Visgraf laboratory at IMPA.

Ralph Hempel is a Professional Engineer in Ontario, Canada and specializes

in designing embedded systems. After learning to program on an HP41C, he

never lost his fascination with small languages and hacking consumer products.

He wrote pbForth for the LEGO MINDSTORMS RCX and then ported Lua to

the NXT. When he’s not wrangling embedded systems, Ralph enjoys mountain

biking in the summer, snowboarding in the winter, and ice hockey all year long.

Ralph Steggink joined Océ in 2001. With a degree in both chemistry and com-

puter science, he now develops controller software for printers. Together with

Wim Couwenberg he prototyped revolutionary concepts using Lua. These cur-

rently find their way into several Océ products. He is an enthusiastic volleyball

player and trainer.

Reuben Thomas is a freelance singer and computer scientist living in Lon-don. He took a BA in Mathematics with Computer Science from Cambridge

University, as well as a doctorate in virtual machines. These days his comput-

ing interests center on contributions to a multitude of open source projects, with

particular emphasis on improving the quality of mature software, and on auto-

matic document processing. He is mostly employed as a classical baritone.

Robert Oates is a professional game programmer specializing in gameplay

systems, artificial intelligence, and machine learning.

Roberto de Beauclair Seixas works with Research and Development at

the Institute of Pure and Applied Mathematics (IMPA) in Rio de Janeiro, as

member of the Vision and Computer Graphics Laboratory (Visgraf). He got

his PhD in Computer Science at PUC-Rio, where he works with the Computer

Graphics Technology Group (Tecgraf). From 1982 to 1998, he worked in the

Computer Science Department at the National Laboratory for Scientific Com-

putation (LNCC). His research interests include Scientific Visualization, Vol-

ume Rendering, Computer Graphics, High Performance Computing, Geometric

Modeling, Military Warfare Simulations, GIS, and Medical Images.

Roberto Ierusalimschy is an Associate Professor at the Catholic University

in Rio de Janeiro. He is the leading architect of Lua and the author of the book

“Programming in Lua”.

Sérgio Alvares Maffra is a MSc and Computer Engineer from PUC-Rio. He’s

been working with Lua at Tecgraf as a software developer for over a decade now.

Steve Gargolinski spent his early programming days hacking together small

games built with code snippets from a QuickBasic programming manual. He

has since evolved into a professional game developer, working as a member of

the technical teams that produced the Zoo Tycoon 2 series, Star Trek: Legacy,

and the upcoming Empire Earth III. Steve is currently working for Blue Fang

Games as an AI Programmer. His interests include baseball, abstract strategy,

practical AI, and walking in the woods.

Tobias S ¨

ulzenbr ¨

uck and Christoph Beckmann are bachelor students of

media systems at the Bauhaus-University of Weimar. Tobias fields of interests

range from web development up to graphics programming. He has implemented

a multi-agent system for simulating construction processes in Lua. Christoph

is also interested in web development and is active in the research field of

computer-supported cooperative work.

Tom ás Guisasola works with Lua since 1995 when he developed with Roberto

Ierusalimschy (his MSc advisor) the first implementation of the hooks mecha-

nism and the debug facilities. Since then he worked mainly with CGILua as

the platform for some administrative systems at PUC-Rio and also contributed

with the Kepler team in the development of LuaLDAP, LuaXMLRPC, LuaSOAP,

LuaDoc, and LuaSQL.

Vadim Groznov began programming at the age of fourteen, was involved

in database programming for a long time, and took part in the creation of a

custom scripting language. He professionally develops video games since 2002.

His extensive experience of system programming allowed for the design and

realisation of complex architectural solutions for game tools at IceHill llc.

Wim Couwenberg holds a PhD in mathematics and is employed at the R&D

department of the European printing and document company Océ, based in The

Netherlands, where he organised the international Lua Workshop 2006. He has

been using Lua in projects ranging from simple data processing scripts to entire

networked applications.

Yuri Takhteyev is a doctoral student at the UC Berkeley School of Informa-

tion studying the role of space in software development communities.

Han Zhao is a shareware programmer in Beijing, P.R. China. Before that he

worked for a mobile-phone design house. Now he uses Lua and C++ for everyday

programming: an isometric role-playing game engine, an action game, and a

shareware product. He also maintains a bit-operation lib LuaBit on LuaForge.

Part I

Programming Techniques

1

Lua Per-Thread Library Context

Doug Currie

Libraries written in C for use with Lua sometimes have a context that can be

modified by the Lua program. For example, in the decNumber library, the Lua

program may select the rounding mode and precision for arithmetic operations.

The decNumber library user expects the context to be applied during library

operations, and remain fixed until explicitly changed.

There are many other examples of library context. Libraries may need to

maintain a per-thread global variable, like the POSIX library’s errno. The

C standard libraries have a current input file and current output file that are

implied for many operations.

It would be wrong for a context setting in one Lua thread to affect the setting

in another Lua thread. The other thread would get an unexpected rounding

error, or an unexpected errno value, for example. Each thread should have

its own context so that the library functions it uses operate the same way

independent of the activities of other threads.

Lua does not provide a per-thread variables mechanism directly, though

there are many ways to create this affect. The solution presented in this gem is

to use the mechanism provided by LUA_ENVIRONINDEX. All functions in the library

share a common closure. In this closure is a table used to map the thread’s

identity, i.e., L to a context. Since only the functions in the library have access to

the common closure, there is no chance of interference from other libraries. The

mechanism is fast, and can be made even faster with caching.

This gem presents the solution in a straightforward implementation, and

adds userdata context functions, caching, and performance measurement in

incremental steps.

4

1 · Lua Per-Thread Library Context

There are other design alternatives to a per-thread context. The C library

API can be revised to eliminate the global context. One way is by forcing the

caller to provide a context argument with every API call. Alternatively, the

API may require a context argument with data constructor calls, storing the

context somewhere in the newly allocated data. Subsequent API calls using this

data reference the context through the data. There are trade-offs with these

alternatives.

Requiring a context with every API call is regular and applicative (sometimes

called “functional”) — the functions in the API can be idempotent. These are nice

features, but while they make reasoning about your program easier in theory,

they give up much in convenience. This method also simply pushes the problem

of where to keep the per-thread context from the library into the library’s client,

the application.

Saving a context with every data structure may make reasoning about your

program even harder, and makes sharing data among threads problematic.

So, in addition to having dubious benefit, it shares many of the drawbacks of

requiring a context with every API call.

One key benefit of a per-thread context is that Lua operators may use the

context when operating on data managed by the library.

For example, the decNumber library uses userdata to implement arbitrary

precision decimal numbers, and uses a metatable to support using Lua oper-

ators with these decimal numbers. The decNumber context has several user

settable arithmetic parameters such as the number of significant digits, and the

rounding mode. So, with a per-thread decNumber library context, the expres-

sion for computing compound interest with decimal numbers is simply:

((rate/100+1)^years)*start

but, if every decNumber function had to have the context supplied, it might look

like this:

decNumber.multiply(CONTEXT,

decNumber.expt(CONTEXT,

decNumber.add(CONTEXT, 1,

decNumber.divide(CONTEXT, rate, 100)),

years),

start)

I suspect you’d prefer the Lua operators to the API with context.

Implementation alternatives

So, how do we implement a per-thread library context?

The Lua 5.1 Reference Manual describes a “thread environment” that is ac-

cessible using LUA_GLOBALSINDEX. But all threads share the same global table as

their thread environment by default. This prevents the thread environment’s

5

use as a per-thread storage mechanism for a library. It would depend on mech-

anisms to create a new thread environment at thread creation time, and these

mechanisms are outside the library’s control.

Libraries compiled and built together with the Lua sources can use the

LUAI_EXTRASPACE mechanism to allocate some per-thread storage. To use this

trick you must recompile the whole Lua library with a proper definition for

LUAI_EXTRASPACE. If two or more libraries use this trick, their use of

LUAI_EXTRASPACE must be coordinated. Clearly this approach doesn’t work for

dynamically loaded libraries and stock Lua.

What’s needed is dynamically allocated per-thread storage.

With help from Lua users on the Lua mailing list, several implementation

alternatives were explored. The first one was using the Lua registry to connect

each thread to a userdata representing the context. The thread itself could

be used as the registry key. The drawback to this approach is that it is not

scalable. If two libraries use this mechanism then they will overwrite each

others’ contexts in the registry.

A fix to the registry approach is to use another level of indirection. Each

library could create a table in the registry, keyed using a light userdata in

the library’s address space, and then this table would be used to associate the

threads to the contexts. While this approach would work, the extra level of

indirection and the double table lookup were unsatisfying.

Using LUA ENVIRONINDEX for library private storage

Lua 5.1 also provides a little documented mechanism using a pseudo-index

called LUA_ENVIRONINDEX. The Lua 5.1 Reference Manual (§3.3) simply says

that “the environment of the running C function is always at pseudo-index

LUA_ENVIRONINDEX”.

What is the environment of a running C function?

Digging into the Lua 5.1 source code, we find uses of LUA_ENVIRONINDEX in

liolib.c and loadlib.c where the function environment mechanism is used

create a table shared privately among all the functions of a library. How does

this work?

When a C function closure is created with lua_pushcclosure, which is used

by luaL_register when registering a library’s C functions for example, the

current C function environment is saved in the new C function’s closure as

its C environment. The function lua_replace may be used to set the current

C function environment to a newly allocated table. This table will then be shared

by all subsequently created C function closures.

A function’s environment may also be set using lua_setfenv — but this is

not nearly as useful for our purposes! It would mean iterating over all the

functions in the library after they were created by luaL_register duplicating

a lot of effort.

So, for example, registering your library’s functions could be done with:

6

1 · Lua Per-Thread Library Context

/* make library private storage in new c function environment */

lua_newtable (L);

lua_replace (L, LUA_ENVIRONINDEX);

/* create library global table and register its functions */

luaL_register (L, LIBRARY_NAME, lib_luaL_reg_array);

The library private storage is now available to all the functions in

lib_luaL_reg_array by using pseudo-index LUA_ENVIRONINDEX with any Lua API

function that takes an index argument. So, for example, accessing the library

private storage from functions in lib_luaL_reg_array can be accomplished with:

lua_pushliteral (L, "somevalue");

/* value to store */

lua_setfield (L, LUA_ENVIRONINDEX, "somekey");

/* key */

to store a value in library private storage, and

lua_getfield (L, LUA_ENVIRONINDEX, "somekey");

to retrieve it.

Using library private storage to implement per-thread

private storage

Now that we have library private storage, how do we use it to implement per-

thread library private storage? The simple answer is to use the current thread

as a key for the library private storage table. Then each Lua thread in the

system will have one value in our library private storage.

Our example accessing the library private storage from functions in

lib_luaL_reg_array becomes:

lua_pushthread (L);

/* key */

lua_pushvalue (L, index);

/* value */

lua_rawset (L, LUA_ENVIRONINDEX);

to save the thread’s value, and

lua_pushthread (L);

/* key */

lua_rawget (L, LUA_ENVIRONINDEX);

to retrieve the thread’s value.

If multiple values must be stored for each thread, we can make the thread’s

value a table or userdata and store several values in it.

There is one further thing we must do to make this mechanism a good Lua

citizen. By using threads as our library private storage table keys, we are

preventing Lua from ever garbage collecting the threads that use our library.

The solution is making our library private storage table a weak table on its

keys. Registering our library’s functions becomes:

7

lua_newtable (L);

/* make environment "private storage" */

lua_createtable (L, 0, 1);

/* its metatable, which is */

lua_pushliteral (L, "__mode");

/* used to make environment */

lua_pushliteral (L, "k");

/* weak in the keys */

lua_rawset (L, -3);

/* metatable.__mode = "k" */

lua_setmetatable (L, -2);

/* set the environment metatable */

lua_replace (L, LUA_ENVIRONINDEX);

/* the new c function env */

/* create library global table and register its functions */

luaL_register (L, LIBRARY_NAME, lib_luaL_reg_array);

One may wonder if using lua_pushlightuserdata(L,L) might be faster than

lua_pushthread(L) as the key for accessing our private storage. There are two

problems with using lua_pushlightuserdata(L,L):

• Lua offers no guarantee that the thread will not be relocated by the garbage

collector; if it is relocated, the value L will change from one library invoca-

tion to another, and our key is useless to identify the thread private data;

• Our private storage table would never be garbage collected since Lua

cannot determine if our light userdata is garbage, so the table entries will

remain in the table until we explicitly remove them.

Fortunately, lua_pushthread(L) is quite fast, convenient, and avoids both prob-

lems.

Using per-thread private storage for per-thread library

context

Now that we have per-thread library-private storage we are ready to implement

per-thread library context. For this implementation we will use userdata for

the per-thread context. This supports adding as many fields of context for each

thread as necessary while having only one value in the thread private storage

for each thread.

For Lua decNumber, a wrapper for the IBM decNumber library, the context is

simply the decContext structure defined by the underlying decNumber library.

The examples in subsequent sections will use Lua decNumber code. To use these

techniques in your own code, simply substitute your own context structure for

the decContext structure.

The mechanism has just a few pieces:

• The weak key table implementation of per-thread library private storage,

above;

• Functions to create and type-check a context userdata;

• Functions to get and set the current thread’s context;

• A function to push current thread’s context to return it to Lua code.

8

1 · Lua Per-Thread Library Context

The functions to create and type-check a context userdata will be familiar

to Lua library authors. The creation function simply allocates storage, and sets

the userdata metatable to a unique metatable that is used for runtime type

checking, along with providing the table of callable methods for the context. The

metatable is created at library load time, and I won’t bore you with the details

since this is a standard Lua library technique — the details are available in the

decNumber library, for example, at LuaForge.

The functions to get and set the current thread’s context are the heart of

the matter. The getter function uses a helper function to push current thread’s

context as a helper. The reason for this will become clearer in the next section

on caching.

/*

ldn_set_context has no stack effect

*/

static void ldn_set_context (lua_State *L, int index)

{

/* make value at index the context for this thread */

if (index < 0) index -= 1;

lua_pushthread (L);

/* key */

lua_pushvalue (L, index); /* value */

lua_rawset (L, LUA_ENVIRONINDEX);

}

static decContext *ldn_push_context (lua_State *L)

{

decContext *dc;

lua_pushthread (L);

/* key */

lua_rawget (L, LUA_ENVIRONINDEX);

if (lua_isnil (L, -1))

{

/* nothing in the thread local state, so make a new context */

lua_pop (L, 1);

dc = ldn_make_context (L);

dc = decContextDefault (dc, LDN_CONTEXT_DEFAULT); /* init */

/* make it the context for this thread */

ldn_set_context (L, -1, dc);

}

else

{

dc = ldn_check_context (L, -1);

}

return dc; /* and leaves context on Lua stack */

}

9

static decContext *ldn_get_context (lua_State *L)

{

decContext *dc;

dc = ldn_push_context (L);

lua_pop (L, 1);

return dc;

}

Caching per-thread library context

While the implementation of per-thread library context using library private

storage is quite fast, there is room for improvement. We can avoid the table

lookup in the library private storage if we cache the last lookup. This will be

beneficial in a couple common cases:

• The library is only used from only one thread;

• The library is used several times between thread yields.

First we will develop a caching approach, and then explore its benefits,

measuring its performance advantage, and discuss some potential drawbacks.

The caching technique is quite straightforward: we simply record the thread

(L) and context on each lookup in the library private storage. Each reference to

the thread context first checks if the reference is from the last thread to perform

a lookup, and if so returns the cached value avoiding the second and subsequent

lookups.

This implementation depends on threads and userdata not being moved by

the garbage collector. Fortunately, this is true for present Lua implementations,

and those in the foreseeable future since the Lua authors “have no intention

of allowing userdata addresses to change during GC” (http://lua-users.org/

lists/lua-l/2006-04/msg00384.html).

Lua doesn’t provide a way to push a userdata from the C library; the pointer

returned by lua_touserdata or luaL_checkudata is incremented past the Lua

userdata header. The userdata header structure is opaque to the C library. Most

of the time the library doesn’t care about this; it just wants a pointer to the

context and doesn’t need to put it on the Lua stack. In these cases the cache

works fine. In other rare cases, the C library must push the context to pass

it as an argument to Lua code or return it to the Lua library caller. In these

situations the cache is simply bypassed.

Another optimization is that ldn_set_context now takes the decContext as

an argument. This avoids having to convert the context value on the stack to a

userdata pointer for storage into the cache. The caller generally has this pointer

in hand at the time of the call.

Our C code is uses preprocessor macros to enable and disable caching so we

can build with either implementation. Here are the updated functions:

10

1 · Lua Per-Thread Library Context

#if LDN_ENABLE_CACHE

static lua_State *L_of_context_cache;

static decContext *context_cache;

#endif

/* the value on stack at index must be

the decContext userdata corresponding to dc

ldn_set_context must have no stack effect

*/

static void ldn_set_context (lua_State *L, int index, decContext *dc)

{

/* make value at index the context for this thread */

if (index < 0) index -= 1;

lua_pushthread (L);

/* key */

lua_pushvalue (L, index); /* value */

lua_rawset (L, LUA_ENVIRONINDEX);

#if LDN_ENABLE_CACHE

/* and cache */

L_of_context_cache = L;

context_cache = dc;

#endif

}

/*

* either we need a decContext on the Lua stack, so we must bypass the

* cache, or we have a cache miss

*/

static decContext *ldn_push_context (lua_State *L)

{

decContext *dc;

lua_pushthread (L);

/* key */

lua_rawget (L, LUA_ENVIRONINDEX);

if (lua_isnil (L, -1))

{

/* nothing in the thread local state, so make a new context */

lua_pop (L, 1);

dc = ldn_make_context (L);

dc = decContextDefault (dc, LDN_CONTEXT_DEFAULT);

/* make it the context for this thread */

ldn_set_context (L, -1, dc);

}

11

else

{

dc = ldn_check_context (L, -1);

#if LDN_ENABLE_CACHE

/* and cache */

L_of_context_cache = L;

context_cache = dc;

#endif

}

return dc; /* leaves context on Lua stack */

}

static decContext *ldn_get_context (lua_State *L)

{

decContext *dc;

#if LDN_ENABLE_CACHE

/* try the cache first */

if (L_of_context_cache == L)

{

dc = context_cache;

}

else

#endif

{

/* go to the per-thread storage next */

dc = ldn_push_context (L);

lua_pop (L, 1);

}

return dc;

}

Note: There is an unlikely but possible failure mode for this cache implementa-

tion. The following things must all happen:

• A thread X uses the per-thread library context;

• It is the last thread to reference the per-thread library context before a

garbage collection;

• The thread X is garbage collected;

• A new thread Y is created and is allocated at the exact same memory

address as X;

• The new thread X is the first to reference the per-thread library context

after the garbage collection.

At this point, thread Y will use the (possibly freed) userdata context of thread X

leading to, at worst, a crash. Even if the userdata context has not been freed

12

1 · Lua Per-Thread Library Context

yet, it is not the context of the new thread, which presumably should be a newly

initialized context. So, for example, thread Y could use a rounding mode and

precision set by thread X rather than the defaults, or could use file handles

thread X established for I/O rather than the standard I/O handles.

The root cause of the problem is that we are holding a reference to the thread

and userdata context that is not reachable from Lua’s root set. Of course, if it

was reachable, we’d have the same memory leak that we fixed using a weak

table for the per-thread library context.

Can this failure mode be eliminated?

Adding a __gc method to the context userdata that invalidates the per-thread

library context cache is probably a good place to start; that will prevent access

to freed memory. This is not a general solution to the problem, though, since the

context may not have been collected yet. The context may not be garbage if it can

be shared by multiple threads (in decNumber it can be shared). Furthermore,

even if the context is not shared, it is not guaranteed to be collected on the

same collection cycle as the thread, so there is still a potential problem with

using the wrong context, that of the freed thread (thread X’s context rather than

thread Y’s).

Unfortunately, Lua does not have a gc hook that is called after every collec-

tion; otherwise a hook function could simply invalidate the per-thread library

context cache after every collection. One can emulate this hook by allocating a

sacrificial userdata with a __gc method that invalidates the cache, and imme-

diately popping it from the stack. It’s __gc method would also create another

identical userdata so that the cache is flushed every collection cycle.

This is a bit tricky, and dependent on non-specified behavior of the garbage

collector. The Lua garbage collector offers no guarantee that either (a) garbage

is collected in any particular order relative to becoming untraceable, or (b) all

garbage is collected on every cycle. There are garbage collectors that collect in

some arbitrary order, e.g., memory address order, and/or only a portion of the

free memory on each collection cycle. The Caml Light GC works that way, for

example. So, if the thread is collected but not the sacrificial userdata on the

same cycle, the trick (now a bad kludge) doesn’t work.

However the trick works with the present Lua 5.1 garbage collector, and

probably with most future implementations. As the sacrificial userdata is al-

ready dead when the cycle starts, it will surely be collected. Even if the Lua im-

plementers introduce generational garbage collection, this userdata will never

move to older generations.

Another solution to eliminating the failure mode is to require that each new

thread calls an initialization function in the library that, perhaps among other

things, invalidates the cache. Since this depends on actions of library users to

prevent the failure mode, it is not ideal. However, if your library needs per-

thread initialization for other reasons, it may be a reasonable fix.

In summary, to avoid the highly unlikely caching failure mode, you should

use a __gc method of the context userdata that invalidates the per-thread li-

brary context, and either require library users in each new thread to call an

13

initialization function that invalidates the cache, or create a sacrificial userdata

that invalidates the cache and regenerates a garbage copy of itself. In the latter

case, if a future Lua has an incompatible garbage collector, look for its new post

collection hook instead!

Of course, the failure mode does not exist if the non-cached implementation

of per-thread library context is used. Whether it is worth the effort to implement

a totally safe cache depends on the expected performance benefit.

That is

examined in the next section.

Testing and performance measurement

The Lua decNumber library is delivered with extensive test suite, both the IBM

decTest suite as well as Lua implementation specific tests. There are over 30,000

test cases. Among these tests are some performance measurements of Lua

decNumber with and without caching of the per-thread library context. There

are specific tests of correctness of the per-thread context using two Lua threads,

and implementation of cache hit and miss counters.

The performance measurement is done with an included Lua library pro-

viding access to the Windows high resolution QueryPerformanceCounter API.

Please experiment with these tools on your system.

Two test cases were studied to determine if per-thread library context caching

is beneficial: a minimal arithmetic loop (one decNumber add and assignment to

a local), and a more complex calculation involving decNumber division, remain-

der, square root, and exponentiation. Both cases were timed in single threaded

mode — this is the best case for caching since all but the first context access will

hit.

The testing method was simple: Time N iterations of an empty loop and

N iterations of each calculation. Repeat this several times and discard the obvi-

ous outliers (best and worst). Subtract the empty loop time from the calculation

loop times. Compare the cache and non-cache results:

• For the simple arithmetic loop, caching gave a bit over a 21% reduction in

compute time. In other words, the arithmetic loop run with the context-

cache-enabled library ran in 79% of the time of the no-context-cache li-

brary.

• For the complex calculation loop, caching gave a bit over a 0.6% reduction

in compute time. Lua table lookup is quite fast compared with this calcu-

lation!

So, as you’d expect, the benefits of caching the library context will depend a

lot on the time complexity of your library functions.

14

1 · Lua Per-Thread Library Context

Conclusion

Caching the per-thread library context provides a small increase in performance.

The caching failure mode identified above is highly unlikely, but catastrophic.

The solutions to avoid the failure are either Lua implementation dependent, or

put a per-thread initialization obligation on library users. Fortunately, the non-

cache version of per-thread library context performs quite well. Unless you need

the small performance gain, or until Lua implements a gc hook, the cache may

be more trouble than it’s worth. If you need the performance, use a solution to

avoid the failure that’s best for your application.

Lua’s LUA_ENVIRONINDEX mechanism has several interesting uses. It was a

joy to discover, and seems to be just the right approach for per-thread library

context. The mechanism is so easy to implement that every library with context

should use per-thread library context.

2

Lua Performance Tips

Roberto Ierusalimschy

In Lua, as in any other programming language, we should always follow the two

maxims of program optimization:

Rule #1: Don’t do it.

Rule #2: Don’t do it yet. (for experts only)

Those rules are particularly relevant when programming in Lua. Lua is famous

for its performance, and it deserves its reputation among scripting languages.

Nevertheless, we all know that performance is a key ingredient of program-

ming. It is not by chance that problems with exponential time complexity are

called intractable. A too late result is a useless result. So, every good program-

mer should always balance the costs from spending resources to optimize a piece

of code against the gains of saving resources when running that code.

The first question regarding optimization a good programmer always asks is:

“Does the program needs to be optimized?” If the answer is positive (but only

then), the second question should be: “Where?”

To answer both questions we need some instrumentation. We should not

try to optimize software without proper measurements. The difference between

experienced programmers and novices is not that experienced programmers are

better at spotting where a program may be wasting its time: The difference is

that experienced programmers know they are not good at that task.

A few years ago, Noemi Rodriguez and I developed a prototype for a CORBA

ORB (Object Request Broker) in Lua, which later evolved into OiL ( Orb in

Lua). As a first prototype, the implementation aimed at simplicity. To avoid

16

2 · Lua Performance Tips

the need for extra C libraries, the prototype serialized integers using a few

arithmetic operations to isolate each byte (conversion to base 256). It did not

support floating-point values. Because CORBA handles strings as sequences of

characters, our ORB first converted Lua strings into a sequence (that is, a Lua

table) of characters and then handled the result like any other sequence.

When we finished that first prototype, we compared its performance against

a professional ORB implemented in C++. We expected our ORB to be somewhat

slower, as it was implemented in Lua, but we were disappointed by how much

slower it was. At first, we just laid the blame on Lua. Later, we suspected

that the culprit could be all those operations needed to serialize each number.

So, we decided to run the program under a profiler. We used a very simple

profiler, not unlike the one described in Chapter 23 of Programming in Lua.

The profiler results shocked us. Against our gut feelings, the serialization of

numbers had no measurable impact on the performance, among other reasons

because there were not that many numbers to serialize. The serialization of

strings, however, was responsible for a huge part of the total time. Practically

every CORBA message has several strings, even when we are not manipulating

strings explicitly: object references, method names, and some other internal

values are all coded as strings. And the serialization of each string was an

expensive operation, because it needed to create a new table, fill it with each

individual character, and then serialize the resulting sequence, which involved

serializing each character one by one. Once we reimplemented the serialization

of strings as a special case (instead of using the generic code for sequences), we

got a respectable speed up. With just a few extra lines of code, the performance

of your implementation was comparable to the C++ implementation.1

So, we should always measure when optimizing a program for performance.

Measure before, to know where to optimize. And measure after, to know whether

the “optimization” actually improved our code.

Once you decide that you really must optimize your Lua code, this text may

help you about how to optimize it, mainly by showing what is slow and what is

fast in Lua. I will not discuss here general techniques for optimization, such

as better algorithms. Of course you should know and use those techniques,

but there are several other places where you can learn them. In this article

I will discuss only techniques that are particular to Lua. Along the article, I will

constantly measure the time and space performance of small programs. Unless

stated otherwise, I do all measures on a Pentium IV 2.9 GHz with 1 GB of main

memory, running Ubuntu 7.10, Lua 5.1.1. Frequently I give actual measures

(e.g., 7 seconds), but what is relevant is the relationship between different

measures. When I say that a program is “X% times faster” than another it

means that it runs in X% less time. (A program 100% faster would take no time

to run.) When I say that a program is “X% times slower” than another I mean

that the other is X% faster. (A program 50% slower means that it takes twice

the time.)

1Of course our implementation was still slower, but not by an order of magnitude.

17

Basic facts

Before running any code, Lua translates (precompiles) the source into an in-

ternal format. This format is a sequence of instructions for a virtual machine,

similar to machine code for a real CPU. This internal format is then interpreted

by C code that is essentially a while loop with a large switch inside, one case for each instruction.

Perhaps you have already read somewhere that, since version 5.0, Lua uses

a register-based virtual machine. The “registers” of this virtual machine do not

correspond to real registers in the CPU, because this correspondence would be

not portable and quite limited in the number of registers available. Instead,

Lua uses a stack (implemented as an array plus some indices) to accommodate

its registers. Each active function has an activation record, which is a stack

slice wherein the function stores its registers. So, each function has its own

registers2. Each function may use up to 250 registers, because each instruction

has only 8 bits to refer to a register.

Given that large number of registers, the Lua precompiler is able to store all

local variables in registers. The result is that access to local variables is very

fast in Lua. For instance, if a and b are local variables, a Lua statement like

a = a + b generates one single instruction: ADD 0 0 1 (assuming that a and b

are in registers 0 and 1, respectively). For comparison, if both a and b were

globals, the code for that addition would be like this:

GETGLOBAL

0 0

; a

GETGLOBAL

1 1

; b

ADD

0 0 1

SETGLOBAL

0 0

; a

So, it is easy to justify one of the most important rules to improve the perfor-

mance of Lua programs: use locals!

If you need to squeeze performance out of your program, there are several

places where you can use locals besides the obvious ones. For instance, if you

call a function within a long loop, you can assign the function to a local variable.

For instance, the code

for i = 1, 1000000 do

local x = math.sin(i)

end

runs 30% slower than this one:

local sin = math.sin

for i = 1, 1000000 do

local x = sin(i)

end

2This is similar to the register windows found in some CPUs.

18

2 · Lua Performance Tips

Access to external locals (that is, variables that are local to an enclosing

function) is not as fast as access to local variables, but it is still faster than

access to globals. Consider the next fragment:

function foo (x)

for i = 1, 1000000 do

x = x + math.sin(i)

end

return x

end

print(foo(10))

We can optimize it by declaring sin once, outside function foo:

local sin = math.sin

function foo (x)

for i = 1, 1000000 do

x = x + sin(i)

end

return x

end

print(foo(10))

This second code runs 30% faster than the original one.

Although the Lua compiler is quite efficient when compared with compilers

for other languages, compilation is a heavy task. So, you should avoid compiling

code in your program (e.g., function loadstring) whenever possible. Unless you

must run code that is really dynamic, such as code entered by an end user, you

seldom need to compile dynamic code.

As an example, consider the next code, which creates a table with functions

to return constant values from 1 to 100000:

local lim = 10000

local a = {}

for i = 1, lim do

a[i] = loadstring(string.format("return %d", i))

end

print(a[10]())

--> 10

This code runs in 1.4 seconds.

With closures, we avoid the dynamic compilation. The next code creates the

same 100000 functions in 1 of the time (0.14 seconds):

10

function fk (k)

return function () return k end

end

19

local lim = 100000

local a = {}

for i = 1, lim do

a[i] = fk(i)

end

print(a[10]())

--> 10

About tables

Usually, you do not need to know anything about how Lua implement tables to

use them. Actually, Lua goes to great lengths to make sure that implementation

details do not surface to the user. However, these details show themselves

through the performance of table operations. So, to optimize programs that use

tables (that is, practically any Lua program), it is good to know a little about

how Lua implements tables.

The implementation of tables in Lua involves some clever algorithms. Every

table in Lua has two parts: the array part and the hash part. The array part stores entries with integer keys in the range 1 to n, for some particular n. (We

will discuss how this n is computed in a moment.) All other entries (including

integer keys outside that range) go to the hash part.

As the name implies, the hash part uses a hash algorithm to store and find its

keys. It uses what is called an open address table, which means that all entries

are stored in the hash array itself. A hash function gives the primary index of a

key; if there is a collision (that is, if two keys are hashed to the same position),

the keys are linked in a list, with each element occupying one array entry.

When Lua needs to insert a new key into a table and the hash array is full,

Lua does a rehash. The first step in the rehash is to decide the sizes of the new

array part and the new hash part. So, Lua traverses all entries, counting and

classifying them, and then chooses as the size of the array part the largest power

of 2 such that more than half the elements of the array part are filled. The hash

size is then the smallest power of 2 that can accommodate all the remaining

entries (that is, those that did not fit into the array part).

When Lua creates an empty table, both parts have size 0 and, therefore,

there are no arrays allocated for them. Let us see what happens when we run

the following code:

local a = {}

for i = 1, 3 do

a[i] = true

end

It starts by creating an empty table a. In the first loop iteration, the assignment

a[1]=true triggers a rehash; Lua then sets the size of the array part of the table

to 1 and keeps the hash part empty. In the second loop iteration, the assignment

a[2]=true triggers another rehash, so that now the array part of the table has

size 2. Finally, the third iteration triggers yet another rehash, growing the size

of the array part to 4.

20

2 · Lua Performance Tips

A code like

a = {}

a.x = 1; a.y = 2; a.z = 3

does something similar, except that it grows the hash part of the table.

For large tables, this initial overhead is amortized over the entire creation:

While a table with three elements needs three rehashings, a table with one

million elements needs only twenty. But when you create thousands of small

tables, the combined overhead can be significant.

Older versions of Lua created empty tables with some pre-allocated slots

(four, if I remember correctly), to avoid this overhead when initializing small

tables. However, this approach wastes memory. For instance, if you create

millions of points (represented as tables with only two entries) and each one

uses twice the memory it really needs, you may pay a high price. That is why

currently Lua creates empty tables with no pre-allocated slots.

If you are programming in C, you can avoid those rehashings with the Lua

API function lua_createtable. It receives two arguments after the omnipresent

lua_State: the initial size of the array part and the initial size of the hash part

of the new table.3 By giving appropriate sizes to the new table, it is easy to avoid

those initial rehashes. Beware, however, that Lua can only shrink a table when

rehashing it. So, if your initial sizes are larger than needed, Lua may never

correct your waste of space.

When programming in Lua, you may use constructors to avoid those initial

rehashings. When you write {true, true, true}, Lua knows beforehand that

the table will need three slots in its array part, so Lua creates the table with

that size. Similarly, if you write {x = 1, y = 2, z = 3}, Lua will create a table

with four slots in its hash part. As an example, the next loop runs in 2.0 seconds:

for i = 1, 1000000 do

local a = {}

a[1] = 1; a[2] = 2; a[3] = 3

end

If we create the tables with the right size, we reduce the run time to 0.7 seconds:

for i = 1, 1000000 do

local a = {true, true, true}

a[1] = 1; a[2] = 2; a[3] = 3

end

If you write something like {[1] = true, [2] = true, [3] = true}, how-

ever, Lua is not smart enough to detect that the given expressions (literal num-

bers, in this case) describe array indices, so it creates a table with four slots in

its hash part, wasting memory and CPU time.

3Although the rehash algorithm always sets the array size to a power of two, the array size can be any value. The hash size, however, must be a power of two, so the second argument is always rounded to the smaller power of two not smaller than the original value.

21

The size of both parts of a table are recomputed only when the table rehashes,

which happens only when the table is completely full and Lua needs to insert a

new element. As a consequence, if you traverse a table erasing all its fields (that

is, setting them all to nil), the table does not shrink. However, if you insert some

new elements, then eventually the table will have to resize. Usually this is not

a problem: if you keep erasing elements and inserting new ones (as is typical in

many programs), the table size remains stable. However, you should not expect

to recover memory by erasing the fields of a large table: It is better to free the

table itself.

A dirty trick to force a rehash is to insert enough nil elements into the table.

See the next example:

a = {}

lim = 10000000

for i = 1, lim do a[i] = i end

--

create a huge table

print(collectgarbage("count"))

--> 196626

for i = 1, lim do a[i] = nil end

--

erase all its elements

print(collectgarbage("count"))

--> 196626

for i = lim + 1, 2*lim do a[i] = nil end

--

create many nil elements

print(collectgarbage("count"))

--> 17

I do not recommend this trick except in exceptional circumstances: It is slow

and there is no easy way to know how many elements are “enough”.

You may wonder why Lua does not shrink tables when we insert nils. First,

to avoid testing what we are inserting into a table; a check for nil assignments

would slow down all assignments. Second, and more important, to allow nil

assignments when traversing a table. Consider the next loop:

for k, v in pairs(t) do

if some_property(v) then

t[k] = nil

-- erase that element

end

end

If Lua rehashed the table after a nil assignment, it would havoc the traversal.

If you want to erase all elements from a table, a simple traversal is the correct

way to do it:

for k in pairs(t) do

t[k] = nil

end

A “smart” alternative would be this loop:

while true do

local k = next(t)

if not k then break end

t[k] = nil

end

22

2 · Lua Performance Tips

However, this loop is very slow for large tables. Function next, when called

without a previous key, returns the “first” element of a table (in some random

order). To do that, next traverses the table arrays from the beginning, looking

for a non-nil element. As the loop sets the first elements to nil, next takes longer

and longer to find the first non-nil element. As a result, the “smart” loop takes

20 seconds to erase a table with 100,000 elements; the traversal loop using pairs

takes 0.04 seconds.

About strings

As with tables, it is good to know how Lua implements strings to use them more

efficiently.

The way Lua implements strings differs in two important ways from what is

done in most other scripting languages. First, all strings in Lua are internalized; this means that Lua keeps a single copy of any string. Whenever a new string

appears, Lua checks whether it already has a copy of that string and, if so,

reuses that copy. Internalization makes operations like string comparison and

table indexing very fast, but it slows down string creation.

Second, variables in Lua never hold strings, but only references to them.

This implementation speeds up several string manipulations. For instance, in

Perl, when you write something like $x = $y, where $y contains a string, the

assignment copies the string contents from the $y buffer into the $x buffer. If

the string is long, this becomes an expensive operation. In Lua, this assignment

involves only copying a pointer to the actual string.

This implementation with references, however, slows down a particular form

of string concatenation. In Perl, the operations $s = $s . "x" and $s .= "x"

are quite different. In the first one, you get a copy of $s and adds "x" to its end.

In the second one, the "x" is simply appended to the internal buffer kept by the $s variable. So, the second form is independent from the string size (assuming

the buffer has space for the extra text). If you have these commands inside loops,

their difference is the difference between a linear and a quadratic algorithm. For

instance, the next loop takes almost five minutes to read a 5MByte file:

$x = "";

while (<>) {

$x = $x . $_;

}

If we change $x = $x . $_ to $x .= $_, this time goes down to 0.1 seconds!

Lua does not offer the second, faster option, because its variables do not

have buffers associated to them. So, we must use an explicit buffer: a table with

the string pieces does the job. The next loop reads that same 5MByte file in

0.28 seconds. Not as fast as Perl, but quite good.

23

local t = {}

for line in io.lines() do

t[#t + 1] = line

end

s = table.concat(t, "\n")

Reduce, reuse, recycle

When dealing with Lua resources, we should apply the same three R’s promoted

for the Earth’s resources.

Reduce is the simplest alternative. There are several ways to avoid the need

for new objects. For instance, if your program uses too many tables, you may

consider a change in its data representation. As a simple example, consider

that your program manipulates polylines. The most natural representation for

a polyline in Lua is as a list of points, like this:

polyline = { { x = 10.3, y = 98.5 },

{ x = 10.3, y = 18.3 },

{ x = 15.0, y = 98.5 },

...

}

Although natural, this representation is not very economic for large polylines, as

it needs a table for each single point. A first alternative is to change the records

into arrays, which use less memory:

polyline = { { 10.3, 98.5 },

{ 10.3, 18.3 },

{ 15.0, 98.5 },

...

}

For a polyline with one million points, this change reduces the use of memory

from 95 KBytes to 65 KBytes. Of course, you pay a price in readability: p[i].x

is easier to understand than p[i][1].

A yet more economic alternative is to use one list for the x coordinates and

another one for the y coordinates:

polyline = { x = { 10.3, 10.3, 15.0, ...},

y = { 98.5, 18.3, 98.5, ...}

}

The original p[i].x now is p.x[i]. With this representation, a one-million-point

polyline uses only 24 KBytes of memory.

A good place to look for chances of reducing garbage creation is in loops. For

instance, if a constant table is created inside a loop, you can move it out the loop,

or even out of the function enclosing its creation. Compare:

24

2 · Lua Performance Tips

function foo (...)

for i = 1, n do

local t = {1, 2, 3, "hi"}

-- do something without changing ’t’

...

end

end

local t = {1, 2, 3, "hi"}

-- create ’t’ once and for all

function foo (...)

for i = 1, n do

-- do something without changing ’t’

...

end

end

The same trick may be used for closures, as long as you do not move them out

of the scope of the variables they need. For instance, consider the following

function:

function changenumbers (limit, delta)

for line in io.lines() do

line = string.gsub(line, "%d+", function (num)

num = tonumber(num)

if num >= limit then return tostring(num + delta) end

-- else return nothing, keeping the original number

end)

io.write(line, "\n")

end

end

We can avoid the creation of a new closure for each line by moving the inner

function outside the loop:

function changenumbers (limit, delta)

local function aux (num)

num = tonumber(num)

if num >= limit then return tostring(num + delta) end

end

for line in io.lines() do

line = string.gsub(line, "%d+", aux)

io.write(line, "\n")

end

end

However, we cannot move aux outside function changenumbers, because there

aux cannot access limit and delta.

25

For many kinds of string processing, we can reduce the need for new strings

by working with indices over existing strings. For instance, the string.find

function returns the position where it found the pattern, instead of the match.

By returning indices, it avoids creating a new (sub)string for each successful

match. When necessary, the programmer can get the match substring by calling

string.sub.4

When we cannot avoid the use of new objects, we still may avoid creating

these new objects through reuse. For strings reuse is not necessary, because Lua

does the job for us: it always internalizes all strings it uses, therefore reusing them whenever possible. For tables, however, reuse may be quite effective.

As a common case, let us return to the situation where we are creating new

tables inside a loop. This time, however, the table contents are not constant.

Nevertheless, frequently we still can reuse the same table in all iterations,

simply changing its contents. Consider this chunk:

local t = {}

for i = 1970, 2000 do

t[i] = os.time({year = i, month = 6, day = 14})

end

The next one is equivalent, but it reuses the table:

local t = {}

local aux = {year = nil, month = 6, day = 14}

for i = 1970, 2000 do

aux.year = i

t[i] = os.time(aux)

end

A particularly effective way to achieve reuse is through memoizing. The basic

idea is quite simple: store the result of some computation for a given input

so that, when the same input is given again, the program simply reuses that

previous result.

LPeg, a new package for pattern matching in Lua, does an interesting use of

memoizing. LPeg compiles each pattern into an internal representation, which

is a “program” for a parsing machine that performs the matching. This compila-

tion is quite expensive, when compared with matching itself. So, LPeg memoizes

the results from its compilations to reuse them. A simple table associates the

string describing a pattern to its corresponding internal representation.

A common problem with memoizing is that the cost in space to store previous

results may outweigh the gains of reusing those results. To solve this problem

in Lua, we can use a weak table to keep the results, so that unused results are

eventually removed from the table.

In Lua, with higher-order functions, we can define a generic memoization

function:

4It would be a good idea for the standard library to have a function to compare substrings, so that we could check specific values inside a string without having to extract that value from the string (thereby creating a new string).

26

2 · Lua Performance Tips

function memoize (f)

local mem = {}

-- memoizing table

setmetatable(mem, {__mode = "kv"})

-- make it weak

return function (x)

-- new version of ’f’, with memoizing

local r = mem[x]

if r == nil then

-- no previous result?

r = f(x)

-- calls original function

mem[x] = r

-- store result for reuse

end

return r

end

end

Given any function f, memoize(f) returns a new function that returns the same

results as f but memoizes them. For instance, we can redefine loadstring with

a memoizing version:

loadstring = memoize(loadstring)

We use this new function exactly like the old one, but if there are many repeated

strings among those we are loading, we can have a substantial performance

gain.

If your program creates and frees too many coroutines, recycling may be an

option to improve its performance. The current API for coroutines does not offer

direct support for reusing a coroutine, but we can circumvent this limitation.

Consider the next coroutine:

co = coroutine.create(function (f)

while f do

f = coroutine.yield(f())

end

end

This coroutine accepts a job (a function to run), runs it, and when it finishes it

waits for a next job.

Most recycling in Lua is done automatically by the garbage collector. Lua

uses an incremental garbage collector. That means that the collector performs

its task in small steps (incrementally) interleaved with the program execution.

The pace of these steps is proportional to memory allocation: for each amount

of memory allocated by Lua, the garbage collector does some proportional work.

The faster the program consumes memory, the faster the collector tries to recycle

it.

If we apply the principles of reduce and reuse to our program, usually the

collector will not have too much work to do. But sometimes we cannot avoid the

creation of large amounts of garbage and the collector may become too heavy.

The garbage collector in Lua is tuned for average programs, so that it performs

reasonably well in most applications. However, sometimes we can improve the

27

performance of a program by better tunning the collector for that particular

case.

We can control the garbage collector through function collectgarbage, in

Lua, or lua_gc, in C. Both offer basically the same functionality, although with

different interfaces. For this discussion I will use the Lua interface, but often

this kind of manipulation is better done in C.

Function collectgarbage provides several functionalities: it may stop and

restart the collector, force a full collection cycle, force a collection step, get the

total memory in use by Lua, and change two parameters that affect the pace

of the collector. All of them have their uses when tunning a memory-hungry

program.

Stopping the collector “forever” may be an option for some kinds of batch

programs, which create several data structures, produce some output based on

those structures, and exit (e.g., compilers). For those programs, trying to collect

garbage may be a waste of time, as there is little garbage to be collected and all

memory will be released when the program finishes.

For non-batch programs, stopping the collector forever is not an option.

Nevertheless, those programs may benefit from stopping the collector during

some time-critical periods. If necessary, the program may get full control of

the garbage collector by keeping it stopped at all times, only allowing it to run

by explicitly forcing a step or a full collection. For instance, several event-

driven platforms offer an option to set an idle function, which is called when

there are no other events to handle.

This is a perfect time to do garbage

collection. (In Lua 5.1, each time you force some collection when the collector

is stopped, it automatically restarts.

So, to keep it stopped, you must call

collectgarbage("stop") immediately after forcing some collection.)

Finally, as a last resort, you may try to change the collector parameters.

The collector has two parameters controlling its pace. The first one, called

pause, controls how long the collector waits between finishing a collection cycle

and starting the next one. The second parameter, called stepmul (from step

multiplier), controls how much work the collector does in each step. Roughly,

smaller pauses and larger step multipliers increase the collector’s speed.

The effect of these parameters on the overall performance of a program

is hard to predict. A faster collector clearly wastes more CPU cycles per se;

however, it may reduce the total memory in use by a program, which in turn

may reduce paging. Only careful experimentation can give you the best values

for those parameters.

Final remarks

As we discussed in the introduction, optimization is a tricky business. There

are several points to consider, starting with whether the program needs any

optimization at all. If it has real performance problems, then we must focus on

where and how to optimize it.

28

2 · Lua Performance Tips

The techniques we discussed here are neither the only nor the most impor-

tant ones. We focused here on techniques that are peculiar to Lua, as there are

several sources for more general techniques.

Before we finish, I would like to mention two options that are at the border-

line of improving performance of Lua programs, as both involve changes outside

the scope of the Lua code. The first one is to use LuaJIT, a Lua just-in-time

compiler developed by Mike Pall. He has been doing a superb job and LuaJIT

is probably the fastest JIT for a dynamic language nowadays. The drawbacks

are that it runs only on x86 architectures and that you need a non-standard Lua

interpreter (LuaJIT) to run your programs. The advantage is that you can run

your program 5 times faster with no changes at all to the code.

The second option is to move parts of your code to C. After all, one of Lua

hallmarks is its ability to interface with C code. The important point in this case

is to choose the correct level of granularity for the C code. On the one hand, if you

move only very simple functions into C, the communication overhead between

Lua and C may kill any gains from the improved performance of those functions.

On the other hand, if you move too large functions into C, you loose flexibility.

Finally, keep in mind that those two options are somewhat incompatible. The

more C code your program has, the less LuaJIT can optimize it.

3

Vardump: The Power of

Seeing What’s Behind

Tobias S ülzenbr ück and Christoph Beckmann

A universal debugging tool is popular in other programming languages, such

as the scripting language PHP. When coming to writing code it seems as an

advantage for us to work with a quite usable built-in function called vardump to

quickly overview variable contents. There the contents are structured in order

by their appearance in arrays or by their data types which makes it easier to

read the variables and refer to them.

This means in detail that when programming you may want to supervise

your data. This abstract description of the problem covers nearly all issues

when dealing with complex program structures or when having trouble because

some variables or classes get out of hand. For instance, you are implementing a

non-trivial algorithm and while testing certain functionality you will print out

variables to control single values.

But what to do with complex data structures? You will need a tool that is

capable of dealing with custom-made structures. For instance, a nested table

containing data should be printed out. Instead of reinventing a special output

function for your problem, it is likely better to use an generic function that covers

all possible data types. This fact becomes more important in dynamically typed

languages like Lua. (In the beginning of experiencing Lua we hardly missed

such a functionality as described above.)

In this gem, we introduce a universal debugging tool for Lua. This solution

covers many demands that programmers have in their development workflow.

30

3 · Vardump: The Power of Seeing What’s Behind

Here is a simple example of vardump in action. It prints out the variable foo

that contains the string “Hello World” .

> foo = "Hello World"

> vardump(foo)

(string) Hello World

Implementation

As shown in Listing 1, vardump is a Lua function with three parameters: one

for the resulting data value and two others reserved for the recursive function

invocation through itself.

First the key parameter is checked for adding a line prefix. This is used when

printing tables to also print out the index of the table cell. Second, depending

on the table depth (i.e., the current iteration step when printing a table), spaces

are added in front of each line to enhance the readability. When calling vardump

with a simple data type, such as a string, the afore mentioned additions to lines

have no influence on those. As said before, this is only interesting when printing

out tables, because the Lua print function automatically adds a line break.

Next the value is checked against some basic types. This includes tables,

functions, threads, userdata, and all other types the data in vardump can have.

When the resulting type is a simple data type, it is displayed with the Lua print

function. The data type is printed in brackets in front of the value. Functions,

threads, userdata, and nil values are printed without their data types at the

beginning of the line. E.g. the vardump of a function will print out its memory

address, as the standard print function will do.

Tables are the universal structure in Lua and they need special handling in

vardump. As tables may contain other tables it is essential to get all information

out of them, no matter how deep the nesting is. The output of vardump for a set

of nested tables is shown in Listing 2.

The function begins with obtaining the current iteration depth. From the

depth value, the amount of fore-standing spaces in determined. A table can

contain other tables or the special metatable. If so, the current value is replaced by the contents of the metatable. Next, vardump is invoked recursively for each

pair in the table. The new value and the corresponding key are two of the

arguments for the call. The third parameter is the current depth, as mentioned

for adding spaces at the beginning of the line.

Conclusion

The functionality of vardump is powerful and a must for any developer. vardump

gives you the transparency you need for every variable you use. One improve-

ment for extending vardump to adapt it to your workflow might be another argu-

ment that describes the maximum iteration depth for tables — this is very useful

when handling large, deeply nested tables.

31

function vardump(value, depth, key)

local linePrefix = ""

local spaces = ""

if key ~= nil then

linePrefix = "["..key.."] = "

end

if depth == nil then

depth = 0

else

depth = depth + 1

for i=1, depth do spaces = spaces .. "

" end

end

if type(value) == ’table’ then

mTable = getmetatable(value)

if mTable == nil then

print(spaces ..linePrefix.."(table) ")

else

print(spaces .."(metatable) ")

value = mTable

end

for tableKey, tableValue in pairs(value) do

vardump(tableValue, depth, tableKey)

end

elseif type(value) == ’function’

or type(value) == ’thread’

or type(value) == ’userdata’

or value

== nil

then

print(spaces..tostring(value))

else

print(spaces..linePrefix.."("..type(value)..") "..tostring(value))

end

end

Listing 1. The vardump function.

32

3 · Vardump: The Power of Seeing What’s Behind

> foo = {"zero",1,2,3,{1,{1,2,3,4,{1,2,{1,"cool",2},4},6},3,vardump,5,6}, 5,{Mary = 10, Paul = "10"},"last value"}

> vardump(foo)

(table)

[1] = (string) zero

[2] = (number) 1

[3] = (number) 2

[4] = (number) 3

[5] = (table)

[1] = (number) 1

[2] = (table)

[1] = (number) 1

[2] = (number) 2

[3] = (number) 3

[4] = (number) 4

[5] = (table)

[1] = (number) 1

[2] = (number) 2

[3] = (table)

[1] = (number) 1

[2] = (string) cool

[3] = (number) 2

[4] = (number) 4

[6] = (number) 6

[3] = (number) 3

function: 0x304650

[5] = (number) 5

[6] = (number) 6

[6] = (number) 5

[7] = (table)

[Mary] = (number) 10

[Paul] = (string) 10

[8] = (string) last value

Listing 2. Output of vardump for nested tables.

4

Serialization with Pluto

Ben Sunshine-Hill

Serialization refers to the process of taking a piece or set of data from a running

program and writing it to a one-dimensional datastream (such as could be

stored in a string or a file), with the goal of restoring that data later from the

datastream. The most common use of serialization is implementing a “save”

feature.

In this instance, the serialized data is the data in a spreadsheet

application, or the state of the gameboard in a chess game. A closely related use

is the creation of “rollback points”, useful in simulations and databases, which

allows the application to revert to an earlier state. The idea of rollback points

is particularly interesting as it relates to coroutines, because the state of the

application is contained not only in the data of the program, but in the execution

state of all extant coroutines. A similar situation can easily be encountered in a

save-game feature in video games, particularly where the state of a character’s

AI is embodied in a coroutine.

The plot, therefore, thickens: like many problems in computer science, seri-

alization starts out sounding trivial, and becomes more complex as the full scope

of the problem is examined. An additional level of complexity arises when one

begins to consider the practical aspects of serialization as opposed to treating it

as an exercise in theory. Indeed, the creation of a serialization system that is

both correct and useful is a remarkably involved task. Because of this, premade

serialization libraries are a convenient way to reduce development time while

ensuring a robust, efficient result. In the ideal case, the complexities of deciding

on a file format and keeping the loading and saving code in sync are swept away

in favor of a simple “serialize this”. Pluto is one such system, which handles the

34

4 · Serialization with Pluto

complexities of serialization and presents as simple an interface as possible (but

no simpler).

The complexities of serialization

This section covers some of the theoretical aspects of serialization. If the phrase

“depth-first graph traversal” makes your eyes glaze over, you can skip to the

next section.

Much of the complexity of serialization arises from the fact that a single piece

of “data” is rarely confined to a single object. The string "foo" is a single object, as is the string "bar", but the table {foo = "bar"} consists of three objects: the table, the key, and the value. And we must deal with more than simple associative mappings: the table {{"foo"},{"bar","baz",{"qux","quux"}}} consists of four tables and five strings (not counting the implicit integer keys). The real

problems, though, begin to arise when the data is structured not as a tree but

as a generalized graph. Consider the data returned by the following function:

function creategraph()

local a = {}

local b = {}

a.foo = b

b.foo = a

return a

end

The returned object is a table, which has as its single value a table, which

has as its single value the first table. This structure cannot be represented

declaratively, because of the cyclic links. Likewise, during deserialization the

structure cannot be rebuilt without visiting at least one of the tables twice: If

the first table is rebuilt first, it cannot refer to a fully rebuilt second table; and

vice versa.

The solution employed by most serialization libraries, including Pluto, is

to layer a new container over the objects. The data returned by creategraph

above could be equivalently generated by the function in Listing 1. The array

objs is first filled with every object in the data, each one initially “empty” but

referenceable. The objects are then filled, one by one, with the other objects

which they reference, and the root objects (the initial set of objects explicitly

chosen to be serialized) are returned. It should be clear from the above code

that the ordering of the objects in obj can be arbitrary: as long as all objects are

created before all objects are filled, the creation and filling of individual objects

can be performed in any order.

In practice, these two phases may be interleaved. The algorithm is closely

based on a depth-first graph traversal, and is as follows. During serialization,

maintain a table of written objects and the integer indices assigned to them.

To write an object, check whether that object has already been written. If it

has, output the integer index assigned to the object when it was first written.

35

function creategraph2()

local objs = {}

local insert = function(tableindex, keyindex, valueindex)

local table = objs[tableindex]

local key = objs[keyindex]

local value = objs[valueindex]

table[key] = value

end

-- creation

objs[1] = {}

objs[2] = {}

objs[3] = "foo"

-- filling

insert(1, 3, 2)

insert(2, 3, 1)

return objs[1]

end

Listing 1.

Otherwise, assign a new integer index to the object and output that index, and

output the object’s data. For other objects that it references, recursively write

those objects. During deserialization, maintain an array of integer indices and

the objects to which they refer. when an object is encountered, read whether that

object has been encountered before. If it has, simply return the object with the

given index. Otherwise, create a new object of the specified type within the array

at the specified index, and then read the object’s data (recursively invoking the

routine to read objects which have been referenced), finally returning the fully

created object.

The correctness of these algorithms may easily be proven: Assuming that

integer indices are assigned incrementally, it may be shown that during seri-

alization no object completes being written unless all other objects with lower

indices already been assigned these indices. Likewise, it may be shown that

during deserialization no object begins being written unless all other objects

with lower indices have already been created, and that all created objects are

fully initialized by the time deserialization completes.

Using Pluto

Pluto is implemented as a Lua module in C, which must be built for a particular

version of the Lua interpreter (due to its direct manipulation of Lua’s data

36

4 · Serialization with Pluto

structures). It can serialize every Lua type except for C functions, which would

not be possible without architectural support. (Such a feature would usually

be undesirable anyway; C functions can be registered as permanent objects,

as described later.) In particular, Pluto correctly supports shared upvalues, a

necessary feature for certain OO systems.

In its simplest form, Pluto can be used as follows:

require("pluto")

obj = "Hello, world!"

encoded = pluto.persist({}, obj)

outfile = io.open("test.plh", "wb")

outfile:write(encoded)

outfile:close()

-- then, later...

infile = io.open("test.plh", "rb")

encoded = io.read("*a")

infile.close()

obj = pluto.unpersist({}, encoded)

print(obj) -- prints "Hello, world!"

This example barely leverages Pluto’s capabilities: obj could be replaced with

an object contained within a complex, interconnected mass of data, and all of

that data would be saved and reloaded correctly.

Custom serialization routines

Not all objects can be blindly serialized without regard to the semantics of

the data. At certain times, special procedures are necessary to serialize data

in such a manner that it will be useful when deserialized. The most obvious

example of this in Lua is userdata. Because Lua has no idea what the content

of the userdata represents, writing that data out as a bytestream may not be

a sensible thing to do, and restoring the data from the bytestream may not

be a safe thing to do. For instance, if the userdata consists of a pointer to an

array of integers, that pointer will be dangling if the object is deserialized on a

subsequent run of the program. If such data is to be persisted, the programmer

will need to supply a custom routine to serialize the userdata using a format

of their own devising (which would then be inlined by the library into the

bytestream), and a matching routine to deserialize it later. Pluto accomplishes

this with a __persist metamethod, which is called by Pluto when the object is

encountered. This metamethod takes the object as its sole argument, and is

expected to return a function which will, when run, produce the original object.

Upvalues are generally necessary to accomplish this. Suppose, for instance, that

we had a userdata-based 2D vertex type, produced by the constructor function

vertex2d(x,y), which we needed to serialize:

37

require("pluto")

function persistvertex2d(v)

local x = v.x

local y = v.y

return function()

return vertex2d(x, y)

end

end

-- obj’s metatable’s __persist method is set to persistvertex2d

obj = vertex2d(3, 4)

encoded = pluto.persist({}, obj)

In this example, persistvertex2d takes a vertex as its argument, and re-

turns a closure which, when called, returns the original vertex. The key here,

however, is that the inner function does not refer to v at all, only to locally-

declared variables. When this closure is persisted in lieu of the original vertex,

it will pull in x and y to use later in reconstruction. Note also that the inner

function does not refer to the C function vertex2d, only to the string containing

its name. If we did not wish to look up the constructor via the global table, we

could instead refer to the vertex2d function as a permanent object.

Permanent objects

When one examines the space of situations where custom serialization routines

are necessary, two distinct patterns emerge. In one pattern, it is fully possible

to serialize the state of the object into a file as long as the system knows how.

In the other pattern, however, the state of the object extends past the scope of

the serialized data. A userdata, for instance, may be a handle to a database, for

instance, or to a hardware function used to access the system time.

One can invent different custom serialization routines for these, of course,

but they all tend to involve the same thing: fixing up references during deseri-

alization with preexisting objects. In the case of the system time function, for

instance, it is likely that during deserialization such a routine will have already

been loaded by a module, just as it was before the creation of the data which was

originally serialized. Ideally, therefore, during serialization the system would

not even attempt to save this routine to disk, but instead describe the routine in

some way (likely through a unique identifier). During deserialization, the same

routine in its new instantiation would be connected with the same identifier,

and the objects being loaded would have their references fixed-up to reference

this new routine. Some convention would be used to ensure that the new ob-

ject would be equivalent to the old one, such as giving the function’s “canonical

name” in the global namespace as its identifier.

38

4 · Serialization with Pluto

Pluto supports this fixup behavior with a “permanents table”. During seri-

alization, a table of permanents is passed in (the empty first argument in the

examples above), with the keys being the permanent objects and the values be-

ing the identifiers for the objects. The identifiers are serialized in the normal

way, and can be of any type, although in practice only integers and strings are

used. During deserialization, the reverse is passed in, with the keys being the

identifiers and the values being the permanent objects. In the example shown

in Listing 2, canvas is a userdata created by the Canvas Draw library, which

refers to the hardware screen; for obvious reasons, it cannot be serialized, and

is instead fixed up via an entry in the permanents table.

One common situation in which the permanents table is required is that of

serializing coroutines. When a coroutine has yielded by calling coroutine.yield,

that C function is still referenced by the coroutine’s callstack. If Pluto tried to

serialize the callstack, it would fail to serialize that value. Therefore, in order

to serialize a running coroutine it is necessary to have coroutine.yield in the

permanents table.

Limitations of Pluto

There are certain guarantees which Pluto does not currently provide. None of

these are fundamental limitations of the technology, but rather implementation

decisions which keep the library’s design simple. If any of these features would

be particularly useful to Pluto’s users, they could be added to a future release.

First, Pluto does not handle byte ordering issues. It is assumed that the

memory representations of numbers will be the same between serialization

and deserialization. This makes Pluto of limited use in network protocols for

cross-platform applications, and for other situations where a differently endian

architecture will be deserializing data.

Secondly, Pluto is not hardened against invalid bytestreams.

Untrusted

bytestreams should not be deserialized in security-critical situations, as they

could crash the application or even enable code-injection attacks.

Finally, Pluto uses an inefficient algorithm for deserializing certain types of

upvalues, requiring a traversal of the entire garbage collection list. This has not

caused any known significant slowdowns, but applications with extremely large

working sets could conceivably experience problems.

Other approaches to general-purpose serialization

In this section, I will examine two other serialization possiblities available to

the Lua programmer, covering both the concepts behind their implementation

and the practicalities of choosing each one for a serialization system. Although

Pluto is a robust and general-purpose library, these approaches have advantages

of their own.

39

require("pluto")

nativecanvas = cd.CreateCanvas(cd.NATIVEWINDOW, nil)

permanents = {[nativecanvas] = "CD canvas"}

drawingagent = {

canvas = nativecanvas,

drawblueline = function(this, x1, y1, x2, y2)

this.canvas.SetForeground(cd.BLUE)

this.canvas.Line(x1, y1, x2, y2)

end,

drawredline = function(this, x1, y1, x2, y2)

this.canvas.SetForeground(cd.RED)

this.canvas.Line(x1, y1, x2, y2)

end}

encoded = pluto.persist(permanents, drawingagent)

outfile = io.open("test.plh", "wb")

outfile:write(encoded)

outfile:close()

-- Then, later (on a different run of the program)...

nativecanvas = cd.CreateCanvas(cd.NATIVEWINDOW, nil)

permanents = {["CD canvas"] = nativecanvas}

infile = io.open("test.plh", "rb")

encoded = io.read("*a")

infile.close()

drawingagent = pluto.unpersist(permanents, encoded)

Listing 2.

40

4 · Serialization with Pluto

LuaPickle

The first approach is found in the “Lua Pickle” library available on the lua-

users.org wiki. This library is implemented entirely in Lua and outputs plain

text files, making it fully cross-platform. Elegantly, it outputs data as a Lua

program, which is simply executed to deserialize the data. The ease with which

this is possible reflects Lua’s pedigree as a data-description language.

LuaPickle does not support custom serialization routines or permanent ob-

jects. That is not a result of any fundamental limitations of the technology,

though, and could be added to the library without too much effort. The pure-Lua

approach, however, does limit the number of built-in types which may be seri-

alized. Userdata, coroutines, and functions are all unsupported, as the built-in

Lua libraries do not provide adequate introspection facilities for them.

As the library requires no nonstandard native code, it is the easiest of the

three to integrate with an existing program. If you are certain you will never

need to serialize any of the unsupported data types, its convenience is un-

matched.

lper

lper is a melding of the Lua virtual machine with LPSM, an off-the-shelf per-

sistent memory manager. By maintaining the VM’s entire memory space in a

disk-backed virtual memory region (a surprisingly straightforward task, thanks

to Lua’s support for custom memory allocation routines), the entire Lua universe

may be written and read. The simplicity of this approach is admirable: after all

the worrying about upvalues and userdata, this type-agnostic system can seri-

alize it all, in a manner reminiscent of Alexander’s cutting of the Gordian Knot.

If this approach is sufficient for your needs, it’s difficult to beat for simplicity,

power, and ease of use.

There are definite tradeoffs, however. The chief issue is that the entire Lua

universe must be saved. This severely limits its usefulness for saving games or

documents, unless an effort is made to segregate the data into a separate VM

instantiation (which presents its own set of difficulties relating to data sharing).

lper is therefore best suited for long-running, processor-intensive simulations.

In such a situation, lper could be used for creating rollback points, where poten-

tially most of the system state must be saved regardless. It is also necessary

to ensure that references to memory not allocated by Lua (such as registered C

functions) remain invariant across invocations. Custom serialization routines

are also unsupported. Finally, lper is an experimental library, which has not

been fully tested and is limited to POSIX environments due to the requirements

of LPSM.

41

Conclusion

Serialization is an important task for many applications, and often a compli-

cated one, particularly if it is not planned for from the outset. At the same time,

it is undeniably prosaic. The implementation of a robust serialization system

is a task to be deferred to third-party solutions whenever possible, to allow the

programmer to concentrate on application-specific tasks. If you are planning out

the technology to be used for an application, or if you need to graft serialization

or persistence onto an existing project, Pluto can minimize the pain involved in

integrating serialization.

5

Abstractions for LuaSQL

Tom ás Guisasola Gorham

This article shows how to build an abstraction layer over LuaSQL to ease

the most common uses of the library made by application developers.

The

reader is expected to know Lua and the basics of LuaSQL: how to install,

open a connection, and execute SQL statements. We will show some common

uses of LuaSQL’s API, extracted from our own experience, and try to develop,

step by step, a set of abstractions to simplify them, aiming at a higher level

programming style.

We will begin by showing an example from which we point out common pieces

of code that are found in many programs. The following four sections will de-

tail those constructions, showing some forms of generalization and abstraction

that should help make the whole program easier to write, maintain and under-

stand. Finally, a complete abstraction is obtained in the form of a library that

encapsulates the main of LuaSQL.

Common uses

Listing 1 shows an example of a common use of LuaSQL library, which includes

almost all the points we plan to examine. These points are marked with num-

bers between parentheses.

The example starts by loading a LuaSQL driver and opening the connection.

This initialization phase is marked by number (1). Then the example builds an

SQL statement (2), sends it to the database and checks for errors (3), and finally

44

5 · Abstractions for LuaSQL

-- Initialization (1)

require"luasql.postgres"

local env = luasql.postgres ()

local conn = assert (env:connect ("lpg"))

-- Building SQL statement (2)

local course_list = { "Music", "Literature", }

local c_list = "’"..table.concat (course_list, "’,’").."’"

local year = 2007

local stmt = string.format ([[

select a.id, a.name

from alumn a inner join course c on (a.course_id = c.id)

where c.name in (%s) and adm_year = %s]], c_list, year)

-- SQL execution and error handling (3)

local cur, err = conn:execute (stmt)

if not cur then

error (err.." SQL = {"..stmt.."}")

end

-- Iteration loop (4)

local id, name = cur:fetch ()

while id do

print (id, name)

id, name = cur:fetch ()

end

-- Closing (5)

cur:close()

conn:close()

env:close()

Listing 1. An ordinary complete sample of LuaSQL use, where the typical phases

are marked by numbers between parentheses.

45

retrieves the results set (4).1 These phases will be analyzed in the following

sections.

Defining a module

As mentioned above, we will develop a Lua module to group all the abstractions

together. We shall use a table to encapsulate the actual LuaSQL connection

and add functions/methods to its metatable. The programmer can access the

actual LuaSQL object to perform other operations such as turn the auto-commit

mode on or off, or call the commit and the rollback methods. Let us start with a

constructor of this new type of object, which will be also responsible for opening

the connection to the database, and a closing function. This will constitute the

file database.lua.

local assert, require, setmetatable = assert, require, setmetatable

module"database"

local mt = { __index = _M, } -- _M is the module itself

function connect (dbname, user, pass, driver)

local luasql = require("luasql."..driver)

local env = assert (luasql[driver] ())

local obj = { conn = assert (env:connect (dbname, user, pass)) }

setmetatable (obj, mt)

return obj

end

function close (obj)

setmetatable (obj, nil)

return obj.conn:close ()

end

We will also write a test file. It will be useful for testing, but also as a set of

use samples.

local database = require"database"

local db = database.connect ("lpg", nil, nil, "postgres")

From now on, we will develop the following two files in parallel: the module

file (database.lua) and the test file. Sometimes we will enhance a piece of

code that had been developed earlier and thus it will be replaced by the new

implementation.

1The code that retrieves the results set can be more compact like this:

for id, name in cur.fetch, cur do print (id, name) end

I chose the more verbose version mainly because I could not found any use of this compact form in a search in the Internet, at least by the time of this writing. Anyway, my point is that legibility could be improved in both forms.

46

5 · Abstractions for LuaSQL

Error handling

LuaSQL handles errors just like the standard Lua libraries: an error is raised

only if the arguments do not follow the types defined by the API. Errors gener-

ated by the database client, such as incorrect SQL syntax, unknown identifiers,

or even violation of database restrictions, are informed in the conventional way,

by returning false and an error message. This behavior provides the program-

mer with the freedom to check for errors only when they show up. Although it is

tedious to write down an if-test everywhere, the fact is that they are not usually

written anywhere! However, a simple function can do this for us. Let us add the

following definition to our module:

function assertexec (self, stmt)

local cur, msg = self.conn:execute (stmt)

return cur or error ((msg or ’’).." SQL = { "..stmt.." }", 2)

end

To test it, let us create a test database and insert some rows into it, not

forgetting to check if it raises errors properly:

assert (pcall (db.assertexec, db, "wrong SQL statement") == false)

db:assertexec[[create table people (

id

integer,

name

varchar (100),

sex

char(1),

tel

varchar (10)

)]]

-- Adding content to the table

db:assertexec"insert into people values (1, ’John Doe’, ’M’, ’12’)"

db:assertexec"insert into people values (2, ’Jane Doe’, ’F’, ’01’)"

db:assertexec"insert into people values (3, ’O\\’Neill’, ’M’, ’98’)"

Now we can try a query and make sure it executes correctly:

local stmt = "select * from people where name = ’John Doe’"

assert (pcall (db.assertexec, db, stmt))

Since it is a frequent mistake to forget the quotes around a string, we should

add a test to make sure the system always raises an error in this situation:

local stmt = "select * from people where name = John Doe"

assert (not pcall (db.assertexec, db, stmt))

Result set iterator

Now we will try to generalize the iteration loop, by adding an iterator factory to

our module:

47

function select (self, stmt)

local cur = self:assertexec ("select "..stmt)

return function ()

return cur:fetch ()

end

end

With this new function, the iteration loop can be replaced by a for-construction.

Besides its conciseness, the for-construction avoids the need for two calls to the

fetch method, one before the loop starts and the other at the end of its body. The

following code should be added to the test file in order to test the new function:

for name, sex in db:select"name, sex from people" do

print (name, sex)

end

If only one row is needed, we can make it simpler:

local n, t = db:select"name, tel from people where id = 1"()

assert (n == "John Doe")

assert (t == "12")

This implementation covers the most common uses, but it restricts the call to

fetch, preventing the return of a table with the values of all columns in a row.

In order to add flexibility and keep the module easy and practical to use, we can

add an optional parameter to our iterator function to indicate whether we want

a table. Additionally, we could use this parameter to indicate the modestring

used by fetch in the construction of the table:

function select (self, stmt, modestring)

local cur = self:assertexec ("select "..stmt)

return function ()

local t

if modestring then t = {} end

return cur:fetch (t, modestring)

end

end

When called with this third argument, the iterator will create a new table

to store the values of the row and return this table. There is no restriction

anymore, but there are two drawbacks: we must specify a modestring to obtain

a table with the values, and we cannot reuse that table. I consider both of

minor importance: this is the price of convenience. Besides, one can use the

raw LuaSQL connection object if necessary.

SQL statement constructors

Our final subject is the creation of SQL statements. This task is completely dif-

ferent from everything done to our module so far. One might prefer to develop

48

5 · Abstractions for LuaSQL

a separate module for that, but we will put everything together for concision.

Our main goal is to provide both practicality and robustness. Practicality can

be achieved with a small set of functions covering the most common SQL state-

ments: delete, insert, update and select. Robustness — at this level — has to be

assured by properly quoting and escaping the sentences, preventing common

mistakes and also reducing tedious work — which is another common cause of

error.

Infrastructure

As we have mentioned, a common mistake is to forget to quote a string, but a

more common one is to forget to escape a quote inside a quoted string. These

arguments should be enough to force us to define functions for escaping and

quoting a given string.

Until LuaSQL 2.1 there was no support for these

operations2 thus both had to be done in Lua.

These operations should be

included in the assembly of the SQL statements but we should also be cautious

about their use, so that we do not escape or quote the same string twice.

Nevertheless sometimes we do not want a quoted string, for example when

using a select as the value of a column (a sub-select), or when using a pre-defined

database value such as NULL or CURRENT TIMESTAMP. Consequently it is important

to let the user differentiate these situations in a convenient way.

Since all SQL expressions could be represented between parentheses — and,

in fact, the case of sub-select have to be done this way —, we decided that

parenthesized strings would not be quoted. Thus, we can write the quote function

in order to quote only strings that are not enclosed by parentheses3:

function escape (s) return (string.gsub (s, "’", "\\%1")) end

function quote (s)

if string.find (s, "^%(.*%)$") then

return s

else

return "’"..escape(s).."’"

end

end

Insert

Now let us consider a change to our test file, establishing that every value

retrieved from the database be checked. A reasonable way to do that is by

defining a Lua table with all the values we want. Then, an automatic routine

could store this data on the database and another routine could retrieve and

check the values, item by item, comparing them to the original data.

2The escape function was added to LuaSQL 2.2 as a consequence of writing this article.

3Some systems require that a single quote be escaped with two single quotes instead of a

backslash, as shown in the code.

49

In order for this to work, we need an insert method in our database connec-

tion. Basically, an insert SQL statement contains three “arguments”: a table

name, an optional list of columns and a list of values. The natural way to pro-

vide lists in Lua is using a table as an array. Better yet, since we want two

corresponding lists we can use the same table to provide both pieces of informa-

tion: the list of columns is the list of table keys, and the list of values is the list

of values associated with these keys. Therefore, our new method only needs to

inform the name of the table to act on and a table with the column-value pairs,

as in:

db:insert ("people", { id=1, name="John Doe", sex="M", tel="12", }) The function that builds the two lists may be added to our infrastructure as

displayed below:

function twolists (tab)

local k, v = {}, {}

local i = 0

for key, val in pairs (tab) do

i = i+1

k[i] = key

v[i] = quote (val)

end

return table.concat (k, ’,’), table.concat (v, ’,’)

end

The twolists function can also be used to build parts of other SQL state-

ments as will be shown later.

Hence, the implementation of the insert method can be:

function insert (self, tablename, contents)

return self:assertexec (string.format (

"insert into %s (%s) values (%s)",

tablename, twolists (contents)))

end

Our test script can be rewritten to automatically populate the database with

data from a table, by using the following code:

-- Set of data

data = {

{ name = "John Doe", sex = "M", tel = "12", },

{ name = "Jane Doe", sex = "F", tel = "01", },

{ name = "O’Neill",

sex = "M", tel = "98", },

}

-- Adding content to the table

for i, row in ipairs (data) do

row.id = i

db:insert ("people", row)

end

50

5 · Abstractions for LuaSQL

Select revisited

We can add the same facility to assemble the SQL statement of our result

set iterator. The second argument, that is, the statement, can be replaced by

a string with a list of columns, followed by the table name, the conditional

expression and any other text. In this way, the iterator will be responsible for

adding some words to guarantee the correct syntax of the statement:

function select (self, columns, tabname, cond, other, modestring)

-- Assemble the SQL statement

tabname = tabname and (" from "..tabname) or ""

cond

= cond and (" where "..cond) or ""

other

= other or ""

local stmt = string.format ("select %s%s%s %s",

columns, tabname, cond, other)

-- Do the query

local cur = self:assertexec (stmt)

return function ()

local t

if modestring then t = {} end

return cur:fetch (t, modestring)

end

end

Sometimes it is important to hide the internals of the implementation. In

this case, however, I believe it is better to expose it.

In other words, it is

important for the programmer to know that the arguments will be joined to form

the final SQL statement, because he is able to use this to his own advantage.

The programmer can exploit the fact that the list of columns is not just a list

of columns and add more text to enhance the SQL statement being built, like

renaming a column, adding two or more columns with a string separator and

others. The same applies to all of the arguments4. Now we can automatically

check each column of each row against the original data:

for row in db:select ("*", "people", nil, "order by id", "a") do row.id = tonumber(row.id)

for col, val in pairs (data[row.id]) do

assert (row[col] == val)

end

end

A subtle point to note is the release of open cursors, which are confidently

left to the garbage collector. LuaSQL’s implementation of fetch5 already closes

4In fact, this new implementation can be used just like the others with the raw SQL statement (removing the “select” word from the beginning) as in: db:select"* from people order by id".

5This behavior — closing the cursor when there are no more rows — is in part a consequence of writing this article and was planned to be added to LuaSQL version 2.2, which should have been released by the time this article is published.

51

the cursor when there is no more rows to return, but if the iterator is not called

to the end, the cursor remains open. In some systems, with severe restrictions,

this practice could make the system get out of resources, therefore the select

iterator have to be used with care. The most effective way to avoid this situation

is to create queries that return the exact number of rows needed, so that the

loop will call fetch until there is no more rows and the cursor will be closed.

Nevertheless, the raw LuaSQL connection is accessible via the conn field and

the usual execute-fetch loop could be used.

Delete

The delete method should be simple, following the same guidelines used for the

insert method: the name of the table and a condition.

function delete (self, tablename, cond)

cond = cond and (" where "..cond) or ""

local stmt = string.format ("delete from %s%s", tablename, cond)

return self:assertexec (stmt)

end

As with the select method, the tablename argument can be the complete

SQL statement including the condition6, so the last argument is optional.

Update

While the insert command requires two comma-separated lists (for column names

and column values), the update command requires a single comma-separated

list of pairs in the form column-name = column-value. Since the where clause

also requires a similar list of pairs, we will define a function to cover both uses

by accepting an optional separator. By providing the string " AND " as the sep-

arator, the function can be used to form a typical condition to the where clause.

function pairslist (tab, sep)

sep = sep or ’,’

local l = {}

for key, val in pairs (tab) do

l[#l+1] = string.format ("%s=%s", key, quote (val))

end

table.sort (l)

return table.concat (l, sep)

end

Since Lua does not guarantee the order of the traversal of a table, I added

a call to table.sort so that pairslist always produces the same string for the

same contents of a table. This predictability makes it easier to test the function.

6As in db:delete"table where status=’invalid’".

52

5 · Abstractions for LuaSQL

local pl = db.pairslist ({ a="I’m a quoted text", b=2, }, " AND ")

assert (pl == [[a=’I\’m a quoted text’ AND b=’2’]])

Now we can define the update method:

function update (self, tabname, contents, cond)

cond = cond and (" where "..cond) or ""

local values = contents and pairslist(contents) or ""

local stmt = string.format ("update %s set %s%s",

tabname, values, cond)

return self:assertexec (stmt)

end

Since the pairslist function is exported from our module, the programmer

can use it to assemble the where clause for the select iterator or for the update

or delete commands.

We should add some lines to test these functions:

assert (db:select ("tel", "people", "id=3")() == data[3].tel) assert (db:update ("people", { tel = 87, sex = "F", }, "id=3") == 1) assert (db:select ("tel", "people", "id=3")() == "87") assert (db:delete ("people", "id=3") == 1)

assert (db:select ("tel", "people", "id=3")() == nil)

Finally, to clean up the test, we drop the table created:

db:assertexec"drop table people"

Extensions

Here we explore the facility to extend this library by showing a pair of examples

and proposing others. LuaSQL is not supposed to be extended directly and for

security reasons it has to be that way. On the other hand, our library is easily

extendable, such as most Lua libraries.

Complete result set

The easiest way to extend our library is by adding a function to it. A useful

function is one that builds the entire result set as an array of rows. It could be

simple as:

function selectall (self, columns, tabname, cond, other, modestring)

modestring = modestring or ’a’

local r = {}

for row in self:select (columns, tabname, cond, other, modestring) do

r[#r+1] = row

end

return r

end

53

A test for the new function could be:

rs = db:selectall ("*", "people", nil, "order by id")

assert (rs[1].name == "John Doe")

assert (rs[2].name == "Jane Doe")

assert (rs[3].name == "O’Neill")

Logging SQL execution

A more elaborated extension can be the redefinition of one of its functions. An

interesting facility that could be exploited is the redefinition of assertexec to

trace SQL execution. This could be used to provide a log of all or part of the SQL

statements executed by an application without the need to change the whole

application. A log could help debugging or even be used to fine-tune the software.

A possible redefinition could be as follows. The debug library is used to obtain

names for the functions on the call stack.

-- Redefinition of ‘assertexec’ to log SQL statements

local old_exec = database.assertexec

local fh = io.open ("sql.log", "w")

database.assertexec = function (self, stmt)

local f2 = debug.getinfo (2, "n").name or "?" -- function name

local f3 = debug.getinfo (3, "n").name or "?" -- function name

local s = stmt:gsub("%%", "%%"):gsub("[\r\n]", ""):gsub("\t", " ") fh:write ("("..f3.."->"..f2.."):"..s.."\n")

return old_exec (self, stmt)

end

Moreover, the redefinition of assertexec could also be used to provide other

features, such as a query rewriting tool for SQL statements or a proxy just like

the recent MySQL Proxy7.

Of course, such an example does not depend on the existence of assertexec,

but the fact that this wrapper already encapsulates the execute method eases

this task. LuaSQL does not provide access to the objects’ metatables for security

reasons, thus there is no way to redefine execute. The only solution I can think

of is to create a wrapper, as we are doing, to intercept the statements sent to the

database driver.

Other ideas

An even more sophisticated (and also useful) extension is the implementation of

a pool or a cache of database connections encapsulated by the connect method.

In both cases, careful must be taken on the release of open transactions and also

on the sharing policy.

A cache of database connections should improve the efficiency of an applica-

tion that repeatedly connects to the same database, does some stuff and release

7http://forge.mysql.com/wiki/MySQL Proxy

54

5 · Abstractions for LuaSQL

the connection, by avoiding the destruction and recreation of these connections.

Since the connections will not be really closed, open transactions have to be ter-

minated either by a commit or a rollback operation. Therefore, the close method should be redefined to do a commit. The rollback operation, on the other hand, have to be executed in case an error occur, a situation that is not so easy to

handle.

An interesting case is a long-term application such as a server. A Web server,

for instance, could offer a pool of connections for its request-attendants, to

optimize the database access. But it has to take care of the forgotten connections

since it is not uncommon to an attendant to trust in a general clean up at the

end of its execution and simply not close anything! Since our connections are

encapsulated by tables, which don’t have finalizers, the garbage collector won’t

help us. Hence, the solution would have to keep a list of used connections to

later proceed with a commit or a rollback, according to the execution status of the attendant.

Discussion

Our implementation is now complete, although it can receive some additions.

The test case should be much better developed but as the goal here was to show

how to do it, I left this task to be done as an exercise.

I think the main functions should check their arguments’ types whenever

possible, using features such as the luaL check* set of functions in the Lua

auxiliary library.

However, these C functions are not exported to Lua and

implementing them in Lua can cause a significant performance penalty. In fact,

this could be the subject for another gem.

A last but not less important point regards my decisions on the API style

and its organization. I chose the arguments of the SQL constructors guided

by our own usage and, I have to confess, changed them a little while writing

this document. Take the result set iterator, for instance: it could receive five

lengthy arguments which can make the call difficult to understand. To reduce

this problem, the function could have been implemented to receive a table with

the arguments in particular fields, as the following example:

db:select{

columns = "col1, col2, col3, col4",

from = "tablename t inner join othertable o",

having = "t.fk = o.id and t.col3 > 10",

groupby = "...",

}

The drawbacks of this approach are the growth of the library size and the

possibility of having to deal with differences between the accepted SQL syntax

of the databases or even limit the use of particular extensions.

The functions I grouped as “infrastructure” (escape, quote, twolists and

pairslist) could be generalized and stored in pre-existing packages, such as

55

string and table. Additionally, I packaged all SQL constructors into another

file which helped reuse the select constructor to build sub-selects. I do not think there is any canonical way to decide whether to put a function in a new module

or inside a pre-existing one.

Conclusion

To illustrate the point we have made, let us rewrite the first example in the

article using the tools developed:

-- Initialization (1)

local database = require"database"

local db = database.connect("lpg", "tomas", nil, "postgres")

-- Building SQL statement (2)

local _, course_list = db.twolists{ "Music", "Literature", }

local year = 2007

local cond = string.format([[c.name in (%s) and adm_year = %s]],

course_list, year)

local tab = "alumn a inner join course c on (a.course_id = c.id)"

-- SQL execution and error handling (3) and Iteration loop (4)

for id, name in db:select("a.id, a.name", tab, cond) do

print(id, name)

end

There is a huge difference from the previous version to this one. The former

explicitly checked for errors, while in the new one, this is performed automati-

cally by the library functions. The iteration loop is now a concise for-construct

without repeated calls to the fetch method. In addition, the SQL statement

construction is now much better supported, which helps build correct and more

legible code in a convenient way.

Finally, this library settles a new ground over which other abstractions could

be defined. Some applications are already constructed on top of it and so are

other libraries. An example is a module that provides facilities to the definition

of classes and objects directly associated with database tables and rows. It takes

advantage of the homogeneity of the API (insert and update methods) and also

of the SQL statements creation (table of fields becomes a where clause).

6

Boostrapping a Forth in

40 Lines of Lua Code

Eduardo Ochs

The core of a conventional Forth system is composed of two main programs:

an outer interpreter, which interprets textual scripts, and an inner interpreter,

which runs bytecodes. The outer interpreter switches between an “immediate

mode”, where words as executed as soon as they are read, and a “compile mode”,

where the words being read are assembled into bytecodes to define new words.

In Forth all variables are accessible from all parts of the system. Several

important words use that to affect the parsing: they read parts of the input text

themselves, process that somehow, and advance the input pointer — and with

that they effectively implement other languages, with arbitrary syntax, on top

of the basic language of the outer interpreter.

Due mostly to cultural reasons, Forths tend to be built starting from very low-

level pieces: first the inner interpreter, in Assembly or C, then the basic libraries

and the outer interpreter, in Forth bytecodes or (rarely) in C. We take another

approach. If we consider that Lua is more accessible to us than C or Assembly —

and thus for us Lua is “more basic” — then it is more natural to start from the

outer interpreter, and the dictionary only has to have the definition for one word,

one that means “interpret everything that follows, up to a given delimiter, as Lua

code, and execute that”. An outer interpreter like that fits in less than 40 lines

of Lua code, and it can be used to bootstrap a whole Forth-like language.

58

6 · Boostrapping a Forth in 40 Lines of Lua Code

Introduction

The real point of this article is to propose a certain way of implementing a Forth

virtual machine; let’s call this new way “mode-based”. The main loop of a mode-

based Forth is just this:

while mode ~= "stop" do modes[mode]() end

In our mode-based Forth, which is implemented in Lua and that we will refer

to as “miniforth”, new modes can be added dynamically very easily. We will

start with a virtual machine that “knows” only one mode — “interpret”, which

corresponds to less than half of the “outer interpreter” of traditional Forths —

and with a dictionary that initially contains just one word, which means “read

the rest of the line and interpret that as Lua code”. That minimal virtual

machine fits in 40 lines of Lua, and is enough to bootstrap the whole system.

But, “Why Forth?”, the reader will ask. “Forth is old and weird, why shouldn’t

we stick to modern civilized languages, and ignore Forth? What do you still

like in Forth?”. My feeling here is that Forth is one of the two quintessential

extensible languages, the other one being Lisp. Lisp is very easy to extend and

to modify, but only within certain limits: its syntax, given by ‘read’, is hard to

change(1). If we want to implement a little language (as in [1]) with a free-from

syntax on top of Lisp, and we know Forth, we might wonder that perhaps the

right tool for that would have to have characteristics from both Lisp and Forth.

And this is where Lua comes in — as a base language for building extensible

languages.

Disclaimer: I’m using the term “Forth” in a loose sense throughout this article.

I will say more about this in the last section.

Forth via examples

Any “normal” Forth has an interactive interface where the user types a line,

then hits the “return” key, and then the Forth executes that line, word by word,

and displays some output; our miniforth does not have an interactive interface,

but most ideas still carry on. Here’s a very simple program; the text on the left of

‘-->’ is the user input, the text on the right is the output from the Forth system.

Note that “words” are sequences on non-whitespace characters, delimited by

whitespace.

5 DUP * .

-->

25 ok

This program can be “read aloud” as this: “Put 5 on the stack; run ‘DUP’, i.e.,

duplicate the element on the top of the stack; multiply the two elements on the

top of the stack, replacing them by their product; print the element at the top of

the stack and remove it from the stack.”

Here’s a program that defines two new functions (“words”, in the Forth

jargon):

59

Figure 1. A 16-bit Forth with primitives. Forth instructions with very high values are primitives.

: SQUARE DUP * ;

-->

ok

: CUBE DUP SQUARE * ;

-->

ok

5 CUBE .

-->

125 ok

It can be read aloud as this: Define two new words: SQUARE: run DUP, then

multiply; CUBE: run DUP, then run SQUARE, then multiply. Now put 5 on the stack,

CUBE it, and print the result.

The words SQUARE and CUBE are represented in the memory as some kind of

bytecode; different Forths use different kinds of bytecodes. Here we are more

interested in “indirect threaded” Forths (see [3]) that store the dictionary in a

separate region of the memory. Some possible representations would be like

in Figures 1, 2, and 3; in these box diagrams all numbers are in hexadecimal,

and we are assuming a big-endian machine for simplicity. Figure 4 shows the

“bytecode” representation that we will use in miniforth. It is not exactly a

bytecode, as the memory cells can hold arbitrary Lua objects, not just bytes,

but we will call it a “bytecode” anyway, by abuse of language.

Here’s a trace of what happens when we run CUBE in miniforth:

RS={ 5 }

mode=head

DS={ 5 }

head="DOCOL"

RS={ 7 }

mode=forth

DS={ 5 }

instr="DUP"

RS={ 8 }

mode=forth

DS={ 5 5 }

instr=1

RS={ 8 1 }

mode=head

DS={ 5 5 }

head="DOCOL"

RS={ 8 3 }

mode=forth

DS={ 5 5 }

instr="DUP"

RS={ 8 4 }

mode=forth

DS={ 5 5 5 }

instr="*"

RS={ 8 5 }

mode=forth

DS={ 5 25 }

instr="EXIT"

RS={ 9 }

mode=forth

DS={ 5 25 }

instr="*"

RS={ 10 }

mode=forth

DS={ 125 }

instr="EXIT"

Note that we don’t have a separate variable for the instruction pointer (IP); we

use the top of the return stack (RS) as IP. The rightmost part of our traces always

describes what is going to be executed, while the rest describes the current state.

So, in the sixth line in the trace above we have RS = { 8, 4 }, and we are going

to execute the instruction in memory[4], i.e., "*", in mode “forth”.

60

6 · Boostrapping a Forth in 40 Lines of Lua Code

Figure 2. A 16-bit Forth with no primitives. All Forth instructions point to heads

(double boxes); each head points to a routine in 8086 machine code.

Figure 3. An imaginary 16-bit Forth with 1-byte heads and variable-length Forth

instructions.

Figure 4. Miniforth. Heads and Forth primitives are represented by strings in the

memory cells. Forth non-primitives are represented by numbers.

61

Bootstrapping miniforth

The program in Listing 1 is all that we need to bootstrap miniforth. It defines

the main loop (run), one mode (interpret), the dictionary (_F), and one word in

the dictionary: %L, meaning “evaluate the rest of the current line as Lua code”.

The program below is a first program in miniforth. It starts with only "%L"

defined and it defines several new words: what to do on end-of-line, on end-of-

text, and "[L", which evaluates blocks of Lua code that may span more than one

line; then it creates a data stack DS and defines the words "DUP", "*", "5", and

".", which operate on it.

subj = [=[

%L _F["\n"] = function () end

%L _F[""]

= function () mode = "stop" end

%L _F["[L"] = function () eval(parsebypattern("^(.-)%sL]()")) end

[L

DS = { n = 0 }

push = function (stack, x)

stack.n = stack.n + 1; stack[stack.n] = x end

pop

= function (stack)

local x = stack[stack.n]; stack[stack.n] = nil;

stack.n = stack.n - 1; return x end

_F["5"]

= function () push(DS, 5) end

_F["DUP"] = function () push(DS, DS[DS.n]) end

_F["*"]

= function () push(DS, pop(DS) * pop(DS)) end

_F["."]

= function () io.write(" "..pop(DS)) end

L]

]=]

-- Now run it. There’s no visible output.

pos = 1

mode = "interpret"

run()

-- At this point the dictionary (_F) has eight words.

After running this program the system is already powerful enough to run

simple Forth programs like, for example,

5 DUP * .

Note that to “run” this Forth program what we need to do is:

subj = "5 DUP * ."; pos = 1; mode = "interpret"; run()

It is as if we were setting the memory (here the subj) and the registers of a

primitive machine by hand, and then pressing its “run” button. Clearly, that

interface could be made better, but here we have other priorities.

62

6 · Boostrapping a Forth in 40 Lines of Lua Code

-- Global variables that hold the input:

subj = "5 DUP * ."

-- what we are interpreting (example)

pos

= 1

-- where we are (1 = "at the beginning")

-- Low-level functions to read things from "pos" and advance "pos".

-- Note: the "pat" argument in "parsebypattern" is a pattern with

-- one "real" capture and then an empty capture.

parsebypattern = function (pat)

local capture, newpos = string.match(subj, pat, pos)

if newpos then pos = newpos; return capture end

end

parsespaces = function () return parsebypattern("^([ \t]*)()") end

parseword = function () return parsebypattern("^([^ \t\n]+)()") end

parsenewline = function () return parsebypattern("^(\n)()") end

parserestofline = function () return parsebypattern("^([^\n]*)()") end

parsewordornewline = function () return parseword() or parsenewline() end

-- A "word" is a sequence of one or more non-whitespace characters.

-- The outer interpreter reads one word at a time and executes it.

-- Note that ‘getwordornewline() or ""’ returns a word, or a newline, or "".

getword = function () parsespaces(); return parseword() end

getwordornewline = function () parsespaces(); return parsewordornewline() end

-- The dictionary. Entries whose values are functions are primitives.

_F = {}

_F["%L"] = function () eval(parserestofline()) end

-- The "processor". It can be in any of several "modes".

-- Its initial behavior is to run modes[mode]() - i.e.,

-- modes.interpret() - until ‘mode’ becomes "stop".

mode

= "interpret"

modes = {}

run = function () while mode ~= "stop" do modes[mode]() end end

-- Initially the processor knows only this mode, "interpret"...

-- Note that "word" is a global variable.

interpretprimitive = function ()

if type(_F[word]) == "function" then _F[word](); return true end

end

interpretnonprimitive = function () return false end

-- stub

interpretnumber

= function () return false end

-- stub

p_s_i = function () end

-- print state, for "interpret" (stub)

modes.interpret = function ()

word = getwordornewline() or ""

p_s_i()

local _ = interpretprimitive() or interpretnonprimitive() or

interpretnumber() or error("Can’t interpret: "..word)

end

Listing 1.

63

The programs above don’t have support for non-primitives; this will have to

be added later. Look at Figure 4: non-primitives, like ”SQUARE”, are represented

in the bytecode as numbers (addresses of heads in the memory[]) and we have

not introduced either the memory or the states “head” or “forth” yet.

Note that the names of non-primitives do not appear in the memory, only

in the dictionary, _F. For convenience in such memory diagrams we will draw

the names of non-primitives below their corresponding heads. For instance, in

Figure 4, we have _F["SQUARE"] = 1 and _F["CUBE"] = 5.

Modes

When the inner interpret runs — i.e., when the mode is “head” or “forth”; see

Figure 5 — , at each step the processor reads the contents of the memory at IP

and processes it. When the outer interpreter runs, at each step it reads a word

from subj starting at pos, and processes it. There’s a parallel between these

behaviors. . .

I have never seen any references to “modes” in the literature about Forth.

In the usual descriptions of inner interpreters for Forth, the “head” mode is not

something separate; it is just a transitory state that is part of the semantics of

executing a Forth word. Also, the “interpret” and “compile” modes do not exist:

the outer interpreter is implemented as a Forth word containing a loop; it reads

one word at a time, and depending on the value of a state variable, it either

“interprets” or “compiles” that word. So, in a sense, “interpret” and “compile”

are “virtual modes”. . .

Let me explain how I arrived at this idea of “modes” — and what I was trying

to do that led me there.

Some words interfere with the variables of the outer interpreter. For ex-

ample, ":" reads the word the pos is pointing at (for example, SQUARE), adds a

definition for that word (SQUARE) to the dictionary, and advances pos. When the

control returns to modes.interpret(), the variable pos is pointing to the posi-

tion after SQUARE — modes.interpret() never tries to process the word SQUARE.

Obviously, this can be used to implement new languages, with arbitrary syntax,

on top of Forth.

Some words interfere with the variables of the inner interpreter — they mod-

ify the return stack. Let’s use a more colorful terminology: we will speak of

words that “eat text” and of words that “eat bytecode”. As we have seen, ":" is

a word that eats text; numerical literals are implemented in Forth code using a

word, LIT, that eats bytecode. In the program below,

: DOZENS 12 * ;

-->

ok

5 DOZENS . 60

-->

ok

the word DOZENS is represented in bytecode in miniforth as:

memory = {"DOCOL", "LIT", 12, "*", "EXIT"}

-- 1

2

3

4

5

-- DOZENS

64

6 · Boostrapping a Forth in 40 Lines of Lua Code

When the LIT in DOZENS executes, it reads the 12 that comes after it, and

places it on the data stack; then it changes the return stack so that in the next

step of the main loop the IP will be 4, not 3. Here is a trace of its execution; note

that there is a new mode, “lit”. The effect of “executing” the 12 in memory[3] in

mode “lit” is to put the 12 in DS.

RS={ 1 }

mode=head

DS={ 5 }

head="DOCOL"

RS={ 2 }

mode=forth

DS={ 5 }

instr="LIT"

RS={ 3 }

mode=lit

DS={ 5 }

data=12

RS={ 4 }

mode=forth

DS={ 5 12 }

instr="*"

RS={ 5 }

mode=forth

DS={ 60 }

instr="EXIT"

The code in Lua for the primitive LIT and for the mode “lit” can be synthe-

sized from the trace. By analyzing what happens between steps 2 and 3, and 3

and 4, we see that LIT and “lit” must be:

_F["LIT"] = function () mode = "lit" end

modes.lit = function ()

push(DS, memory[RS[RS.n]])

RS[RS.n] = RS[RS.n] + 1

mode = "forth"

end

so from this point on we will consider that the traces give enough information,

and we will not show the corresponding code.

Note that different modes read what they will execute from different places:

“head”, “forth”, and “lit” read from memory[RS[RS.n]] (they eat bytecode),

whereas “interpret” and “compile” read from subj, starting at pos (they eat text).

Our focus here will be on modes and words that eat bytecode.

Virtual modes

How can we create words that eat bytecode, like LIT, in Forth? In the program

below, the word TESTLITS call first LIT, then VLIT; VLIT should behave similarly

to LIT, but LIT is a primitive and VLIT is not.

memory = {"DOCOL", "R>P", "PCELL", "P>R", "EXIT",

--

1

2

3

4

5

--

VLIT <---------------+

--

|

"DOCOL", "LIT", 123, 1, 234, "EXIT",}

--

6

7

8

9

10

11

--

TESTLITS

65

Here is a trace of TESTLITS:

t=0

RS={ 6 }

mode=head

PS={

}

DS={

}

head="DOCOL"

t=1

RS={ 7 }

mode=forth

PS={

}

DS={

}

instr="LIT"

t=2

RS={ 8 }

mode=lit

PS={

}

DS={

}

data=123

t=3

RS={ 9 }

mode=forth

PS={

}

DS={ 123 }

instr=1

t=4

RS={ 10 1 }

mode=head

PS={

}

DS={ 123 }

head="DOCOL"

t=5

RS={ 10 2 }

mode=forth

PS={

}

DS={ 123 }

instr="R>P"

t=6

RS={ 3 }

mode=forth

PS={ 10 }

DS={ 123 }

instr="PCELL"

t=7

RS={ 4 }

mode=pcell

PS={ 10 }

DS={ 123 }

pdata=234

t=8

RS={ 4 }

mode=forth

PS={ 11 }

DS={ 123 234 }

instr="P>R"

t=9

RS={ 11 5 }

mode=forth

PS={

}

DS={ 123 234 }

instr="EXIT"

t=10 RS={ 11 }

mode=forth

PS={

}

DS={ 123 234 }

instr="EXIT"

This is a full solution, so start by ignoring the cells 2, 3, and 4 of the memory,

and the lines t=5 to t=8 of the trace. From t=5 to t=9 what we need to do is

push(DS, memory[RS[RS.n - 1]])

RS[RS.n - 1] = RS[RS.n - 1] + 1

where the –1 is a magic number: roughly, the number of ”call frames” in the

stack between the call to VLIT and the code that will read its literal data,

negated. In other situations this could be –2, –3, . . . One way to get rid of that

magic number is to create a new stack — the “parsing stack” (PS) — and to have

“parsing words” that parse bytecode from the position that the top of PS points

to; then a word like VLIT becomes a variation of a word, PCELL, that reads a cell

from memory[PS[PS.n]] and advances PS[PS.n]. The code for VLIT given above

shows how that is done — we wrap PCELL as "R>P PCELL P>R" — and from the

trace we can infer how to define these words.

Note that the transition from t=2 to t=3 corresponds to the transition from

t=4 to t=10; the mode being “lit” corresponds to having the address of the head

of VLIT at the top of RS, and the mode being “head”; using this idea we can

implement virtual modes in Forth. Better yet: it all becomes a bit simpler if we

regard the mode as being an invisible element that is always above the top of

RS. So, an imaginary mode “vlit” would be translated, or expanded, into a 1 (the

head of VLIT), plus a mode “head”; or another word, similar to VLIT, would just

switch the mode to “vlit”, and the action of that word would be to expand it into

the head of VLIT, plus the mode “head”.

A bytecode for polynomials

A polynomial with fixed numerical coefficients can be represented in memory as

first the number of these coefficients, then the value of each of them; for example,

P (x) = 2x3 + 3x2 + 4x + 5.5 is represented as {...,

4, 2, 3, 4, 5.5, ...}.

We will call this representation — number of coefficients, then coefficients — the

“data of the polynomial”. Let’s start with a primitive, PPOLY, that works like

66

6 · Boostrapping a Forth in 40 Lines of Lua Code

PCELL, in the sense that it reads the data of the polynomial from the memory,

starting at the position PS[PS.n], and advancing PS[PS.n] at each step. This

PPOLY takes a value from the top of the data stack — it will be 10 in our

examples — and replaces it with the result of applying P on it, —P(10)—, which

is 2345.5 for the example above.

By defining POLY from PPOLY, as we defined VLIT from PCELL

: POLY R>P PPOLY P>R ;

we get a word that eats bytecode; a call to POLY should be followed by data of a

polynomial, just like LIT is followed by a number. And we can also do something

else: we can create new heads, DOPOLY and DOADDR, and represent polynomials as

two heads followed by the data of the polynomial. The program and trace below

test this idea.

memory = {"DOPOLY", "DOADDR", 4, 2, 3, 4, 5.5,

--

1

2

3

4

5

6

7

--

P(X)

&P(X)

--

^---------------------+

--

|

"DOCOL",

"LIT", 10, 1, "EXIT"}

--

8

9

10

11 12

--

TESTDOPOLY: put 10 on the stack and call P(X)

RS={ 8 }

mode=head

PS={

}

DS={

}

head="DOCOL"

RS={ 9 }

mode=forth

PS={

}

DS={

}

instr="LIT"

RS={ 10 }

mode=lit

PS={

}

DS={

}

data=10

RS={ 11 }

mode=forth

PS={

}

DS={ 10 }

instr=1

RS={ 12 1 }

mode=head

PS={

}

DS={ 10 }

head="DOPOLY"

RS={ 12 forth }

mode=ppolyn

PS={ 3 }

DS={ 10 }

n=4

RS={ 12 forth }

mode=ppolyc

PS={ 4 }

DS={ 10 }

n=4 acc=0 coef=2

RS={ 12 forth }

mode=ppolyc

PS={ 5 }

DS={ 10 }

n=3 acc=2 coef=3

RS={ 12 forth }

mode=ppolyc

PS={ 6 }

DS={ 10 }

n=2 acc=23 coef=4

RS={ 12 forth }

mode=ppolyc

PS={ 7 }

DS={ 10 }

n=1 acc=234 coef=5.5

RS={ 12 forth }

mode=ppolye

PS={ 8 }

DS={ 10 }

acc=2345.5

RS={ 12 }

mode=forth

PS={ 8 }

DS={ 2345.5 }

instr="EXIT"

The trace above does not show what &P(X) does; the effect of running &P(X) is

to put the address of the beginning of data of the polynomial, namely, 3, into the

data stack. Note how a polynomial — which in most other languages would be a

piece of passive data — in Forth is represented as two programs, P(X) and &P(X),

that share their data. Compare that with the situation of closures in Lua — two

closures created by the same mother function, and referring to variables that

were local to that mother function, share upvalues.

67

A bytecode language for propositional calculus

Here is another example. Let’s write ‘=>’ for “implies”, and ‘&’ for “and”. Then

(Q=>R)=>((P&Q)=>(P&R)) is a “formula”, or a “proposition”, in Propositional Calculus; incidentally, it is a tautology, i.e., always true.

In some situations, for example, if we want to find a proof for that proposition,

or if we want to evaluate its truth value for some assignment of truth values to

P, Q, and R, we need to refer to subformulas of that formula. If we represent the

formula in bytecode using Polish Notation (not Reverse Polish Notation! Can

you see why?) then this becomes trivial:

memory = { "=>", "=>", "Q", "R", "=>", "&", "P", "Q", "&", "P", "R" }

--

1

2

3

4

5

6

7

8

9

10

11

Subformulas can now be referred to by numbers: the position in the memory

where they start. We can write a word to parse a proposition starting at some

position in the memory; if that position contains a binary connective like ‘=>’ or

‘&’, then that word calls itself twice to parse the subformulas at the “left” and

at the “right” of the connective. If the word memoizes the resulting structure

by storing it in a table named formulas, then re-parsing the formula that starts

at the position, say, 6, becomes very quick: the result is formulas[6], and the

pointer should be advanced to formulas[6].next. Here are the contents of that

table after parsing the formula that starts at memory[1].

1: { addr=1, cc="=>", l=2,

r=5,

next=12, name="((Q=>R)=>((P&Q)=>(P&R)))" }

2: { addr=2, cc="=>", l=3,

r=4,

next=5,

name="(Q=>R)" }

3: { addr=3,

next=4,

name="Q" }

4: { addr=4,

next=5,

name="R" }

5: { addr=5, cc="=>", l=6,

r=9,

next=12, name="((P&Q)=>(P&R))" }

6: { addr=6, cc="&",

l=7,

r=8,

next=9,

name="(P&Q)" }

7: { addr=7,

next=8,

name="P" }

8: { addr=8,

next=9,

name="Q" }

9: { addr=9, cc="&",

l=10, r=11, next=12, name="(P&R)" }

10: { addr=10,

next=11, name="P" }

11: { addr=11,

next=12, name="R" }

(Meta)Lua on miniforth

The parser for the language for Propositional Calculus in the last section had to

be recursive, but it didn’t need backtracking to work. Here is a language that

is evidently useful — even if at this context it looks like an academic exercise —

and whose parser needs a bit of backtracking, or at least lookahead. Consider

the following program in Lua:

foo = function ()

local storage

return function () return storage end,

function (x)

storage = x end

end

68

6 · Boostrapping a Forth in 40 Lines of Lua Code

It can be represented in bytecode in miniforth as:

memory = {

"foo", "=", "function", "(", ")",

"local", "storage",

"return", "function", "(", ")", "return", "storage", "end", ",",

"function", "(", "x", ")", "storage", "=", "x", "end",

"end",

"<eof>" }

One way of “executing” this bytecode made of string tokens could be to pro-

duce in another region of the memory a representation in Lua of the bytecode

language that the Lua VM executes; another would be to convert that to an-

other sequence of string tokens — like what MetaLua [5] does. Anyway, there’s

nothing special with our choice of Lua here — Lua just happens to be a simple

language that we can suppose that the reader knows well, but it could have been

any language. And as these parsers and transformers would be written in Lua,

they would be easy to modify.

Why Forth?

Caveat lector: there is no single definition for what “Forth” is. . . Around 1994

the community had a big split, with some people working to create an ANSI

Standard for Forth, and the creator of the language and some other people going

in another direction, and not only creating new Forths that went against ideas

of the Standard, but also stating that ANS Forth “was not Forth”. I can only

write this section clearly and make it brief if I choose a very biased terminology;

also, I’m not going to be historically precise, either — I will simplify and distort

the story a bit to get my points across. You have been warned!

Forth was very popular in certain circles at a time when computers were

much less powerful than those of today. Some of the reasons for that popularity

were easy to quantify: compactness of programs, speed, proximity to machine

code, simplicity of the core of the language, i.e., of the inner and the outer

interpreters. None of these things matter so much anymore: computers got

bigger and faster, their assembly languages became much more complex, and

we’ve learned to take for granted several concepts and facilities — malloc and

free, high-level data structures, BNF — and now we feel that it is “simpler” to

send characters through stdout than poking bytes at the video memory. Our

notion of simplicity has changed.

In the mid-90s came the ANS-Forth Standard, and with it a way to write

Forth source that would run without changes in Forths with different memory

models, on different CPU architectures. At about the same time the creator

of the language, Chuck Moore, started to distance himself from the rest of the

community, to work on Forths that were more and more minimalistic, and on

specialized processors that ran Forth natively.

69

My experience with (non-Chuck-Moore-) Forth systems written before and

after the ANS Standard was that in the pre-ANS ones the format of the bytecode

was stated clearly, and users were expected to understand it; in Forths written

after the Standard the bytecode was not something so mundane anymore — it

became a technical detail, hidden under abstractions.

Old Forths were fun to use. When I was a teenager I spent hundreds of

evenings playing with Forths on an IBM-PC: first FIG-Forth and MVP-Forth,

then HS-Forth, a commercial Forth whose memory model (8086 machine code,

dictionary and Forth definitions in different memory segments, indirect-threaded,

no primitives, multiple heads) inspired some of the ideas in this article. At one

point I spent some weeks writing a program that constructed a “shadow im-

age” of the Forth segment, with a letter or a number for each byte in a head,

a ‘.’ for each byte in a Forth instruction, ‘_’ and ‘$’ for bytes in literal numbers

and strings, ‘<’ and ‘>’ for the bytes that were addresses in backward or forward

jumps (i.e., the two bytes following each call to BRANCH or 0BRANCH) — and

spaces for the unknown bytes, as I didn’t have the source for the whole core sys-

tem, and some words were tricky to decompile. . . Then I printed the result, in

five pages, each with a grid of 64x64 characters, and addresses at the borders;

that gave me a map of all the bytes in words in the core system that were not

defined in assembly language.

I’ve met many people over the years who have been Forth enthusiasts in the

past, and we often end up discussing what made Forth so thrilling to use at

that time — and what we can do to adapt its ideas to the computers of today.

My personal impression is that Forth’s main points were not the ones that I

listed at the beginning of this section, and that I said that were easy to quantify;

rather, what was most important was that nothing was hidden, there were no

complex data structures around with “don’t-look-at-this” parts (think on garbage

collection in Lua, for example, and Lua’s tables — beginners need to be convinced

to see these things abstractly, as the concrete details of the implementation

are hard), and everything — code, data, dictionaries, stacks — were just linear

sequences of bytes, that could be read and modified directly if we wished to. We

had total freedom, defining new words was quick, and experiments were quick to

make; that gave us a sense of power that was totally different from, say, the one

that a Python user feels today because he has huge libraries at his fingertips.

A Forth-like language built on top of Lua should be easier to integrate with

the rest of the system than a “real” Forth written in C. Also, it’s much easier

to add new syntaxes and bytecode languages to a mode-based Forth than to a

conventional one. And we are not forced to store only numbers in the memory;

we can store also strings — I’ve used strings for primitives and heads here

because this makes programs more readable —, or any other Lua objects, if we

need to.

I do not intend to claim that miniforth is compact — in terms of memory

usage — or efficient, or useful for practical applications. But the natural ways

for doing things in Forth were different from the ways that are natural in today’s

systems; and I believe that miniforth can be used to give to people glimpses into

70

6 · Boostrapping a Forth in 40 Lines of Lua Code

interesting ways of thinking that have practically disappeared, and that have

become hard to communicate.

Conclusion

After a draft of this article had been written, Marc Simpson engaged in a long

series of discussions with me about Forths, Lisp, SmallTalk, several approaches

to minimality, etc., and at one point, over the course of one hectic weekend

in December, 2007, he implemented a usable (rather than just experimental)

dialect of Forth — based mainly on Frank Sergeant’s Pygmy Forth and Chuck

Moore’s cmForth, and borrowing some ideas from this article — on top of Ruby

(“RubyForth”), and later ported his system to Python and C. A port of it to Lua

is underway.

I thank Marc Simpson and Yuri Takhteyev for helpful discussions.

References

[1] Jon Bentley: More Programming Pearls, Addison-Wesley, 1990 (chapter 9:

Little Languages).

[2] James T. Callahan: HS-Forth (program and manual). Harvard Softworks,

1986–1993.

[3] Anton Ertl: Threaded Code. http://www.complang.tuwien.ac.at/forth/

threaded-code.html

[4] Brad Rodriguez: A BNF Parser in Forth. http://www.zetetics.com/bj/

papers/bnfparse.htm

[5] Fabien Fleutot: MetaLua. http://metalua.luaforge.net/

[6] Kein-Hong Man: A No-Frills Introduction to Lua 5.1 VM Instructions.

http://luaforge.net/docman/view.php/83/98

7

Effecting Large-Scale Change

(with little trauma)

using Metatables

S érgio Alvares Maffra & Pedro Miller Rabinovitch

Introduction

In real-world conditions, software maintenance becomes as important as soft-

ware development. Environments change, business partners choose different

strategies, technology evolves — and devolves. Distributed systems become cen-

tralized, and centralized ones get scattered around. In particular, requirements

have a peculiar way of being significantly altered once a project approaches com-

pletion. . . or the day after it has been deployed.

It becomes more and more important to be able to quickly adapt to these

changing conditions. In the following sections we’ll discuss the application of a

particularly powerful feature of Lua to this end. We’ll start by presenting a short

review of metamethods along with a couple of simple examples. We will then

show how Lua’s metatables were used to dramatically change the performance

profile of an application with little effort. Finally, we conclude showing a few

examples where metatables can help developers change the key features of a

system even if it’s already in a late development phase or even in production.

72

7 · Effecting Large-Scale Change (with little trauma) using Metatables

Metamethods and environments

Metatables offer an unique way to intercept and change language events in

specific objects. By specifying code to be executed when simple operations are

attempted on specific objects, developers can provide a scripting environment

where values behave the way they’re expected to, which is particularly useful

where non-programmers are concerned (e.g., AI scripting or other problem do-

main specific scripting). However, much more interesting applications are avail-

able. The index and newindex metamethods, in particular, are specially useful

since they allow one to check for variable resolution and provide missing values

on the fly in a just-in-time fashion. This, of course, is perfect for default values

and inheritance (as described in detail in Programming in Lua) and lazy evalu-

ation procedures, where values are computed as required, but always accessed

in a transparent and consistent manner.

The index metamethod is called every time a table is indexed and there’s no

appropriate value set for the relevant key. Listing 1 shows an example of index

being used to calculate values on the fly as the cache (or look-up) table is indexed.

The first time the user attempts to index the table where there’s no value set,

the metamethod is called. After the computing is done, the metamethod sets the

value at the table, memoizing it for the next time it’s needed. When the same

key of the table is indexed again further down the line, the value is already

there, and the method is not called.

The newindex metamethod works in an analogue fashion — it is called every

time the table has a new value being set. This means that we can watch a table

for change and record every time one of its values is updated by an operation, as

long as we keep the relevant fields of the table empty (typically by using a proxy

table).

The mechanisms above become even more useful because of Lua’s environ-

ment functions, getfenv and setfenv. These functions allow us to change the

global environment of any given executing routine. The call getfenv(0), in par-

ticular, returns a reference to the table that contains all global variables in our

program at the current context, and which will be inherited when we require

other modules. Since it’s a table, we can set metamethods on it, and watch as

global variables are accessed and updated should we so desire.

Our sample problem

Consider an application heavily based in Lua. As development progresses, the

number of Lua files in the project increases, and the dependency graph grows

more and more complex at each iteration. At a certain point, the application

starts taking a long time to compile at run time, since all the modules are loaded

right at the beginning of the execution in a long series of requires or dofiles.

This series of module loading instructions also presume a certain order, since

dependency issues must not be ignored. At first, this is taken in stride — the

target equipment that will run the application is well-known and established as

73

local tableOfHeavyData = {}

setmetatable( tableOfHeavyData,

{

__index = function( tbl, key )

-- calculate the required value

local data = performHeavyComputing( key )

-- cache response in the table

tbl[key] = data

return data

end

}

)

-- do serious computing

function performHeavyComputing( x )

print( "computing the value of "..x )

return x * x -- dude. Heavy.

end

print( "value for 2 is "..tableOfHeavyData[2] )

for i = 1, 3 do

print( "value for "..i.." is "..tableOfHeavyData[i] )

end

--[[ Output:

computing the value of 2

value for 2 is 4

computing the value of 1

value for 1 is 1

value for 2 is 4

computing the value of 3

value for 3 is 9

--]]

Listing 1. An example of the index metamethod.

74

7 · Effecting Large-Scale Change (with little trauma) using Metatables

being of “high performance”. Lines such as “we’ll just tell them to buy more

RAM” are heard and management is confident of the project success. Boat

catalogs are browsed.

This is all well and good until requirements change. Perhaps the code will

have to run on a less powerful platform, such as portable devices. Perhaps the

client can’t afford the extra budget for better equipment. Or perhaps the testing

cycle just got way too long, since each time the application is run, everything is

loaded into memory in one big shot.

Now we have a problem. Our hypothetical application is bloated; its modules

and libraries are not well separated; the libraries it depends on are taking

much more cycles than expected; what is one to do? Picture modules named

pic.lua that define functions with naming conventions as diverse as pic open,

PICdecode, and pngPICformat.

Add a couple of list of pictures or images

tables. Multiply that by, say, 30 or 40 functions defined in each of 20+ modules.

Throwing a couple of interns at the problem probably won’t give the best results.

We had such a problem in an actual application we developed — in our case,

the graphical interface library which the application depended on went through

a large change and started taking a lot longer to create dialogs. The change

was for the better as far as the GUI presentation was concerned, of course.

But running it, even if one was just trying to check on the latest changes, was

taking way too long. Granted, our application was not the nightmarish vision

we presented above, but we’re making a point here.

Well, if your development is in a language as powerful as Lua, you can solve

your problems1 in about an hour with the judicious use of metamethods and the

replacement of a couple of system functions. We will present a solution as we

analyze a sample implementation in the following section.

Resolution through “Origins”

The new release of the GUI library used in our application was expected with

great anxiety at the time. We had been eagerly waiting for it for about a year.

We couldn’t wait to use the new features (and get rid of the old bugs!). Unfortu-

nately, what should have been a joyful occasion, became a great disappointment.

Moreover, we had a big problem in our hands. Sooner or later, we would have to

upgrade to the latest release, since there was no support for the old ones.

A deeper investigation showed us that the GUI library was taking a lot longer

to create dialogs due to the new widget placement algorithm. In our application

all dialogs were created at startup, what explained the humongous loading time

we were experiencing. Fortunately, the solution was, actually, quite simple:

the dialogs should be individually created when needed and not all at once at

startup. Given that our application is composed of quite a few hundred source

files, this solution fits well in the “easier said than done” category. At first glance,

1Actually, you can solve a couple of your problems. The memory and processing issue will be solved, but the perverse naming conventions are more into the realm of physical punishment.

75

-- generated in 06/19/07 21:09:16 by origins

origins.data = {

["alpha_create"] = "libAlpha",

["gamma_function1"] = "libGamma",

["alpha_function2"] = "libAlpha",

["alpha_global"] = "libAlpha",

["alpha_function1"] = "libAlpha",

["beta_name"] = "libBeta",

["alpha_constant1"] = "libAlpha",

["beta_constant1"] = "libBeta",

["alpha_name"] = "libAlpha",

["beta_function1"] = "libBeta",

["beta_table"] = "libBeta",

["gamma_name"] = "libGamma",

["beta_create"] = "libBeta",

}

Listing 2. A sample data file

that would require us to track down and alter every reference made to each of

the dialogs used in the application. Thankfully, all GUI code was written in Lua,

which allowed us to adopt a better approach.

The loading time could be reduced if the require calls that created dialogs

were removed from our initialization methods. But, that would leave us with

a lot of missing values in our hands. As mentioned in Section 7, the index

metamethod can be used to provide missing values on the fly. Therefore, our

dialogs could be created when needed by loading their defining modules in an

index metamethod set in the global environment.

By using this solution we avoided going through all the code of the appli-

cation. It required, however, knowing the values defined by each module in

the application. A simple table containing module names that are retrieved

from variable names, like the one defined in Listing 2, is all that was necessary.

Granted, creating this table can still require a lot of work. Fortunately, the table

in Listing 2 was generated automatically by using the newindex metamethod.

We have implemented the solution in the form of a library we’ve dubbed

“Origins”. The library works in a two-phase approach:

1. During setup, the application is executed with the appropriate environ-

ment, and the library detects all functions and variables that are loaded,

tracing each (by name) to its original module;

2. During run time, the application is executed and, as it tries to access the

functions and variables from each module, the library detects the attempt

and loads the required module.

Each phase depends mainly in a different meta-method. We will examine each

one below.

76

7 · Effecting Large-Scale Change (with little trauma) using Metatables

function origins:startWatching()

origins.original_require = require

require = origins.new_require

setmetatable( getfenv(0), origins_metatable )

end

function origins:stopWatching()

setmetatable( getfenv(0), nil )

require = origins.original_require

end

Listing 3. Watching for definitions.

function origins.new_require( name, ... )

print( "[origins] processing "..name )

-- preserve previous filename (so we know where variables come from)

local previousFilename = origins.currentFilename

-- set current context

origins.currentFilename = name

-- call the original function

origins.original_require( name, unpack( arg ))

-- restore context

origins.currentFilename = previousFilename

end

Listing 4. Hooking the require function.

Setup

During the setup phase we use the newindex metamethod to establish which

module is providing each global function and variable. This is done by the

function startWatching presented in Listing 3.

First, we hook the loading

functions we’re interested in (require in this case; dofile or any other functions

that should be processed as well) by replacing their global reference with our

own versions. These work as illustrated in Listing 4, keeping track of the lua

file (and therefore library module) being currently processed. Our metatable

is set on the global environment (acquired via getfenv(0)).

The newindex

metamethod, shown in Listing 5, notes each global variable that is set and keeps

track of the lua file that was being processed at the time of its definition.

After these proceedings, we can load our application normally. As each new

global is set, our variable catalog is built, and by the time the application is up

and running — having loaded every variable we’re interested in — we can save

the stored catalog in a data file that will be used in run time. This is illustrated

in Listing 6, and a sample data file is represented in Listing 2. Notice that the

data file is simply a Lua code file and that the table data keeps an entry for each

variable with the path to its original loading module.

77

local origins_metatable = {

__newindex = function( table, key, value )

--print( "[origins] newindex: ", key )

origins.data[key] = origins.currentFilename

rawset( table, key, value )

end,

__index = function( table, key )

--print( "[origins]

index: ", key )

local source = origins.data[key]

if source then

origins.original_require( source )

end

return rawget( table, key )

end,

}

Listing 5. The newindex and index metamethods.

function origins:saveData( filename )

self:stopWatching()

filename = filename or self.defaultFileName

local fout = io.open( filename, ’wt’ )

fout:write( "-- generated in "..os.date().." by origins\n" )

fout:write( "origins.data = {\n" )

for i,v in pairs( self.data ) do

fout:write( string.format( "\t[%q] = %q,\n", i, v ))

end

fout:write( "}\n" )

fout:close()

end

Listing 6. Saving the catalog data.

Quick setup

It is true that one can execute the setup phase by running the application as

normal after calling startWatching. However, Listing 7 shows an alternative —

artificially loading each module used by the application in a single stretch. This

will be enough to refresh all necessary data in most cases, and a proper data

file can be generated. This has the additional benefit that we can easily call the

setup script in an automated build system, automatically updating the data file

as we change the modules.

Runtime

At run time, the application doesn’t load its libraries at startup as usual. In-

stead, it loads the “origins” data file through the loadData function, which is

78

7 · Effecting Large-Scale Change (with little trauma) using Metatables

-- prepare the application library list

require "origins"

origins:startWatching()

-- these should be in the order they’ll be required

require "libUtil"

require "libBeta"

require "libGamma"

require "libAlpha"

-- now save the data

origins:saveData()

Listing 7. Loading modules at setup time.

function origins:loadData( filename )

filename = filename or self.defaultFileName

print( "[origins] loading "..filename )

dofile( filename )

origins:startWatching()

end

Listing 8. Loading the catalog data.

depicted in Listing 8. This loads the reference catalog (through a trivial dofile)

and sets the runtime index metamethod, shown in Listing 5, which looks for a

reference in the catalog in order to load the required module.

After the loading is done, the value that was being sought should be avail-

able. The application is not even aware that a module was being loaded on

the fly, since it was waiting for the variable referencing to occur while the

metamethod was running.

Limitations

The library uses variable and function names as keys; therefore, conditional

file loading might cause problems if they define functions by the same name.

Consider the code in Listing 9. The execution path taken during setup time

would be the one “Origins” considers as the source of out write.

There is a way to pause processing, however (the stopWatching method). This

enables developers to circumvent said limitation. The use of a quick setup script

as described previously would also resolve the situation by not including either

module and only loading them at the original point in run time, as intended by

the code.

Even if conditional loading is not an issue, what should happen if a global

variable is set in more than one module?

Such a naming conflict does not

79

if mode == "latex" then

dofile( "out_latex.lua" )

else

dofile( "out_html.lua" )

end

-- out_write() is defined in both the files above

out_write( mydata )

Listing 9. Conditional dofiles could cause problems.

have a resolution we can deem as correct, since even in Lua, the name clash

would cause one of the values to be overwritten. We chose not to address this

issue in this implementation, but some alternatives would include loading all

the modules that defined the conflicting variable (perhaps in the order they

originally appeared), printing out warning messages, or even firing an error

during setup.

Further development

The system introduced here could be the basis for further development. Con-

sider a scheme where standard Lua libraries installed at predetermined path

locations were loaded at run time as code in execution needed them, without

preloading them through require. We could delay library module loading un-

til code is necessary by implementing “Origins” on a system-wide range. One

could get rid of requires by stipulating that all libraries should be installed

through a program responsible for managing a library function catalog, much

in the manner of a package manager. After that, any running applications that

tried to execute cataloged functions would have the appropriate module loaded

and ready at the first call attempt. Module dependencies, of course, would be

handled automatically.

Some work would have to be done to keep different versions working correctly

together when required. Packaging schemes with property files that describe

required module versions come to mind. But scripting, in particular, would

greatly benefit from such a scheme — specially if an automated download and

installation procedure was available at run time.

Other uses for metatables

Metatables can be used in many ways to significantly change system behavior

with little source code alteration. We present some examples below.

Mutexes for concurrent access. If we change an application from a single-threaded approach to a multi-threaded one, access to shared values becomes an issue. Fields in a table (or even global variables) could be watched by a basic

80

7 · Effecting Large-Scale Change (with little trauma) using Metatables

monitor paradigm or any kind of mutex solution. This would be easy to

implement with metamethods on the lookout for change and access.

Remote procedure calls. Suppose we want to move a component of a modular

system to an out-of-process location (and maybe to a remote location ac-

cessible by the network). We could use the call metamethod to automat-

ically serialize parameters and return values and handle socket commu-

nication transparently to the executing process. Of course, this has the

disadvantage of hiding important issues like timeouts and communica-

tion failures — but on the other hand, programming remains simple and

the whole system can be changed in one fell swoop without any perilous

search-and-replacing.

Transparent interprocess communication. Not only procedure calls can be sent

to other processes; one could implement a shared memory system using

sockets or any kind of shared memory access and employ metatables to

keep access transparent to the local developer.

Dynamic library loading. Instead of preloading all modules required by the sys-

tem, we could wait to load each of them as they become needed. This can

be done by watching the global environment for access, as shown in the

main section of this gem.

It’s important to note that these changes are easily made and undone. Since

no code is changed in the original application, just setting or not setting the ap-

propriate metatables will activate or deactivate the corresponding functionality.

This makes the approach perfect for experimentation, since alternative methods

can be kept along with working code in the same development branch.

Another way metatables can help us is on debugging or measuring a system’s

correctness or performance profile. This can be done in ways such as:

Tracing field access. One could use the index and newindex metamethods, paired

with other debugging information (such as those provided by the debug li-

brary) to trace which execution paths are altering and accessing a variable

or field.

Asserting variable values. Instead of relying on getter and setter functions, one could use appropriate metamethods to check that certain variables always

have valid values.

Profiling function calls. The same procedure described for tracing field access,

above, could be used to trace function calls. Although a hook could be used

to trace every function call in the system and then check for the function

names we’re interested in, it is probably faster to set a limited number of

metamethods. With appropriate timing information or even if we’re just

tracking the number and/or origin of calls made, one can obtain quite a lot

of interesting information with such a system in place.

81

Conclusion

We have presented in this gem a simple use of what we consider as one of the

most versatile and powerful features of Lua. It is easy to see how a simple

change implemented this way could alter the execution of a large system as

a whole, making a significant and perceptible difference on application perfor-

mance and usage. In our case it represented the difference between a bit of de-

velopment work and rewriting the entire GUI system in order to spare process-

ing and memory. By implementing the dynamic loading mechanism presented

here we prevented much unnecessary expenditures with little development cost.

The most important lesson to keep, however, is that metatables offer a way to

effect system-wide change with little use of search-and-replacing and other po-

tentially traumatic methods.

Part II

Design Techniques

8

MVC Web Development

with Kepler

Andr é Carregal and Yuri Takhteyev

Kepler is a Lua-based web development platform that is modular and flexible.

Kepler 1.1 adds support for the developing web applications using the increas-

ingly more popular Model-View-Controller (MVC) style. This article discusses

several approaches to MVC web development and shows how they can be imple-

mented in Kepler.

Introduction

Kepler1 is an open source web development platform based on Lua that brings

many of Lua’s advantages to the development of web applications.2 Like Lua,

Kepler is small, portable, and flexible. Kepler 1.1 provides support for web ap-

plication development that follows the Model-View-Controller (MVC) paradigm,

bringing to Lua some of the benefits provided by popular web frameworks writ-

ten for other programming languages.

MVC refers to the division of application code into three sections and was

originally brought to desktop GUI programming by Smalltalk. When applied to

1http://www.keplerproject.org/

2Since this article addresses readers already familiar with Lua, we do not discuss here the exact advantages that Lua offers in general or specifically for web development. Such advantages are discussed, however, on the Kepler site.

86

8 · MVC Web Development with Kepler

Web development, this metaphor has to be adjusted due to the different nature

of the interaction: the model handles data manipulation and storage, the view handles client-side interaction, and the controller responds to requests using the model for data manipulation and generating views that are used by the client.

Having the model clearly separated is an important part of MVC and the

success of such frameworks as Rails had much to do with their handling of

the model. The implementation of a model, however, is not specific to web

development and we will thus assume here a pre-existing model.

We will

similarly avoid the discussion of the view contents in much detail, since the

structure of the view is no way specific to Lua. Instead, we will focus on two

problems that must be resolved by the controller: dispatching incoming requests

(originating from a view) and generating a response that contains a view for the

next request.

Request dispatching

Overview

The general problem of HTTP request dispatching is to map an incoming request

to an action within the system. The action must at the minimum generate a

response that will be sent to the client, but may also have side effects, such as

altering the data stored in the model. Here is a general model of the way the

MVC controller handles an HTTP request:

For the purpose of this article we will consider a simplified version of this

model, which assumes that the controller relies only on the URL and uses

neither other parts of the request nor the internal state: We do not discuss the

use of HTTP headers and POST parameters, though those parts of the request

are already provided by Kepler in an simple way and their use presents no

87

conceptual challenges.

For the purpose of the this article we will use the term “URL” to refer only

to the local part of a URL (/<PATH>?<QUERY>), since the host domain and port

number should not concern the web application. As the server dispatches the

request, it consumes some part of PATH. The remaining part of PATH is passed to

the application and we will refer to it as PATH INFO.

A common approach to dynamic web development is to structure the ap-

plication as a collection of functions (in the broad sense of the word) and in-

terpret each request as an identifier of a function and a set of parameters to

this function. The traditional use of CGI scripts follows this model: each CGI

script acts as a function that accepts a set of parameters (encoded in the QUERY

part of the URL). An application is then structured as a collection of scripts,

each responsible for a different type of request. Under this approach, a re-

quest for wiki/show.cgi?p=HomePage is interpreted as a call to a specific function

(/wiki/show.cgi) which is called with one parameter (p=HomePage). If the user

clicks on a link to edit the page, they will generate a request for a different

function, such as /wiki/edit.cgi?p=HomePage.

Kepler supports this style of web development through “Lua Scripts” or “Lua

Pages”. This style, however, has been increasingly criticized in the recent years

for the insufficient separation between view generation and the application logic.

In this article we will show how to use Kepler to implement some of the different

alternative approaches that are currently used in some of the popular web

frameworks. In doing so, we want to show a range of options available to the

developer.

One way of making the dispatching more flexible is to change the way func-

tions and parameters are stored and represented. For instance, we can triv-

ially map /wiki/show onto a Lua function called show, passing to this function

the table representing the QUERY parameters. (This would be very similar to

the approach used, for example, by CherryPy.) Additionally, we may want to

avoid passing parameters via QUERY, as many people find QUERY-free URLs more

“clean”.

We can do this by implementing a dispatcher similar to Rails’ “routes”, which

will let us map a URL prefix and a sequence of parameters onto a function that

would accept a table of parameters:

URLs = {

{"/show/$page_name/$version", show_page}

}

This would map a request for /wiki/show/Home Page/23 onto a call to

show_page{page_name="Home_Page, version="23"}

which would return version 23 of the page. See Example 1 in this section for the

implementation of this approach.

A yet more flexible and very popular method (used, for example, by Django

and web.py) is to determine the function by matching the URL against a list of

patterns, each associated with a different function. E.g.:

88

8 · MVC Web Development with Kepler

URLs = {

{"/show/([%w_-]*)/(%d*)", show_page},

{"/save/([%w_-]*)", save_page}

}

The implementation of this approach is discussed in Example 2.

In some cases, however, we may want to think of URLs not as selecting

functions and passing parameters to them, but as selecting objects and sending

messages to those objects. This approach is particularly useful if we want to

make it easy to override the handling of the action on a per object basis, or if we

want to allow custom actions for the objects. For example, we may want some

pages of the wiki to be displayed in a special way or to respond to altogether

different requests: a page called “History” could have the ability to return the

RSS feed for the recent wiki edits. This type of dispatching is presented in

Example 3.

A complete solution to URL dispatching should also provide for a way to

generate URLs that would fit with the dispatching syntax. The ease of doing so

would depend on the specific dispatching mechanism. We show how to do this

for Examples 1 and 2.

Kepler setup

If you do not yet have Kepler installed, please follow the instructions on the

Kepler site. Kepler 1.1 default request handler is CGILua, which defaults to

CGI-like dispatching with URLs like /index.lp, but also allows the use of more

sophisticated dispatchers. The example code available on the Lua Gems site

include a gem directory that should be copied to your CGILUA APPS directory. After

that, access /app.lua/gem in your browser to get the examples home page. From

that page you can run all the examples.3

The following examples will show three different approaches to implement-

ing URL dispatching for a wiki, each using a different URL dispatching model.4

Example 1: mapping a sequence to a function and parameters

In this first example we’ll use simple patterns that are similar to Rails’ “routes”,

which will allow us to simultaneously identify a function and collect named

parameters. The patterns contains $name for each part of PATH INFO that is to

be used as a named parameter. All parameters are considered optional and it is

up to the function to determine how to handle unspecified parameters. All code

shown in this example is provided in example1.lua.

In the case of the wiki example, we may want to map the requests as follows:

3The URLs used in this article assume that Kepler is run with Xavante (a lightweight HTTP

server written in Lua and included with Kepler). For other servers, please see the Kepler documentation.

4Due to space limitations the examples in the text don’t show the full HTML used in the sources.

89

URLs = {

{"/show/$page_name/$version", show_page, "show"},

{"/history/$page_name/$year/$month/$date", show_history, "history"},

{"/diff/$page_name/$version1/$version2", show_diff, "diff"},

}

Here each mapping consists of three values: a path pattern, a function, and

the name of the mapping (used later for generating URLs). So a request for

/example1/diff/Home Page/24/25 would call

show_diff{page_name="Home_Page", version1="24", version2="25"}

In order to actually dispatch the request we will need a pattern matching

function and a function to iterate over the table looking for matches:

-- Checks if a URL matches a pattern

function match(url, pattern)

local params = {}

-- convert the pattern into a Lua-style pattern

local lua_pattern = string.gsub(pattern, "(/$[%w_-]+)", "/?([^/]*)")

-- extract param values from the URL

local param_values = {string.match(url, lua_pattern)}

-- save them in table fields

local i = 1

for name in string.gmatch(pattern, "/$([%w_-]+)") do

params[name] = param_values[i]

i = i + 1

end

-- return params or nil

return next(params) and params

end

-- Maps the correct function for a URL

function map(url)

for i, v in ipairs(URLs) do

local pattern, f, name = unpack(v)

local params = match(url, pattern)

if params then

return f, params

end

end

end

We can now use the map() function to dispatch a request:

f, params = map(cgilua.script_vpath)

if f then

return f(params)

end

90

8 · MVC Web Development with Kepler

For this example to work we need to also implement functions show page,

show history and show diff. We discuss their implementation later in “Content

generation”. For now, however, we will only mention that most of such functions

will need to generate URLs pointing back to the wiki. We can do this by reusing

our URLs table:

function makeurl(action_name, params)

for i, v in ipairs(URLs) do

local pattern, f, name = unpack(v)

if name == action_name then

local url = string.gsub(pattern, "$([%w_-]+)", params)

url = cgilua.urlpath.."/"..cgilua.app_name..url

return url

end

end

end

end

So we could then write:

local template = " <LI><A HREF=%s>v. %s (%s)</A></LI>"

string.format(template, makeurl("show", p), p.version, p.time_stamp))

to generate a link for a diff between versions 24 and 25 of Home Page.

This form of dispatching is so commonly used that CGILua offers a built-in

dispatcher for it. Using this dispatcher our example would have simply to return

the routing of the URL table:

URLs = {

{"/show/$page_name/$version", show_page, "show"},

{"/history/$page_name/$year/$month/$date", show_history, "history"},

{"/diff/$page_name/$version1/$version2", show_diff, "diff"},

}

return cgilua.dispatcher.route(URLs)

Note that cgilua.dispatcher.route implements the functions map() and

match() shown above. We have shown their implementation here to highlight

the similarities and differences between the examples.

Example 2: mapping through Lua pattern matching

In our previous example we presented an approach to request routing that is

simple and elegant, but may not give you enough control. Some of the popular

web frameworks opt for a more generic approach, mapping regular expressions

onto functions. We can do the same with Lua patterns:

URLs = {

{"/show/([^%/]*)/?(%d*)", show_page},

91

{"/history", show_history},

{"/history/([^%/]*)", show_history},

{"/history/([^%/]*)/(%d*)/?", show_history},

{"/history/([^%/]*)/(%d*)/(%d*)/?", show_history},

{"/history/([^%/]*)/(%d*)/(%d*)/(%d*)/?", show_history},

{"/diff/(%d*)/(%d*)", show_diff},

}

Note that this mapping is more verbose, but allows for more precise selection of

URLs. To implement this we use replace the match function of Example 1 with

a simple wrapper around string.match:

-- Matches a pattern to an URL

function match(url, pattern)

local match = {string.match(url, pattern.."$")}

return next(match) and match

end

We also make a small change to map() since our URLs table now has only two val-

ues per row. We then use map() in just the same way as we did in example1.lua.

All code shown in this example is provided in example2.lua.

This approach does not allow for such simple generation of URLs as was the

case in Example 1 so this would have to be done manually. (A more complicated

version of this method would make URL-generation possible, but will not be

discussed here.)

Example 3: delegating action selection to objects

In this last example, adapted from the way request dispatching is done in

Sputnik,5 we will use PATH INFO to identify the resource object (a wiki page in our

case) and the name of the action that the object would be asked to perform. We

leave the choice of response to the object however, allowing different objects to

respond differently to the same request. To avoid mixing parts of the controller

into the model, the model will store references to functions defined elsewhere. In

the case of our wiki, the URLs will look like /example3/Page Name.action name,

optionally followed by additional parameters in the QUERY part of URL. For

example,

/example3/Home_Page.diff?version=24&version2=25

If no .action name is specified, we shall assume that the action to be show. By

default, each action corresponds to a function defined by the controller, but

some pages can behave differently. For instance, we might want /History to

show recent changes to all pages instead of its own content, /History.rss to

return such changes as an RSS feed, /Calendar to show a list of events, and

/Calendar.ical to return this list in the iCalendar format. In other words, for

5http://sputnik.freewisdom.org/

92

8 · MVC Web Development with Kepler

both History and Calendar pages we want to redefine how show is handled as

well as define additional actions (rss and ical) that might make little sense for

other pages.

We implement our dispatching as follows:

local mask = "/([%w_-]+)%.?([^%./]-)$"

local page_name, action_name = string.match(cgilua.script_vpath, mask)

if action_name == nil or action_name == "" then

action_name = "show"

end

local page = model.get_page(page_name, cgilua.QUERY.version)

local action_function = load_action_function(page.actions[action_name])

local content, content_type =

action_function(page, page_name, cgilua.QUERY)

Note that the function that is called in the end is determined by the object.

The set of functions is thus not limited a priori — each object can support dif-

ferent functions, and we can remap them easily. In fact, in the case of Sputnik

the functions can be remapped by the visitor to the site. Note that page.actions

contains names of functions and we rely on load action function() to get the

actual callable functions:

function load_action_function(action_name)

-- if action contains a dot, assume it’s defined in an external

-- module. if it doesn’t, assume it refers to a global function

if string.find(action_name, "%.") then

local mask = "([%w_]*)%.([%w_]*)"

module_name, function_name = string.match(action_name, mask)

local m = require(module_name)

return m[function_name]

else

return _M[action_name]

end

end

If Home Page is a standard wiki page, page.actions["show"] should give us a

function that simply converts the content of the page to HTML. For the History

page, however, we would want "show" to display recent changes to all pages. To do this we create a new module and define there additional functions that can return wiki history as HTML or RSS (see the file /gem/lua/example3 history.lua).

We then add a new field to the object representing the History page, overrid-

ing its action for show and adding a new one for rss (see the file /gem/lua/

fake wiki model.lua for the whole data structure):

93

Home_Page = {

title = "Home Page",

actions = {

show = "show_page",

diff = "show_diff",

history = "show_history",

}

}

History = {

title = "Site History",

actions = {

show = "example3_history.show_history",

diff = "show_diff",

history = "show_history",

rss = "example3_history.show_wiki_history_as_rss",

}

}

Since we are using standard QUERY parameters, generating URLs can take

advantage of the functionality provided by cgilua:

function makeurl(page_name, action_name, params)

local url = "/"..page_name

if action_name and action_name ~= "show" then

url = url.."."..action_name

end

if next(params) then

url = url.."?"..cgilua.urlcode.encodetable(params)

end

return cgilua.urlpath.."/"..cgilua.app_name..url

end

Content generation

Overview

After dispatching, the controller must generate the necessary parts of an HTTP

response: the HTTP status, the HTTP headers, and the content part. This

section focuses on the approaches to generating content. One approach is to

generate the content programmatically, pushing one string after another into

a buffer. Another approach is to define template strings that are filled with

content when they are processed. We call the first method “scripting” and the

second method “templating”.

94

8 · MVC Web Development with Kepler

Scripting

The most basic way to do scripting in Kepler is to use cgilua.put to push bits of

content. Example 1 uses this approach:

function show_history(params)

local page = model.get_page(params.page_name)

local history =

page:get_history(params.year, params.month, params.date)

cgilua.htmlheader()

cgilua.put("<H1>"..page.title.." History</H1>")

cgilua.put("<UL>")

for i, p in ipairs(history) do

p.page_name = params.page_name

cgilua.put(string.format(" <LI><A HREF=%s>v. %s (%s)</A></LI>", makeurl("show", p), p.version, p.time_stamp))

end

cgilua.put("</UL>")

end

which would give us something like:

<H1>Home Page History</H1>

<UL>

<LI><A HREF=/example1/show/Home_Page/3>v. 3 (2007-05-29 20:02:01)</A></LI>

<LI><A HREF=/example1/show/Home_Page/2>v. 2 (2007-05-29 10:03:31)</A></LI>

<LI><A HREF=/example1/show/Home_Page/1>v. 1 (2007-05-29 08:20:00)</A></LI>

</UL>

Alternatively, we can use a module like HTK6 to generate the same HTML

(see example4.lua):

require"htk"

function show_history(params)

local page = model.get_page(params.page_name)

local history =

page:get_history(params.year, params.month, params.date)

cgilua.htmlheader()

cgilua.put(htk.H1{page.title.."History"})

items = {}

for i, p in ipairs(history) do

p.page_name = params.page_name

local url = makeurl("show", p)

table.insert(items, htk.LI{htk.A{href=url, "v. ", p.version,

"(", p.time_stamp, ")"}})

end

cgilua.put(htk.UL(items))

end

6HTK, by Tom ás Guisassola, is available at http://www.tecgraf.puc-rio.br/∼tomas/htk/.

95

Templating

Another content generation approach uses a template string with placeholders

for dynamic content. Such templates allow inclusion of arbitrary code, which

blurs the separation between code and presentation; they have both advantages

and disadvantages. Alternatively, one can use “safe templates” that can only

call the functions that are explicitly given to them. We present here two simple

libraries for those two cases, each implemented in under 150 lines of Lua code.

Kepler’s solution to arbitrary code templates is “Lua Pages”, which are text

files with syntax for two types of placeholders. The first allows arbitrary Lua

code, using <% cgilua.put(title) %>.

The second is a placeholder for a sin-

gle Lua expression, using <%= title %>

The Lua Pages pre-processor makes global substitutions on the template,

searching for matching pairs of markup and generating the corresponding Lua

code, which can then be executed.

To use Lua Pages with the dispatching methods discussed in the previous

section, we can call them explicitly from inside a Lua script. In example5.lua

we implement show history as follows:

function show_history(params)

local page = model.get_page(params.page_name)

local history =

page:get_history(params.year, params.month, params.date)

local env = {

page = page, history = history, page_name = params.page_name,

cgilua = cgilua, ipairs = ipairs, makeurl = makeurl,

}

cgilua.handlelp ("example5.lp", env)

end

This function delegates most of the content generation to example5.lp, which

looks like this:

<H1><%= page.title %> History</H1>

<UL>

<% for i, p in ipairs(history) do

p.page_name = page_name %>

<LI>

<A HREF="<%=makeurl("show", p)%>"> v.<%=p.version%>

(<%=p.time_stamp%>)</A>

</LI>

<% end %>

</UL>

Kepler also provides a solution for safe templates through Cosmo, a library that

allows two types of template placeholders: $var name and $fn name[[template]].

When a template is filled (using cosmo.fill function), a table must be provided

in addition to the template string. If cosmo.fill encounters a $var name pattern

96

8 · MVC Web Development with Kepler

it will simply look up the value in the table and substitute it. If it finds some-

thing like $fn name[[...]], it will look up the fn name field in the table but will

assume the corresponding value to be a function. Cosmo will then call this func-

tion in a coroutine, expecting it to yield one or more tables (using cosmo.yield).

Each table that is yielded will be used to fill the template inside [[...]], and

all the resulting text will be concatenated and inserted into the output. For

example:

show_history_template = [==[

<H1>$title History</H1>

<UL>

$list_versions[[<LI><A HREF="$url"> v. $version ($time_stamp)</A></LI>]]

</UL>]==]

function show_history(params)

local page = model.get_page(params.page_name)

local history =

page:get_history(params.year, params.month, params.date)

cgilua.htmlheader()

cgilua.put(cosmo.fill(show_history_template,

{ title = page.title,

list_versions = function()

for i, p in ipairs(history) do

p.page_name = params.page_name

p.url = makeurl("show", p)

cosmo.yield(p)

end

end }))

end

Conclusion

Kepler allows for MVC-style web development using many of the currently

popular approaches.

Instead of locking the user into a specific solutions to

such problems as request dispatching and content generation, Kepler focuses

on making web applications portable across operating systems and servers,

letting the application developers choose higher-level solutions appropriate to

their specific case. Your choice between the above mentioned approaches to

request dispatching may depend on how “clean” you want your URLs to be, how

much control you need over them, and whether the system needs to support

resource-specific actions. Similarly, the choice of method for content generation

may depend on the degree to which you want to separate design work from

programming. Lua Pages may offer a simpler solution in cases where design

and coding is done by the same person, while a safe template solution like Cosmo

may make your life easier if the design work is to be delegated to a designer or

even to anonymous end users.

9

Filters, Sources, Sinks & Pumps

or Functional programming for the rest of us

Diego Nehab

Certain data processing operations can be implemented in the form of filters.

A filter is a function that can process data received in consecutive invocations,

returning partial results each time it is called. Examples of operations that can

be implemented as filters include the end-of-line normalization for text, Base64

and Quoted-Printable transfer content encodings, the breaking of text into lines,

SMTP dot-stuffing, and many others. Filters become even more powerful when

we allow them to be chained together to create composite filters. In this context,

filters can be seen as the internal links in a chain of data transformations.

Sources and sinks are the corresponding end points in these chains. A source

is a function that produces data, chunk by chunk, and a sink is a function that

takes data, chunk by chunk. Finally, pumps are procedures that actively drive

data from a source to a sink, and indirectly through all intervening filters. In

this article, we describe the design of an elegant interface for filters, sources,

sinks, chains, and pumps, and we illustrate each step with concrete examples.

Introduction

Within the realm of networking applications, we are often required to apply

transformations to streams of data. Examples include the end-of-line normaliza-

tion for text, Base64 and Quoted-Printable transfer content encodings, breaking

98

9 · Filters, Sources, Sinks, and Pumps

text into lines with a maximum number of columns, SMTP dot-stuffing, gzip

compression, HTTP chunked transfer coding, and the list goes on.

Many complex tasks require a combination of two or more such transforma-

tions, and therefore a general mechanism for promoting reuse is desirable. In

the process of designing LuaSocket 2.0, we repeatedly faced this problem. The

solution we reached proved to be very general and convenient. It is based on the

concepts of filters, sources, sinks, and pumps, which we introduce below.

Filters are functions that can be repeatedly invoked with chunks of input,

successively returning processed chunks of output.

Naturally, the result of

concatenating all the output chunks must be the same as the result of applying

the filter to the concatenation of all input chunks. In fancier language, filters

commute with the concatenation operator. More importantly, filters must handle

input data correctly no matter how the stream has been split into chunks.

A chain is a function that transparently combines the effect of one or more

filters. The interface of a chain is indistinguishable from the interface of its

component filters. This allows a chained filter to be used wherever an atomic

filter is accepted. In particular, chains can be themselves chained to create

arbitrarily complex operations.

Filters can be seen as internal nodes in a network through which data will

flow, potentially being transformed many times along the way. Chains connect

these nodes together. The initial and final nodes of the network are sources and

sinks, respectively. Less abstractly, a source is a function that produces new

chunks of data every time it is invoked. Conversely, sinks are functions that

give a final destination to the chunks of data they receive in successive calls.

Naturally, sources and sinks can also be chained with filters to produce filtered

sources and sinks.

Finally, filters, chains, sources, and sinks are all passive entities: they must

be repeatedly invoked in order for anything to happen. Pumps provide the

driving force that pushes data through the network, from a source to a sink,

and indirectly through all intervening filters.

In the following sections, we start with a simplified interface, which we

later refine. The evolution we present is not contrived: it recreates the steps

we ourselves followed as we consolidated our understanding of these concepts

within our application domain.

A simple example

The end-of-line normalization of text is a good example to motivate our initial

filter interface. Assume we are given text in an unknown end-of-line convention

(including possibly mixed conventions) out of the commonly found Unix (LF),

Mac OS (CR), and DOS (CR LF) conventions. We would like to be able to use the

following code to normalize the end-of-line markers:

local CRLF = "\013\010"

local input = source.chain(source.file(io.stdin), normalize(CRLF))

local output = sink.file(io.stdout)

pump.all(input, output)

99

This program should read data from the standard input stream and nor-

malize the end-of-line markers to the canonic CR LF marker, as defined by the

MIME standard. Finally, the normalized text should be sent to the standard

output stream. We use a file source that produces data from standard input,

and chain it with a filter that normalizes the data. The pump then repeatedly

obtains data from the source, and passes it to the file sink, which sends it to the standard output.

In the code above, the normalize factory is a function that creates our normal-

ization filter, which replaces any end-of-line marker with the canonic marker.

The initial filter interface is trivial: a filter function receives a chunk of input

data, and returns a chunk of processed data. When there are no more input

data left, the caller notifies the filter by invoking it with a nil chunk. The filter

responds by returning the final chunk of processed data (which could of course

be the empty string).

Although the interface is extremely simple, the implementation is not so

obvious. A normalization filter respecting this interface needs to keep some kind

of context between calls. This is because a chunk boundary may lie between the

CR and LF characters marking the end of a single line. This need for contextual

storage motivates the use of factories: each time the factory is invoked, it returns

a filter with its own context so that we can have several independent filters

being used at the same time. For efficiency reasons, we must avoid the obvious

solution of concatenating all the input into the context before producing any

output chunks.

To that end, we break the implementation into two parts: a low-level filter,

and a factory of high-level filters. The low-level filter is implemented in C and

does not maintain any context between function calls. The high-level filter fac-

tory, implemented in Lua, creates and returns a high-level filter that maintains

whatever context the low-level filter needs, but isolates the user from its inter-

nal details. That way, we take advantage of C’s efficiency to perform the hard

work, and take advantage of Lua’s simplicity for the bookkeeping.

The Lua part of the filter

Below is the complete implementation of the factory of high-level end-of-line

normalization filters:

function filter.cycle(lowlevel, context, extra)

return function(chunk)

local ret

ret, context = lowlevel(context, chunk, extra)

return ret

end

end

function normalize(marker)

return filter.cycle(eol, 0, marker)

end

100

9 · Filters, Sources, Sinks, and Pumps

The normalize factory simply calls a more generic factory, the cycle factory,

passing the low-level filter eol. The cycle factory receives a low-level filter,

an initial context, and an extra parameter, and returns a new high-level filter.

Each time the high-level filer is passed a new chunk, it invokes the low-level

filter with the previous context, the new chunk, and the extra argument. It is

the low-level filter that does all the work, producing the chunk of processed data

and a new context. The high-level filter then replaces its internal context, and

returns the processed chunk of data to the user. Notice that we take advantage

of Lua’s lexical scoping to store the context in a closure between function calls.

The C part of the filter

As for the low-level filter, we must first accept that there is no perfect solution

to the end-of-line marker normalization problem. The difficulty comes from

an inherent ambiguity in the definition of empty lines within mixed input.

However, the following solution works well for any consistent input, as well as

for non-empty lines in mixed input. It also does a reasonable job with empty

lines and serves as a good example of how to implement a low-level filter.

The idea is to consider both CR and LF as end-of-line candidates. We issue

a single break if any candidate is seen alone, or if it is followed by a different

candidate. In other words, CR CR and LF LF each issue two end-of-line markers,

whereas CR LF and LF CR issue only one marker each. It is easy to see that this

method correctly handles the most common end-of-line conventions.

With this in mind, we divide the low-level filter into two simple functions.

The inner function pushchar performs the normalization itself. It takes each

input character in turn, deciding what to output and how to modify the context.

The context tells if the last processed character was an end-of-line candidate,

and if so, which candidate it was. For efficiency, we use Lua’s auxiliary library’s

buffer interface:

#define candidate(c) (c == CR || c == LF)

static int pushchar(int c, int last, const char *marker,

luaL_Buffer *buffer) {

if (candidate(c)) {

if (candidate(last)) {

if (c == last)

luaL_addstring(buffer, marker);

return 0;

} else {

luaL_addstring(buffer, marker);

return c;

}

} else {

luaL_pushchar(buffer, c);

return 0;

}

}

101

The outer function eol simply interfaces with Lua. It receives the context

and input chunk (as well as an optional custom end-of-line marker), and returns

the transformed output chunk and the new context. Notice that if the input

chunk is nil, the operation is considered to be finished. In that case, the loop

will not execute a single time and the context is reset to the initial state. This

allows the filter to be reused many times:

static int eol(lua_State *L) {

int context = luaL_checkint(L, 1);

size_t isize = 0;

const char *input = luaL_optlstring(L, 2, NULL, &isize);

const char *last = input + isize;

const char *marker = luaL_optstring(L, 3, CRLF);

luaL_Buffer buffer;

luaL_buffinit(L, &buffer);

if (!input) {

lua_pushnil(L);

lua_pushnumber(L, 0);

return 2;

}

while (input < last)

context = pushchar(*input++, context, marker, &buffer);

luaL_pushresult(&buffer);

lua_pushnumber(L, context);

return 2;

}

When designing filters, the challenging part is usually deciding what to store

in the context. For line breaking, for instance, it could be the number of bytes

that still fit in the current line. For Base64 encoding, it could be a string with the

bytes that remain after the division of the input into 3-byte atoms. The MIME

module in the LuaSocket distribution contains many other examples.

Filter chains

Chains greatly increase the power of filters. For example, according to the

standard for Quoted-Printable encoding, text should be normalized to a canonic

end-of-line marker prior to encoding. After encoding, the resulting text must be

broken into lines of no more than 76 characters, with the use of soft line breaks

(a line terminated by the = sign). To help specifying complex transformations

like this, we define a chain factory that creates a composite filter from one or

more filters. A chained filter passes data through all its components, and can be

used wherever a primitive filter is accepted.

The chaining factory is very simple. The auxiliary function chainpair chains

two filters together, taking special care if the chunk is the last. This is because

the final nil chunk notification has to be pushed through both filters in turn:

102

9 · Filters, Sources, Sinks, and Pumps

local function chainpair(f1, f2)

return function(chunk)

local ret = f2(f1(chunk))

if chunk then return ret

else return ret .. f2() end

end

end

function filter.chain(...)

local f = arg[1]

for i = 2, #arg do

f = chainpair(f, arg[i])

end

return f

end

Thanks to the chain factory, we can define the Quoted-Printable conversion

as such (the encode and wrap factories are also part of LuaSocket’s MIME

module):

local qp = filter.chain(normalize(CRLF), encode("quoted-printable"),

wrap("quoted-printable"))

local input = source.chain(source.file(io.stdin), qp)

local output = sink.file(io.stdout)

pump.all(input, output)

Sources, sinks, and pumps

The filters we introduced so far act as the internal nodes in a network of

transformations. Information flows from node to node (or rather from one filter

to the next) and is transformed along the way. Chaining filters together is our

way to connect nodes in this network. As the starting point for the network, we

need a source node that produces the data. In the end of the network, we need

a sink node that gives a final destination to the data.

Sources

A source returns the next chunk of data each time it is invoked. When there

are no more data, it simply returns nil. In the event of an error, the source can

inform the caller by returning nil followed by the error message.

Below are two simple source factories. The empty source returns no data,

possibly returning an associated error message. The file source yields the

contents of a file in a chunk by chunk fashion:

function source.empty(err)

return function()

return nil, err

end

end

103

function source.file(handle, io_err)

if handle then

return function()

local chunk = handle:read(2048)

if not chunk then handle:close() end

return chunk

end

else return source.empty(io_err or "unable to open file") end

end

Filtered sources

A filtered source passes its data through the associated filter before returning it

to the caller. Filtered sources are useful when working with functions that get

their input data from a source (such as the pumps in our examples). By chaining

a source with one or more filters, such functions can be transparently provided

with filtered data, with no need to change their interfaces. Here is a factory that

does the job:

function source.chain(src, f)

return function()

if not src then

return nil

end

local chunk, err = src()

if not chunk then

src = nil

return f(nil)

else

return f(chunk)

end

end

end

Sinks

Just as we defined an interface for a source of data, we can also define an

interface for a data destination. We call any function respecting this interface a

sink. In our first example, we used a file sink connected to the standard output.

Sinks receive consecutive chunks of data, until the end of data is signaled

by a nil input chunk. A sink can be notified of an error with an optional extra

argument that contains the error message, following a nil chunk. If a sink

detects an error itself, and wishes not to be called again, it can return nil,

followed by an error message. A return value that is not nil means the sink

will accept more data.

Below are two useful sink factories. The table factory creates a sink that

stores individual chunks into an array. The data can later be efficiently concate-

104

9 · Filters, Sources, Sinks, and Pumps

nated into a single string with Lua’s table.concat library function. The null

sink simply discards the chunks it receives:

function sink.table(t)

t = t or {}

local f = function(chunk, err)

if chunk then table.insert(t, chunk) end

return 1

end

return f, t

end

local function null()

return 1

end

function sink.null()

return null

end

Naturally, filtered sinks are just as useful as filtered sources. A filtered

sink passes each chunk it receives through the associated filter before handing

it down to the original sink. In the following example, we use a source that

reads from the standard input. The input chunks are sent to a table sink,

which has been coupled with a normalization filter. The filtered chunks are

then concatenated from the output array, and finally sent to standard out:

local input = source.file(io.stdin)

local output, t = sink.table()

output = sink.chain(normalize(CRLF), output)

pump.all(input, output)

io.write(table.concat(t))

Pumps

Although not on purpose, our interface for sources is compatible with Lua iter-

ators. That is, a source can be neatly used in conjunction with for loops. Using

our file source as an iterator, we can write the following code:

for chunk in source.file(io.stdin) do

io.write(chunk)

end

Loops like this will always be present because everything we designed so far

is passive. Sources, sinks, filters: none of them can do anything on their own.

The operation of pumping all data a source can provide into a sink is so common

that it deserves its own function:

105

function pump.step(src, snk)

local chunk, src_err = src()

local ret, snk_err = snk(chunk, src_err)

if chunk and ret then return 1

else return nil, src_err or snk_err end

end

function pump.all(src, snk, step)

step = step or pump.step

while true do

local ret, err = step(src, snk)

if not ret then

if err then return nil, err

else return 1 end

end

end

end

The pump.step function moves one chunk of data from the source to the sink.

The pump.all function takes an optional step function and uses it to pump all

the data from the source to the sink. Here is an example that uses the Base64

and the line wrapping filters from the LuaSocket distribution. The program

reads a binary file from disk and stores it in another file, after encoding it to the

Base64 transfer content encoding:

local input = source.chain(

source.file(io.open("input.bin", "rb")),

encode("base64"))

local output = sink.chain(

wrap(76),

sink.file(io.open("output.b64", "w")))

pump.all(input, output)

The way we split the filters here is not intuitive, on purpose. Alternatively,

we could have chained the Base64 encode filter and the line-wrap filter together,

and then chain the resulting filter with either the file source or the file sink. It

doesn’t really matter.

Exploding filters

Our current filter interface has one serious shortcoming. Consider for example

a gzip decompression filter. During decompression, a small input chunk can be

exploded into a huge amount of data. To address this problem, we decided to

change the filter interface and allow exploding filters to return large quantities

of output data in a chunk by chunk manner.

More specifically, after passing each chunk of input to a filter, and collecting

the first chunk of output, the user must now loop to receive other chunks from

the filter until no filtered data are left. Within these secondary calls, the caller

106

9 · Filters, Sources, Sinks, and Pumps

passes an empty string to the filter. The filter responds with an empty string

when it is ready for the next input chunk. In the end, after the user passes a

nil chunk notifying the filter that there are no more input data, the filter might

still have to produce too much output data to return in a single chunk. The user

has to loop again, now passing nil to the filter each time, until the filter itself

returns nil to notify the user it is finally done.

Fortunately, it is very easy to modify a filter to respect the new interface. In

fact, the end-of-line translation filter we presented earlier already conforms to

it. The complexity is encapsulated within the chaining functions, which must

now include a loop. Since these functions only have to be written once, the user

is rarely affected. Interestingly, the modifications do not have a measurable neg-

ative impact in the performance of filters that do not need the added flexibility.

On the other hand, for a small price in complexity, the changes make exploding

filters practical.

A complex example

The LTN12 module in the LuaSocket distribution implements all the ideas we

have described. The MIME and SMTP modules are tightly integrated with

LTN12, and can be used to showcase the expressive power of filters, sources,

sinks, and pumps. Below is an example of how a user would proceed to define

and send a multipart message, with attachments, using LuaSocket:

local smtp = require"socket.smtp"

local mime = require"mime"

local ltn12 = require"ltn12"

local message = smtp.message{

headers = {

from = "Sicrano <sicrano@example.com>",

to = "Fulano <fulano@example.com>",

subject = "A message with an attachment"},

body = {

preamble = "Hope you can see the attachment" .. CRLF,

[1] = {

body = "Here is our logo" .. CRLF},

[2] = {

headers = {

["content-type"] = ’image/png; name="luasocket.png"’,

["content-disposition"] =

’attachment; filename="luasocket.png"’,

["content-description"] = ’LuaSocket logo’,

["content-transfer-encoding"] = "BASE64"},

body = ltn12.source.chain(

ltn12.source.file(io.open("luasocket.png", "rb")),

ltn12.filter.chain(

mime.encode("base64"),

mime.wrap()))}}}

107

assert(smtp.send{

rcpt = "<fulano@example.com>",

from = "<sicrano@example.com>",

source = message})

The smtp.message function receives a table describing the message, and

returns a source. The smtp.send function takes this source, chains it with the

SMTP dot-stuffing filter, connects a socket sink with the server, and simply

pumps the data. The message is never assembled in memory. Everything is

produced on demand, transformed piece by piece, and sent to the server in

chunks, including the file attachment which is loaded from disk and encoded

on the fly.

Conclusion

In this article, we introduced the concepts of filters, sources, sinks, and pumps

to the Lua language. These are useful tools for stream processing in general.

Sources provide a simple abstraction for data acquisition. Sinks provide an ab-

straction for final data destinations. Filters define an interface for data trans-

formations. The chaining of filters, sources and sinks provides an elegant way

to create arbitrarily complex data transformations from simpler components.

Pumps simply push the data through.

Acknowledgments

The concepts described in this text are the result of long discussions with David

Burgess. A version of this text has been released on-line as the Lua Technical

Note 012, hence the name of the corresponding LuaSocket module, LTN12.

Wim Couwenberg contributed to the implementation of the module, and Adrian

Sietsma was the first to notice the correspondence between sources and Lua

iterators.

10

Lua as a Protocol Language

Patrick Rapin

This article describes the use of Lua as a communication vector between a client

and a server programs. In addition to some implementation choices, we discuss

the advantages and drawbacks of this approach, with a special point made on

security.

Background

Our company is using a custom source control system program called Code-

Administrator, a tool I wrote several years ago in C++ using Microsoft Foun-

dation Classes (MFC). The program has run to satisfaction until now; however

it has some limitations: it can only run on Windows and it cannot be used over

a regular Internet connection, only through a virtual private network.

We tried to find a way to keep the compatibility of the source database and

version numbers, while adding support for Unix-based systems and Internet

functionality. An idea for a solution arose: rewrite the core of the program using

Lua, into a new tool called Lua CodeAdministrator or LCA. At first at least,

there is no need to port all features, notably the administrative tasks, since it

is aimed to be a user add-on to the original tool rather than a full replacement.

Although the goal of this article is to discuss the ideas beyond the protocol used,

it will refer to this particular program when needed for the explanation.

110

10 · Lua as a Protocol Language

Choice of language

What are the advantages of using Lua to implement a version checking utility?

Compared to compiled languages, using a scripting language simplifies the

implementation a lot because:

• It is easy to ensure the portability of the program.

• The code is typically shorter compared to C functions.

• It is easy to customize functions, for example by overriding global vari-

ables.

• Configuration files can be written in the same language as the main pro-

gram.

• The protocol itself can use the same language, which is the main subject of

this article.

Other scripting languages would certainly also fit the requirements for this

tool. The reasons we prefer Lua are the following:

• It is fast, compared to most other scripting languages.

• It is small, thus there is no practical problem embedding it into programs.

• It is easy to compile on all platforms.

• We have a very good experience with the language since we are using it for

our printers.

Protocol

Like for example CVS, LCA can be run in four different modes:

Standalone. The user directory and the repository can both be accessed directly,

either on a local hard drive or over a mapped network drive. There is no

need to worry too much about security in this case: we can assume that the

file system management already checks for read and write authorizations.

Client. The user directory can be accessed directly by the program, and any

read or write action to the repository must be performed through a request

over the network to the server. Security is not a problem on this side; we

assume that the user has authorized access to his computer. But the client

must be able to provide security checks to the server.

Server. Unlike the other two modes, which are run once for each operation

requested, the server must run permanently, as a daemon. It has full

direct access to the repository, but each time it needs reading or writing

to the user directory, it issues requests back to the client. Security is an

important issue for any program accepting requests from the Internet. The

user must first log in, and data can be encrypted to ensure confidentiality.

Other tricks are used, as discussed later.

111

CGI Server. It has direct access to the repository, but no knowledge of a user

directory. It is not permanent like the previous mode, but is run when

needed by the Web server through the CGI interface. It is able to browse

the database and produce regular HTML output.

While the repository database format is already strictly defined and cannot

be changed without breaking compatibility, we have complete freedom over the

protocol used to exchange data. In order to simplify the coding and to unify all

concepts, we are using Lua code as the native format.

Any transferred data has the form of some Lua script, normally just a call

to a global function with parameters. Parameters values can be arbitrary com-

plex: they may contain big table constructors or embedded files encoded with

the string.format("%q") feature. This function is very helpful to embed binary

data into valid literal Lua strings. The result is a quoted string, with problem-

atic characters escaped and the other ones copied verbatim.

We also need a command terminator, to synchronize the client and the server

through the socket. It is customary in Internet protocols to use a line feed

optionally preceded by a carriage return. As this pattern may be present in valid

Lua code, we prefer to use the single null character ‘\0’. This character can only

appear inside long strings (those of the form [[· · ·]]), a construction never used

in the implementation. A second pattern, consisting of a semicolon immediately

followed by a line feed, can only be found at the end of a Lua statement. If the

whole request is too big, it is possible to execute it piece by piece by cutting it

along this pattern.

A typical transaction looks like the following. The client initiates a socket

and sends a request to the server as a Lua chunk followed by a null byte. The

server listens to the socket until it finds the termination character. After some

basic security checks, it will compile the code and run it. During the execution,

results and additional requested information are formatted into another Lua

chunk, and sent back to the client followed by the null terminator. The client

will then compile and run the response code. An advantage here is that the

protocol is fully symmetrical: the client always initiates the connection, but work

can be requested by both ends. Also, we can deal with complex situations with

hierarchical data and callbacks, without having to define a complicated protocol.

The following code shows the skeleton for the server function (without log-

ging and security checks). Please note the use of setfenv, forcing the chunk to

run in a protected environment, where the only global functions are the ones

we need to export. The "*z" parameter is a little extension made to LuaSocket

library allowing us to read all data up to a null byte (excluded from the result

string). At the end, a complete garbage collection is performed. The main server

function consists simply in calling ExecuteJob in protected mode and collecting

garbage. In case of any error occurring in the job function, the server recovers

automatically, because all resources, including sockets, are local variables that

will be freed or closed with the collection.

112

10 · Lua as a Protocol Language

local function ExecuteJob()

local penv = CreateProtectedEnv()

local sock = assert(socket.bind(config.Host, config.Port))

local connection = assert(sock:accept())

repeat

local line, errmsg, rest = connection:receive("*z")

line = line or rest

local funct = assert(loadstring(line))

setfenv(funct, penv)

funct()

collectgarbage()

until errmsg

end

function Server()

while true do

pcall(ExecuteJob)

collectgarbage()

end

end

The forced garbage collection performed in both functions is not strictly

necessary, especially in Lua 5.1, which collects garbage incrementally. But it

helps to minimize the memory usage and to close resources in case of errors. It

is probably a good idea to free all temporary data after a command execution, so

that no useless memory remains allocated during the server idle time.

As a very simple example, here is the checkout of one single file, hello.lua:

ExtractFiles("1.0.1", { "hello.lua", }, "~/")\0

WriteFile("~/hello.lua", "print(\"Hello World!\")", 1023454677)\0

The first line is the client request, sending the desired file version, the list of files

to retrieve, and the destination directory. The second line is the server response,

asking the client to save the given data into a file specifying its full file name

and modification time.

Compression

The original database format uses BZIP2 compression library to drastically

decrease its size. By the way, we were surprised to observe that CA databases

for typical C projects have an overall compression ratio of about 95% (or 20

times)! LCA must of course use this library to open the source code database.

The same algorithm is also used to compress the Lua source code stored inside

the executable file, and to compress data sent over the Internet socket. The

latter is optional, because on a local network the transfer time is probably lower

113

than the compression time, while the inverse yields true over a slow Internet

connection. An auxiliary global Load function is used for this task. Its prototype

is:

Load(flags, string_data)

where flags is a combination of Boolean values, indicating whether or not

string data is compressed, and whether or not it is encrypted (see below). The

string data is again generated with the string.format("%q") feature.

Encryption

Optionally, data can also be encrypted using an MD5 library in cypher-feedback

mode. We chose this algorithm simply because a standard Lua module exists.

The security of this algorithm is certainly enough for our application. The same

Load function is used for this case. This is no requisite; an idiom like this one

would do the same thing, although it is a little more verbose:

loadstring(Uncompress(Decrypt(string_data)))()

Of course, when combining compression and encryption, we must first apply

compression and then encryption; inversely, first decryption and then decom-

pression. This is because an encrypted message compresses very badly, since it

just looks like random data.

Secured mode

There are two running modes for the server: secured and unsecured.

The

unsecured mode is targeted to be used inside a secured local network, while

the secured mode could be opened on the whole Internet.

In secured mode, the authorized user must log in with a password before

he can make any operation. An MD5 hash of a challenge phrase is used for

the authentication procedure. Before login is complete, the only global value in

the environment is the authorization function itself. The server will check and

refuse any request that do not look like a valid authorization request.

If the login succeeds, most of the other custom functions become available, as

in the unsecured mode. Critical functions that could destroy the database may

only be executed over a secured local network.

Benchmark

A small benchmark run between the original MFC program and the Lua-based

version showed surprising results, which may be of interest to other Lua pro-

grammers. For the key feature of extracting a version, the equivalent of cvs

checkout -r tag, LCA in standalone mode happens to take roughly the same

114

10 · Lua as a Protocol Language

time as its C++ counterpart. It uses however about twice as much of memory.

We can probably explain these measures with the observations in the next para-

graph.

All CPU intensive operations (compression, checksum computation, merge)

are implemented in C code, using the same libraries. The core work of both

implementations is to build the file lists of any version from the incremental

database. For these, we mainly use strings, and data structures like arrays and

hashes. In Microsoft Foundation Classes these are implemented using CString,

CArray and CMap objects respectively. In Lua, we of course use native strings

and tables. While Lua is itself interpreted, which is a performance penalty over

C++, its native implementation of strings and tables seems to be faster than

the equivalent MFC classes. Concerning the memory usage, we can argue that

objects are not freed immediately when they are not needed anymore, like in

C++, but incrementally collected by the garbage collector. Using default settings,

the garbage collector of Lua 5.1 has a step pause of 200%, meaning that it waits

until the memory used has doubled compared to last collection before running

again. This factor is approximately the one we saw in the benchmark.

In client/server mode, if both programs runs on the same computer, the

checkout time using raw transfers is about 20% higher than in standalone mode.

The overhead rises to about 60% when using both compressed and encrypted

transfers.

Security

Security is a major issue for any system opening sockets over the Internet. As

the protocol consists of plain scripting code, this is quite an invitation for hackers

to send malicious code to the server!

Library functions

Fortunately, Lua gives us some weapons to fight against hacking. First, in this

language we have full control of which functions are exported into the global

environment. None of the standard Lua functions is present in the environment

used to evaluate external command scripts.

These functions are in reality

present in Lua state memory, but only as local variables, so there is no way

to access them even with a malicious code.

Of the standard libraries, the coroutine, math, table and string libraries

are normally harmless. They are not exported nevertheless because we do not

need them, and do not want to give these facilities to the outside world.

On the other hand, the io, os, and debug libraries are very dangerous. If

a hacker has direct access to io.open or os.execute function, he can delete

or create files on the server system nearly as he wants (just limited by the

operating system permissions).

The base function dofile and the package

library may be used to run external Lua code present somewhere on the server

hard disk, providing the hacker has the knowledge of the place to find these.

115

The debug library opens a more subtle security backdoor. Using debug.getlocal,

you can access local variables, and with debug.getupvalue, non-local variables.

These are normally completely inaccessible to an outside program, but if a

malicious code can use debug.getupvalue to gain access to the os table stored as

a non-local variable, we have lost the game.

Buffer overrun

The most common security issues using networking programs are the so-called

buffer overrun bugs. They typically occur when reading data coming from the

outside world without checking for the maximum size. A simple example in C is

this one:

int InputNumber(void)

{

char buffer[100];

gets(buffer);

return atoi(buffer);

}

Instead of the expected number, a hacker can send more than 100 non-null

characters and patch the function return address normally found in memory

just after the buffer. This forces the program to jump to an arbitrary address,

typically the function buffer itself, where the hacker just placed some executable

code!

Here is the very good news: this type of bugs is impossible in Lua language

(and in most other scripting languages), provided that there is no remaining

bugs in the implementation that could be exploited by malicious people. Be-

cause Lua has been used and tested much more than our application, we can

reasonably assume that there is no security issue related to buffer overruns.

Denial of service

A hacker may also attack a server trying to overflow the computer performance.

There are plenty of possibilities there: opening hundreds of simultaneous con-

nections without closing them, overwhelming the bandwidth by requesting huge

amount of data, asking the server for too complex requests, etc. This type of at-

tack is not as harmful as the previous ones, because no private data can be

stolen this way from the server, and there should not be any data loss. Restart-

ing the server program (or the whole computer) is enough to recover from such

a problem. Nevertheless, it harms regular users, who won’t be able to access the

server for some period of time. Some critical services cannot afford such a risk

of interruption and must take important measures against denial of service at-

tacks. For others, it may be enough to guarantee that no data loss or corruption

occurs, and that the server will be up again in a reasonable amount of time.

Using Lua as a protocol language makes it very easy to overwhelm the server.

Here are some examples:

116

10 · Lua as a Protocol Language

1. while true do end

2. function f() return f() end; f()

3. a = " "; for i=1,math.huge do a = a..a end

4. a = " "; a=a..a; a=a..a; a=a..a; a=a..a; ...

5. a = "An infinite string.

An infinite string.

An ...

The first request uses 100% CPU time, and never finishes.

The second

example too: as this is a tail call, it does not consume any stack space and

there is no limit on the call level. The third line will exponentially eat all the

available virtual memory on the computer (physical memory and swap file),

until an out-of-memory error occurs. This shows that loops and tail calls are

dangerous and should be forbidden. But even without loops we can achieve the

same result with a finite but long enough command as shows code 4. If we also

forbid concatenation, there is still the possibility to send a huge command like

code 5, supposing the hacker has even bandwidth and time for his attack.

It is very difficult to protect against all possible attacks. However, a small

number of checks can be done to drastically decrease vulnerability. There are not

always necessary; it depends on the application design and the desired security

level.

• Limit the number of simultaneous connections.

• Place a maximum size for any request.

• Before login is complete, refuse any request not matching a strict string

pattern.

• Place a lower limit for memory allocation than the total available on the

computer. For that you just have to provide a custom lua Alloc function to

lua newstate, counting allocated memory.

• Run a separate program that will monitor CPU and memory usage of the

server, like top on Unix or the task manager on Windows. If this usage

goes higher than a reasonable limit, the program will kill and rerun the

server.

• Forbid some Lua virtual machine opcodes. This can be achieved by first

compiling the received chunk, then analyzing the binary code. We just

have to avoid 3.5 instructions out of 38:

– JMP with a negative offset, found at end of while loops;

– FORLOOP, at end of for loops;

– TAILCALL, used for tail calls;

– CONCAT, used for string concatenation.

117

The code below implements this check. Notice that it needs private headers

and structures. So it is neither portable nor advisable: use it only when

necessary.

#include "lstate.h"

#include "lopcodes.h"

static int checkfct(const Proto* f)

{

const Instruction* code=f->code;

int i,n=f->sizecode;

for (i=0; i<n; i++)

{

Instruction instr=code[i];

OpCode o=GET_OPCODE(instr);

if(o == OP_FORLOOP || o == OP_CONCAT || o == OP_TAILCALL)

return 0;

if(o == OP_JMP && GETARG_sBx(instr) < 0)

return 0;

}

n = f->sizep;

for(i=0;i<n;i++)

if(!checkfct(f->p[i]))

return 0;

return 1;

}

static int check_opcode (lua_State *L)

{

luaL_checktype(L, 1, LUA_TFUNCTION);

lua_pushboolean(L, checkfct(clvalue(L->base)->l.p));

return 1;

}

Conclusion

This experience shows that it is possible to use Lua as a protocol language

over an Internet socket. Such a protocol simplifies the implementation and

debugging of the communication tool, if a Lua interpreter is used for other tasks

as well. However, it is clearly not a good choice for critical services, because

the scripting language opens a number of security issues. This approach is best

targeted to quickly developed enterprise tools, run over a secured local area

network.

11

Lua Script Packaging

Han Zhao

Why do we need to package script files together?

In real world applications, the logic or data described in Lua will be distributed

into many .lua files that may be scattered in different directories.

During

development this directory structure reflects the organization of modules or

resources (if Lua serves as a data description language). But when release time

comes, there might be too many files to deliver. After installation, all script

source files will be visible to the end user — that may not be acceptable for a

commercial product for both maintenance and security reasons. A packaging

mechanism is necessary to avoid exposing the internal details of an application.

First try: luac -o

During development, .lua files are usually organized in a directory hierarchy. If

you use luac to compile .lua source files into a single pre-compiled file as in

luac -o dest-file src-file-list

the directory hierarchy will be lost — all source files are compiled and packed

together flatly. For simple projects, it’s possible to build a utility to map the

directory hierarchy to the flat structure of the release package. But there will be

a maintenance overhead — the developer will have to keep in mind where a file

is located in the release package. We don’t want our modularization strategy for

120

11 · Lua Script Packaging

development to compromise our deployment requirements. Sometimes we also

need to dynamically load a .lua resource file from the package, but luac doesn’t

offer this facility either.

Mock require and dofile

To load Lua files from a package, we need to modify the default behavior of

standard require and dofile.

How does the standard require work?

According to Programming in Lua Chapter 15.1, the require function works in

following steps:

“Its first step is to check in table package.loaded whether the module

is already loaded. If so, require returns its corresponding value.

Therefore, once a module is loaded, other calls to require simply

return the same value, without loading the module again. If the

module is not loaded yet, require tries to find a loader for this module

[. . . ] Its first attempt is to query the given library name in table

package.preload. If it finds a function there, it uses this function as

the module loader.”

Besides the internal mystery require does, require itself is just a normal Lua

function which we can replace with our mocked package-friendly one.

Mock require

We’ll leverage the module loading mechanism described above to build a deco-

rator around the standard require to do the extra work: plugging our module

loader onto package.preload table to load a .lua file from package and leaving

the rest to the standard require.

The mocked require looks like this:

-- rename standard require

local lua_require = require

-- mocked require

function require(mod_name)

-- redirect loading function to our package (.dat) loader

package.preload[mod_name] = fio_loader

-- Lua standard loading routine

lua_require(mod_name)

end

The package loader (fio_loader) will convert the given module name to a

format needed by the packaging format. Then fio_loader will load the module

(.lua file) referenced by module name from the package:

121

function fio_loader(mod_name)

print(’fio require ’..mod_name)

-- replace ’/’ with ’\’ for path separator in package (.dat)

mod_name = string.gsub(mod_name, ’/’, ’\\’)

local ref_name = mod_name..’.lua’

local ret = FioG.c_load_chunk_from_dat(ref_name, FioG.fio_script_dat)

if(type(ret) == ’string’) then

-- error while loading chunk

print(ret)

elseif(type(ret) == ’function’) then

-- if the loader returns any value, require returns this value and

-- stores it in table package.loaded to return the same value in

-- future calls for this same library

-- passing in the mod name, module(..., package.seeall) will need it

return ret(mod_name)

else

error(’unknown chunk type ’..type(ret))

end

end

In the code above FioG is a name space.

The .dat package format will be

described in later section. You can choose a common format or implement a

specific one for your project.

FioG.fio_script_dat is a global user data of type DatFileReg. It holds the

.dat package and keeps a map from module name to an offset to access a specific

module file in the package.

FioG.c_load_chunk_from_dat is an imported C function. It will locate the

script in the package, load it using lua_load, and return the result: a Lua

function or an error message. Since lua_load just loads the chunk without

running it, we need to invoke the returned function with the module name

passed in:

return ret(mod_name)

(For Lua 5.0, the module name (mod_name) can be ignored. If you’re using the

module function introduced in Lua 5.1: module(..., package.seeall), it is

obligate to be passed in.)

The implementation of FioG.c_load_chunk_from_dat in C++ is:

122

11 · Lua Script Packaging

static int s_load_chunk_from_dat(lua_State* L){

const char* ref_name = lua_tostring(L, 1);

DatFileReg* reg = (DatFileReg*)lua_touserdata(L, 2);

// locate the script chunk in the packed .dat file

reg->offset(ref_name);

// construct a packed file to read

DatFile file(reg->handle());

// callback argument:

// struct ZipWrap{

//

DatFile* fp_;

//

int left_bytes_;

// };

ZipWrap zip_wrap = {&file, reg->size(ref_name)};

// load script, {file and file size} is the opaque value

// passed to the lua_Reader callback function

int load_rslt = lua_load(L, call_back_lua_Reader, &zip_wrap, ref_name);

// return result of lua_load

return 1;

}

DatFile is the actual reader of packed script — it implements the gzip algorithm

for .dat file. It’s held by a ZipWrap struct which is passed into lua_load as the

opaque data value for the lua_Reader callback function:

const char* call_back_lua_Reader(lua_State* L, void* data, size_t* size){

// unpack zip wrap to get dat file and number of bytes to read

DatFile* fp = ((ZipWrap*)data)->fp_;

int& left = ((ZipWrap*)data)->left_bytes_;

// load script into buffer

int nread = fp->read(chunk_buf, min(left, NBUF));

switch(nread){

case -1:

// gzip read error

assert(false && "gzip reading failed");

return 0;

case 0:

// eof

return 0;

default:

// discount bytes to read

left -= nread;

}

*size = nread;

return chunk_buf;

}

123

call_back_lua_Reader works just as lua_load requested: “Every time it

needs another piece of the chunk, lua_load calls the reader, passing along its

data parameter. The reader must return a pointer to a block of memory with a

new piece of the chunk and set size to the block size. The block must exist until

the reader function is called again. To signal the end of the chunk, the reader

must return NULL. The reader function may return pieces of any size greater

than zero.”

chunk_buf is a global buffer of NBUF bytes to hold and return the read script

chunk. It can be a static buffer or a dynamically allocated one from heap. You

can try different buffer sizes to balance the loading time and space requirement.

In my projects, a 100K buffer in heap works well (1K and 10K buffers will

consume a little more time).

Since it is hard to get original size of a zipped module file without completely

inflating it, we keep the size in DatFileReg and pass it to call_back_lua_Reader

in the field ZipWrap->left_bytes_.

Mock dofile

The dofile function “opens the named file and executes its contents as a Lua

chunk.” Unlike require, each time dofile invoked, a fresh piece of script will be

loaded and executed.

dofile is useful when you use Lua for configuration or resource description.

For example, in a game project, a map is a .lua file which contains tables of

cells, items, and critters. When the player steps into a new level, dofile will

dynamically load the map file into memory to construct the level. Like mocked

require, mocked dofile also relies on FioG.c_load_chunk_from_dat, as shown

in Listing 1. Here we use absolute path (home_dir) for dofile, e.g., to load a

game map:

dofile(home_dir..’master.dat\\level\\dungeon.lua’)

Then pattern matching is applied on the path to get the directory where the

script is and the reference name (in the example, the directory is ‘master.dat’

and the reference name is ‘level/dungeon.lua’). After adjusting reference name

format, we are able to load the file from corresponding packed .dat file (here,

FioG.fio_master_dat) just like what we’ve done for mock require.

Sometimes it’s not necessary to pack all resource files into packages. In the

code above, when directory is ‘savegame’, we use standard dofile (renamed to

lua_dofile) to load the saved game record which is just a .lua file saved in

‘savegame’ directory. Thus we have the flexibility to load file from both package

and normal file system; this is a powerful mechanism for loading Lua-described

resources.

From separate Lua files to packed .dat — How to organize?

In development, we use separate files (in hierarchical directory) to modularize

and use require to model dependencies between modules. It’s important that

124

11 · Lua Script Packaging

function FioG.mock_dofile()

-- save standard dofile

local lua_dofile = dofile

function dofile(filename)

-- find script directory and reference name

local b, e, dir, ref = string.find(filename,

home_dir..’([%a%.]+)\\([%a%p%d]+)’)

if(b) then

-- adjust for reference name in .dat

ref = string.gsub(ref, ’/’, ’\\’)

local ret = nil

if(dir == ’master.dat’) then

ret = FioG.c_load_chunk_from_dat(ref, FioG.fio_master_dat)

elseif(dir == ’critter.dat’) then

ret = FioG.c_load_chunk_from_dat(ref, FioG.fio_critter_dat)

elseif(dir == ’savegame’) then

-- saved game is not packed in .dat, using lua’s dofile

lua_dofile(filename)

return

else

error(’load file unknown dir type ’.. dir)

end

if(type(ret) == ’string’) then

error(ret)

end

-- execute loaded chunk

ret()

else

error(’load file invalid format, ’..filename)

end

end

end

Listing 1.

125

the packaging (or deploying) strategy doesn’t interfere with the development

file organization structure. A smooth transfer from separate development files

to packed ones for release is necessary. Otherwise, we will have to manually

maintain the mapping between development structure and release package

structure.

As we’ve seen, there’s a code snippet to adjust reference name in both mocked

require and dofile. This snippet automatically maps the development struc-

ture to release package structure. And the mapping reserves the directory struc-

ture (e.g., ‘utility/utf8’) in the reference name.

For require, during development we put all the files under a directory named

‘script’, creating subdirectories for submodules if necessary.

The standard

require will find and load these files based on package.path or LUA_PATH and

reference name (e.g., require’utility/utf8’).

In the release build, we pack all the files in ‘script’ directory into a .dat file

with the reference names and file sizes built in. The packed .dat file will be

loaded during startup (in a DatFileReg object named FioG.fio_script_dat in

Lua) and the standard require is also mocked here. Later when we require a

module, the mocked one will be invoked. It’ll find the script in the .dat and load

it as described in previous section.

dofile works in a similar but more flexible way: you can choose to pack

resources into more than one packed file for a better organization.

When to mock?

It depends on your project. The following loading steps work well in several

projects:

1. Two config.lua files: one for development and one for release. It should

contains a field ‘packed’ which is false for development and true for release.

The config.lua file works like a compiler switch for mocking.

2. Mock functions (mock require, mock dofile) in a separate .lua file. The

two mock functions are defined and loaded here.

3. Your first Lua entry file. This is the bootstrap file which requires other .lua

files. After reading the config.packed field from C++ , we can decide which

loader to invoke: if the project is in development phase, using the standard

lua_load; or in release, using a function like FioG.c_load_chunk_from_dat

to load. The file should contain following initialization code at the top:

if(config.packed) then

mock_require()

mock_dofile()

end

For released project, all later require and dofile are mocked hereafter.

126

11 · Lua Script Packaging

Choose a packaging algorithm/utility

The mocked require and dofile hides the file loading mechanism from Lua. We

can choose any packaging algorithm that best suits our needs. Though the zlib-

alike .dat format we’ve used for illustration is good enough for general purpose

packaging, you can build your own home-brew format or even no packaging at

all, just a dumb wrapper around the standard require and dofile.

.dat format

Here is a brief description of the .dat format.

.dat is a format used in Black Isle Studio’s Fallout role-playing game series

for game resource packaging. Its simple structure makes it a good candidate.

A .dat file contains a sequence of gzip compressed files with a record at the tail

describing each file’s reference name, file size, and offset.1 In our implementa-

tion, DatFileReg holds the .dat file, maintains a map from reference name to

offset; DatFile is responsible for reading a single zip file (located by a reference

name) in which a resource file is packed. The format has been well studied

and supported by the Fallout modding community — there are many .dat pack-

ers/unpackers available (both command line and GUI ones).2 It’s convenient to

choose this format instead of inventing a new one and building the tool set from

scratch.

Compiled vs plain .lua files

Simple packaging of plain .lua files has one drawback: anyone can view the

source code with a text editor. To protect the source code, you can pack compiled

.lua files instead of plain source files. Since the Lua file loader treats both

formats in the same way, here’s a trick to make our mocked require and dofile

still work for compiled files: when compiling, you just need to name the output

file the same as the original .lua file:

luac -o output\foo.lua foo.lua

Or use the following command if you’re working in batch mode on Windows:

for %%f in (*.lua) do luac -o output\%%f %%f

Please note that the compiled file is still easy to be hacked — there are some

decent decompilers for Lua out there. You can use the technique described here

as a starting point to build more advanced features like accessing verification or

encryption into your packaging algorithm to protect the source code.

1 .dat file format description: http://wiki.fifengine.de/index.php?title=DAT architecture#

DAT2

2 .dat file tools: http://www.teamx.ru/eng/files/utils/ F2 DAT-files packer/unpacker (DAT2), command line tool; DatMan! Light, GUI tool.

127

Patching

Another consideration on choosing a packaging algorithm is patching. After

the product gets shipped, we’ll have to maintain it, fix bugs, or upgrade. The

modified script files need to be delivered to the user and installed there.

For small projects, we can just repack all files together and release the new

package. But this is not convenient or feasible for large projects. For example, in

a game project you might have packed the scripts with other resources (image,

video, sound, map files, etc.) into one file (perhaps hundreds of megabytes). It’s

awkward for a player to download a big patch just to update several kilobytes of

script files.

Depending on your project, the patching requirement might be vital. There

are two approaches for patching: replacing an existing file in the package, or

appending one or more files to the package.

The ‘append’ approach is easier to implement. If other decisions like encryp-

tion get in the way of the ‘replace’ approach, you can use the ‘append’ approach

instead: the new file will be appended at the tail of the package and you only

need to adjust the mapping from reference name to the new file’s offset without

touching the existing old file which is buried in the package.

Several utilities for .dat format have implemented both the ‘replace’ and the

‘append’ functionalities.

Conclusion

We started from Lua’s basic facilities (require, dofile, and the binding API) to

build a flexible and powerful packaging mechanism. With Lua script packaging,

we can setup a direct mapping from the development structure to the deploy-

ment structure that hides internal organizing details from the end user. The

application will be easier to develop, deploy, and maintain.

The technique described here has been applied in two game projects and a

shareware product.

12

Objects, Lua-style

Reuben Thomas

Object-oriented programming is one of the most thoroughly explored and hotly

debated topics in Lua, as in so many languages that lack built-in objects. Con-

siderable effort has been expended to implement objects in many different ex-

isting styles. Often the aim is to integrate smoothly with existing OO systems,

in C, C++ or Java; sometimes it seems to degenerate into a “me-too” exercise.

Rarely is the question approached from the other direction: what is the most

Lua-ish way to do objects? In a sense, the answer is already present: the lan-

guage already has basic support for object orientation, with tables as objects,

the table:function() syntactic sugar for method dispatch, and the __index

metamethod for inheritance. However, there is no built-in instantiation or sub-

typing mechanism.

I present a prototype-based object implementation in 35 lines lines of Lua

(most of which is actually just three important functions on tables). Although

the framework is based on a consideration of the design of Lua, rather than that

of any particular OO system, it has been successfully used with existing code,

in particular to wrap non-OO C code into easier-to-use OO Lua. It fits well with

the Lua philosophy, and adds little overhead, in space, time or complexity.

Having presented, examined and discussed the model and implementation, I

end by re-examining the justification for programming in an OO style in Lua at

all, and suggest that it is needed less often than one might think.

130

12 · Objects, Lua-style

The object model

It’s so simple that there’s little more to do than state the obvious. First, note that

there is no a priori distinction between objects and classes, though it is common to make “class objects” that are used to create all objects of a class, and for no

other purpose. Below, class objects’ names are capitalized. Similarly, fields and

methods are only distinguished by type: a field which is a function, or a table

with a __call metamethod, is usually considered to be a method.

Create an object: object = prototype{value, ...; field = value, ...}

object is created by cloning prototype (typically, a class object). The table

passed to it is used to initialize fields as follows: keyed fields are straight-

forwardly initialized, while unkeyed fields (to be precise, non-negative in-

teger fields) are assigned to the fields whose keys are listed in the prototype

object’s init field, as a convenience (any excess arguments are simply kept

as a fields with numeric indices). This is best illustrated:

Point = Object{_init = {"x", "y"}}

p = Point{-5, 3; color = "blue"}

assigns −5 to p’s x field, 3 to its y field, and “blue” to its color field. (Note

that we use a table rather than a normal argument list so that both named

and unnamed fields may be conveniently initialized.)

Strictly speaking, the above is only true when the prototype object’s _code

field is unmodified. It may usefully be overridden to add initialization code.

An object has no record of its prototype. One can be made explicitly, or a

_prototype field could be set by the default _clone method.

By convention, fields have string keys, and private fields have keys starting

with an underscore.

Since private fields aren’t hidden, it’s up to the

programmer to ensure they don’t clash.

Access field: object.field

Since object fields are normal table entries, the standard syntax is used to

read and write to them.

Call method: object:method(...)

As for field access, method invocation works using the standard syntax.

Call class method: Class.method(object, ...)

The obvious way to call a class method is used: simply use dot rather than

colon notation, to pass the object explicitly.

A judicious justification

By now, some readers are probably verging on apoplexy, either because of the use

of prototypes, or because of the object model’s extreme simplicity. Such readers

131

may be particularly irritated that the design was foisted on them without mo-

tivation or justification. I did this because I thought it better to let the design

speak for itself before wading into the inevitable controversy. Nonetheless, the

design is a good general-purpose object implementation for Lua:

It is simple. The simple, even minimal, design fits with the Lua philosophy. For

every OO devotee bemoaning its naivety there will be a Lua purist who

thinks it’s superfluous.

Prototypes are natural in Lua. Prototypes sit well with Lua’s weakly-typed and

dynamic nature.

It works for ad-hoc wrapping. . . As well as working for pure Lua programs, this

object model can be used for ad-hoc wrapping of other object models. I’ve

used it to make OO interfaces to C structures, for example (sadly, it’s not

code I can share).

. . . but doesn’t claim to be the one true way. If you’re trying to make an extensive, easy-to-use, and above all automatic wrapper for another object model,

you should be rolling your own Lua implementation, because it’s easy and

will work better than compromising with an existing model: this is a case

in which having multiple OO implementations is a good thing.

A delightful detour

Our object implementation needs some basic functions which should be in any

Lua programmer’s toolbox: clone, merge and rearrange. The following imple-

mentations are taken from the stdlib project (http://luaforge.net/projects/

stdlib). The code verges on the trivially simple; precisely for this reason I repro-

duce it here for the reader’s enjoyment. Of crucial importance is that all three

routines are functional: they do not have any side effects. Though Lua is an

imperative language, it is well-suited to a functional style, which in my opinion

should be used whenever applicable, as it encourages clear, robust and re-usable

code.

clone makes a shallow copy of a table, including any metatable:

function clone(t)

local u = setmetatable({}, getmetatable(t))

for i, v in pairs(t) do

u[i] = v

end

return u

end

merge merges two tables. The merge, like assignment, goes right to left: fields

of the second argument override those of the first, but the result’s metatable,

if any, is that of the first argument. The left-hand argument, though, is not

overwritten.

132

12 · Objects, Lua-style

function merge(t, u)

local r = clone(t)

for i, v in pairs(u) do

r[i] = v

end

return r

end

rearrange rearranges the keys of a table. Its first argument is a map from old

keys to new keys, and its second argument is a table. Only the keys mentioned

in the map are rearranged.

function rearrange(p, t)

local r = clone(t)

for i, v in pairs(p) do

r[v] = t[i]

r[i] = nil

end

return r

end

(Note that both here and later I omit distracting details important in a

production implementation, such as packaging the code as a requirable module.

This is done in stdlib.)

Implementation

Given the functions above, the actual object implementation is brief.

Object = {

_init = {},

_clone = function(self, values)

local object = merge(self, rearrange(self._init, values))

return setmetatable(object, object)

end,

__call = function(...)

return (...)._clone(...)

end,

}

setmetatable(Object, Object)

The careful reader will want to check that the innocuous-looking first line of

_clone really does what it should, and note that the odd-looking implementation

of __call really is correct: the first (...) adjusts the list to one element, the

object, while the second passes the entire argument list to _clone, which is really

a method, so its first argument is indeed the object itself.

133

Weaknesses

The default _clone could be made to discard excess numbered initializers, but

that feels un-Lua-ish, as it imposes behavior that is not required for correct

functioning.

There are also some obvious major omissions. First, since our objects can be

indexed just like ordinary Lua tables, there’s nothing to stop the programmer

treating them as such. In other words, we lack information hiding, one of the

main planks of object orientation. I don’t think this is a problem, however. Lua

is not designed for opacity, and is not a good choice when strong type discipline

is required.

(This is not to ignore Lua’s excellence as a language for safe,

sandboxed scripting, which rests on its namespace control.)

Secondly, there is no multiple inheritance. With prototypes, multiple inher-

itance is often replaced by aggregation: a number of classes objects are cloned

and merged together. For example:

o = merge(c1._clone(), merge(c2._clone(), c3._clone()))

This could be abbreviated

o = subclass(c1, c2, c3)

where subclass is defined:

function subclass(...)

local r = {}

for c in {...} do

r = merge(r, c)

end

return r

end

I didn’t include this in the package because I haven’t yet needed it.

Inconclusion

In conclusion, we can pertinently wonder whether having a general-purpose

object system in Lua is useful at all. The principle advantages of an OO style

in Lua are encapsulation of readable syntax for well-structured data types

supporting a limited range of operations. However, it’s often possible to write

shorter, clearer code without objects.

For example, when processing poorly

structured, unstructured or arbitrarily structured data, such as text, tag soup or

XML, a table-oriented approach is often clearer, and a functional style briefer.

One example of this is the utility functions used to implement objects, which

perform general table operations. The use of general-purpose functions, helps

ensure that the object model has no undefined behavior and is robust, as well as

avoiding the need to write special-purpose code.

The real conclusion is that Lua is flexible, and lends itself to a variety of

approaches. Each should be used when appropriate, but none taken too far.

13

Exceptions in Lua

John Belmonte

Despite the well known advantages of using exceptions for program errors, the

mechanism is underutilized in Lua — both in quantity and quality. One aspect of

this relates to the Lua core and standard library, which tend to raise exceptions

only in the most serious situations such as parse errors, type errors, and invalid

arguments. When exceptions are thrown, they are exclusively string values

which are not enumerated as part of the API. Tables, the primary data structure,

yield nil for a nonexistent key rather than raise an error. All of this leads to an

unspoken bias in Lua that exceptions are something to be thrown but rarely

caught — that they are serious errors which normally go unhandled. In the few

situations where we do catch them, no distinction is made with respect to the

cause of the error.

The core and standard libraries arguably work well as they are, and their

use of errors may not warrant meddling. But why are exceptions also under-

utilized within Lua programs and third party modules? One problem is the

unfriendliness of Lua’s protected call interface to programmers expecting a na-

tive try–catch construct. This in turn discourages library authors from using

exceptions for fear of alienating users. The inability to use coroutines within a

protected call also works to limit uptake by libraries.

Lua possesses the necessary building blocks for exceptions; however, rough

edges appear when one tries to assemble them. This perpetuates disuse of

exceptions and strengthens anti-exception patterns such as signaling errors

by way of return values. To break the cycle, we first need to promote idioms

and know-how for richer use of exceptions. As more Lua developers encounter

136

13 · Exceptions in Lua

the same rough spots, the necessary motivation will exist for some incremental

improvements in the core and language itself.

This gem intends to start the process by presenting some exception tools and

know-how for Lua. First we spell out a criteria as to when a function should raise

an exception versus simply return an error status. For handling the exceptions,

we present a simple try–catch idiom that works with today’s stock Lua. We then

cover why custom error objects are important and address gaps in Lua regarding

their use. Finally, we set out to find the right pattern for exception safety in Lua.

What is an error?

What failure situations should be considered a first-class error, warranting the

use of exceptions? Calling a function with invalid arguments is an obvious

error. In contrast, a negative result from a string matching function is normally

not considered an error. In between these is an expanse of various error-like

situations. What about an attempt to append to a read-only file; a failed hash

table lookup; a database conflict; or an HTTP connection failure? We need a

guideline for evaluating these.

On this subject, “Programming in Lua” suggests that if an error cannot be

easily avoided, it should be signaled with a return code rather than exception.

This logic is geared towards letting you handle error situations without the need

for a try–catch — a decidedly conservative view on the use of exceptions. What

effect does it have on a program?

Let’s consider a Lua program which outputs the length of a file given its

name on the command line:

local f = io.open(arg[1])

local length = f:seek(’end’)

print(length)

The program lacks error handling — it may be the work of a novice program-

mer or a lazy expert programmer. How does it behave when things go wrong?

Let’s try an input file, “abc”, which doesn’t exist:

$ lua file-length.lua abc

lua: file-length.lua:2: attempt to index local ’f’ (a nil value)

The good news is that an unhandled exception occurred, causing the program

to return a non-zero exit code. This is the bare minimum behavior we need

from a command-line program on error. The error message, however, is not very

helpful. In this simple program we can look at the source code and quickly

deduce that io.open returned nil instead of a file object, causing an error on

call of the seek method. In a complex program, debugging could be much more

difficult. The file object could be passed to a different place in the program, and

perhaps not used until long after the io.open call.

137

Wrapping the io.open call in assert would address this error situation,

producing an exception with an accurate location and message.1 However, the

novice programmer didn’t consider that, and the expert programmer either

didn’t think his program would be used so foolishly, or didn’t care. In large

programs such negligence can go unnoticed until a certain obscure code path

is encountered.

Arguably, it’s better not to present the opportunity for an

oversight.

A more liberal guideline for errors is this: if a failure situation is most

often handled by the immediate caller of your function, signal it by return

value. Otherwise, consider the failure to be a first-class error and throw an

exception. The effect is to use exceptions when errors are communicated two

levels up the call stack or higher (including possible program termination). This

is intended to extract the best value from exceptions. When an error is likely to

traverse several levels, we relieve intermediate code from having to propagate

the error — a task which is error prone and clutters both code and API. On

the other hand, when a failure is usually consumed by the caller, we spare the

extravagance and expense of a throw and catch.

What is the outcome when this guideline is applied to io.open? It’s sub-

jective, but programs usually have a strong dependency on the files they open.

When a problem occurs — whether it be a full storage device, permission error,

or missing file — it tends to require handling at a high level in the program, if

it is handled at all. It’s a good guess that the error will be traveling up past the

immediate caller of the I/O function.

A simple try–catch construct

Now that we’ve planted the seed for more exceptions, we can focus on how

to catch them. As mentioned, Lua lacks the common try–catch construct for

dealing with exceptions, which may put off some programmers. By creating

something in pure Lua close to that familiar construct, perhaps we can lower

the barrier to more extensive use of exceptions.

Lua supports catching of exceptions through a functional interface, namely

pcall. It expects that the code to be attempted is itself defined as a function.

Those constraints leave us with few options — our try–catch will have to be

functional also, with the “try” and “catch” blocks of code passed in as functions.

Nonetheless, with the help of in-line anonymous functions and some creative

formatting, we can approach the feel of a native try–catch construct. Here is a

template for use of our utility function, simply called “try”:

1Wrapping a call with assert assumes it follows the convention of returning a nil and error message tuple on failure. The convention can’t be used, however, if nil or false happen to be valid outputs. It can also interfere with code readability when a function has multiple outputs and the caller elects not to wrap with assert (e.g., a function returns coordinates x and y, but on error y doubles as a message).

138

13 · Exceptions in Lua

try(function()

-- Try block

end, function(e)

-- Catch block.

E.g.:

--

Use e for conditional catch

--

Re-raise with error(e)

end)

The catch function, should it be invoked, receives the error object as an ar-

gument. After inspecting the error, it can elect to either suppress the exception

by taking no action, re-raise the existing error, or throw a different error.

A notable limitation of using functions to define our code blocks is that flow

control statements, such as return and break, cannot cross outside the try–catch.

For example, the following code would not work as expected:

function foo()

try(function()

if some_task() then

return 10 -- does not cause foo() to return 10

end

end, function(e)

-- ...

end)

return 20

end

Lua’s pcall operates by calling the function given to it. Any exception will

be trapped, returning nil and the error object. Based on that, the definition of

our try function is trivial:

function try(f, catch_f)

local status, exception = pcall(f)

if not status then

catch_f(exception)

end

end

Unfortunately coroutines do not mix well with pcall, so this will preclude

their use within our try block. The problem is well known and has various

workarounds, ranging from a pcall replacement implemented in pure Lua to

an extensive Lua core patch.

Custom error objects

Putting our new try–catch construct to use, let’s say we have a transactional

database application. If a database conflict error occurs — perhaps because two

programs tried to increment the same balance field of some record — we’d like to

retry the transaction. Coding our simplistic example:

139

try(function()

do_transaction()

end, function(e)

log(’Retrying database transaction’)

do_transaction()

end)

The issue here is that we end up retrying the transaction not only when

there is a database problem, but also for any other error. This could mask

bugs such as calling a function with the wrong arguments, producing strange

program behavior. Clearly we want to be more selective by handling only the

errors we understand and letting the rest pass through. Given the common

practice of throwing strings, however, this becomes tricky. We are faced with

fragile parsing of exception messages which may change in the future, especially

if they originate from a third party’s module.

To address this problem, we take advantage of the often-overlooked ability of

error to throw values other than strings. A table is the natural choice, leaving

room for expanded functionality by way of methods and internal state. The

database module might simply contain the following definition:

ConflictError = {}

This approach serves not only to allow positive identification of an exception,

but also to enumerate the errors which can be raised by a module — it should be

considered part of the API. Now the database module can signal a conflict with

error(ConflictError) and our catch function can be refined as follows:

function(e)

if e == db.ConflictError then

log(’Retrying database transaction’)

do_transaction()

else

error(e)

-- re-raise

end

end

A new problem is lurking however. What if the database conflict should go

unhandled? Let’s simulate the situation in Lua’s interactive interpreter:

> error({})

(error object is not a string)

Unfortunately, the uncaught exception handler which lives inside Lua’s stan-

dard interpreter refuses to do anything with a non-string error value. We’re

missing the human-readable message and call stack which are essential for lo-

cating the source of the error. The required improvement to the interpreter

is minor however: just pass the error value through tostring before invoking

debug.traceback. This change is planned for the next version of Lua. With this

change in the interpreter, and by enhancing our error object with an appropriate

__tostring metamethod, the behavior becomes:

140

13 · Exceptions in Lua

> MyError = setmetatable({},

>>

{__tostring = function() return ’My error occurred’ end})

> error(MyError)

My error occurred

stack traceback:

[C]: in function ’error’

stdin:1: in main chunk

[C]: ?

While this is a significant improvement, there is sill one detail missing from

the trace: the file name and line number of the exception. Normally, with

a string value, the error function adds this information at the point of the

exception by prefixing it to the string. For other value types the location is

omitted. While the association between an error and its location might best be

maintained by the Lua core, such a change would be substantial. A compromise

is to alter the error function to store the location in a field (assuming the value

is a table) and have it picked up by the interpreter’s handler. This location fix

and the aforementioned tostring fix are available together as a “custom errors”

patch to Lua. (See http://lua-users.org/wiki/LuaPowerPatches.)

Continuing with our database application, suppose we wish to catch any

exception specific to the database module. Or perhaps the module author decides

to distinguish between read and write conflicts using separate error types, while

our handler remains interested in both cases. It would be unfortunate to have

to spell out each error to be caught when all we mean is “any database module

error” and “any conflict error”, respectively. This suggests the need for an error

hierarchy, where we can test if a certain instance belongs to a given class of

errors.

In other languages, an error hierarchy tends to be defined by class inheri-

tance. In Lua we are free to do the same, but without a standard class system

the error values from various modules and our own code will lack a common root

and API. As a compromise, the database module author might make a utility

available for testing inheritance among the module’s own objects. The imple-

mentation should be robust, yielding a negative result for foreign values. Our

catch function then becomes:

function(e)

if db.instance_of(e, db.ConflictError) then

log(’Retrying database transaction’)

do_transaction()

else

error(e)

end

end

Note that such an inheritance test becomes mandatory should we choose to

make error objects something more than a simple table constant. For an error

141

having internal state, a new instance must be created for each exception thrown.

In that case equality cannot be used to identify the exception.

The argument for custom errors is that a human-readable error message,

while essential, should be only one component of a richer error object. Errors

should be enumerated as part of an API, providing the ability to positively

identify exceptions and perhaps locate their place within a hierarchy of errors.

Custom error objects can also serve to store arbitrary state at the time of an

exception, which may be useful for debugging and error reporting. All of this is

light work for Lua tables, although the need for hierarchy testing does present

an interoperability issue between modules.

Exception safety

With exceptions comes the issue of exception safety — proper cleanup of acquired

resources and program state when an exception does occur. Acquired resources

might include memory allocated from special pools, device handles, and mutex

objects. Consider the following simplistic function to paint a logo onto the screen:

function display_logo(display_buffer, x, y)

local canvas = allocate_canvas(50, 50)

render_logo(canvas)

display_buffer:lock()

display_buffer:copy(canvas, x, y)

display_buffer:unlock()

canvas:free()

end

During the course of this function we acquire a graphic canvas (perhaps

off-screen video memory) and a lock on the display buffer. If the render_logo

function happens to throw an exception then the canvas may not be freed

in a timely manner — it may happen automatically when the canvas value is

garbage collected, but we don’t know when that will be. More seriously, if the

display_buffer:copy call throws an exception because the input coordinates

are out of range, the display is never unlocked. Clearly, if resources like this are

going to be exposed to the scripting environment, we need a way to free them

even if an exception occurs.

Even if we decide not to expose management of critical resources to scripting,

there are common cases where we must ensure that some program state is

restored despite an error. Say we’d like the text output of a certain third party

function directed to a file, but the module has been hard-coded to use standard

output. We could work around the limitation by changing Lua’s default output

temporarily:

local out = io.output()

io.output(log_file)

somelib.do_task()

io.output(out)

142

13 · Exceptions in Lua

The problem here is that if do_task throws an exception, the default output

will never be restored. One may argue that restoring this state doesn’t matter

because the process will be terminated anyway. This overlooks the possibility

that the exception may be handled at a higher level in the execution stack, al-

lowing the program to continue. That a certain error is too dire to be intercepted

usually turns out to be a myopic view since, at the highest level of the program,

there are always options such as reattempting an operation or switching to a

failover routine. This makes proper exception safety especially important when

implementing a library, where the author cannot imagine all usage scenarios.

Now that we’ve identified the need for exception safety, how is it accom-

plished? The solutions are all variations on one theme: install cleanup code

to be run on exit of the current scope, whether that be normally or by exception.

Traditionally, programming languages have two mechanisms available for this.

One is the try-finally construct, where the scope is defined by a “try” block, and

the cleanup code placed in a “finally” block which always runs afterward. As a

language construct, however, try-finally has fallen out of fashion. To consider

why, let’s pretend that Lua supported try..finally..end and apply it to our

display_logo function:

function display_logo(display_buffer, x, y)

local canvas = allocate_canvas(50, 50)

try

-- no such thing in Lua

render_logo(canvas)

display_buffer:lock()

try

display_buffer:copy(canvas, x, y)

finally

display_buffer:unlock()

end

finally

canvas:free()

end

end

The issue is one of code readability and maintenance. A nested try-finally is

needed for each consecutive resource acquired, making the flow of the original

program difficult to follow. Moreover, although having “try” come before “finally”

is most intuitive and the common layout, it tends to maximize the distance

between acquisition and cleanup code. This problem becomes more pronounced

as the size of the function grows — to the point where the programmer cannot

see them on the screen together, and could modify one without considering the

other.

The other traditional mechanism for exception safety is the use of a custom

object which is referenced solely by the local scope. The cleanup code exists

in the destructor of the object so that it will be invoked as the value goes out

of scope. Rather than defining an ad hoc type for each cleanup situation —

which becomes verbose and a burden to maintain — the use of a generic “scope

143

manager” object is becoming common (e.g., the C++ scope guard pattern, or the

D “scope” statement). A scope manager allows the registration of arbitrary code

which will be called at scope exit. Since registration can take place multiple

times and throughout the scope, it enables natural placement of cleanup code. In

some languages it’s possible for the manager to know if the scope exited normally

or by exception, further enhancing the utility of this pattern.

Unfortunately, Lua provides no way to hook into scope exit.2 As object de-

struction is subject to the whim of the garbage collection system, the trick of

using an object referenced only by the local scope does not provide deterministic

cleanup. As in our try–catch implementation, however, it’s possible to approxi-

mate such a hook by way of an explicit function scope and pcall. We’ll use that

to create a simple scope manager in Lua for our cleanup needs.

A simple scope manager

We define a utility function “scope”, which takes a single function argument and

calls it. Within the environment of the given function, an on_exit function is

made available for registering cleanup functions. Here is how the scope utility

looks when applied to our display_logo example:

function display_logo(display_buffer, x, y)

scope(function()

local canvas = allocate_canvas(50, 50)

on_exit(function() canvas:free() end)

render_logo(canvas)

display_buffer:lock()

on_exit(function() display_buffer:unlock() end)

display_buffer:copy(canvas, x, y)

end)

end

Notice that no nesting is needed for consecutively acquired resources as in

the try-finally solution. Also, each piece of cleanup code is positioned logically so

that, as the code is read from top to bottom, one can see exactly when it becomes

active within the scope.

To round out our cleanup utility, we’ll make two more registration functions

available within the scope: on_failure and on_success. The on_failure hook

might be used to roll back a pending database transaction or other tentative

state change. Although a try–catch could be used here instead, on_failure is

more readable and avoids having the user take responsibility for re-raising the

caught error. The on_success hook will likely be least used of the three, but

2The ability to hook into scope exit is the only fundamental building block I’ve noticed as missing from Lua 5.1. I hope that this can be resolved in a future version of the language — perhaps by creating a new class of variable which notifies its value when it goes out of scope, or by adding a construct along the lines of Python’s “with” statement.

144

13 · Exceptions in Lua

again it offers more flexibility on placement of cleanup code. Here is the scope

function implementation:

function scope(f)

local function run(list)

for _, f in ipairs(list) do f() end

end

local function append(list, item)

list[#list+1] = item

end

local success_funcs, failure_funcs, exit_funcs = {}, {}, {}

local manager = {

on_success = function(f) append(success_funcs, f) end,

on_failure = function(f) append(failure_funcs, f) end,

on_exit = function(f) append(exit_funcs, f) end,

}

local old_fenv = getfenv(f)

setmetatable(manager, {__index = old_fenv})

setfenv(f, manager)

local status, err = pcall(f)

setfenv(f, old_fenv)

-- NOTE: behavior undefined if a hook function raises an error

run(status and success_funcs or failure_funcs)

run(exit_funcs)

if not status then error(err, 2) end

end

Like the try–catch implementation, this scope hook suffers from an incom-

patibility with coroutine yield, and the inability to use flow control statements

across the scope’s boundary (i.e., return, break, etc.). A more fundamental limi-

tation exists however: cleanup code itself must not raise an exception. Allowing

this would create at least two ambiguities: 1) if an exception happens in one

piece of cleanup code, should the entire cleanup contract be invalidated? 2) if

there are multiple, logically parallel exceptions, which is to be propagated? The

situation is best avoided and, in the implementation presented, its behavior is

left undefined.

A slightly different design for the scope utility would be to pass the manager

object to the user’s function as an argument. Besides eliminating the complexity

of making on_exit and the other registration methods appear implicitly within

the function, this would allow the manager to be passed to utility functions.

For example, the allocate_canvas function could take a scope manager as an

optional argument, and in that case register the canvas cleanup code for us.

On the other hand, the explicit manager variable makes the user’s code more

verbose in the simple case, and opens the door for confusion should someone try

to operate on the manager of an already expired scope.

This pattern to assist with exception safety is the final component in our bag

of exception tools. Combined with custom error objects, which allow discern-

145

ing between errors, and a try–catch construct implemented in pure Lua, pro-

grammers can explore richer use of exceptions in their programs and libraries.

Limitations and rough spots exist for sure, but hopefully this situation is tempo-

rary — the authors of Lua have a good track record of improving the flexibility

of the language and its implementation over time.

Part III

Algorithms

and

Data Structures

14

Word Ladders

Gavin Wraith

Word Ladders or Doublets is a word game whose invention has been attributed to Lewis Carroll. The idea is to transform one word into another by changing

only a single letter at each step.

Here is a simple example:

BEST → PEST → POST → POSE → ROSE → RISE → RISK

We present a small Lua program which, given a lexicon of words as a com-

mand line argument, takes a word from the standard input and prints the words

in the lexicon that can be obtained from it by such transformations. This pro-

gram is really just an excuse for presenting a more abstract application: a mod-

ule for calculating the strata of the connected component of a vertex in an undi-

rected graph.

Undirected graphs

From the earliest days of programming this game has been a vehicle for demon-

strating algorithms about undirected graphs. An undirected graph may be de-

fined as a set of vertices together with a Boolean-valued function on the set of

unordered pairs of distinct vertices, which tells which vertices are joined by an

edge. In what follows the term graph should be read as undirected graph.

If two vertices can be joined by a sequence of edges then we say that they

belong to the same component of the graph. Given a lexicon of words we can

construct a graph whose vertices are words, where two vertices are joined by an

150

14 · Word Ladders

edge if the corresponding words differ by only one letter. The Word Ladder game

is about finding a path from a given word to another.

The mathematical notion of graph is an abstraction.

The Word Ladder

game has various features which are thrown away by this abstraction — the

significance of the words, for example, or the fact that we can enumerate words

in lexicographic order. If a program is to be reusable it should be as faithful as

possible to the mathematical abstraction, and avoid building in specific features

of this or that example. However, there is a problem. Mathematicians tend to

use the notion of set in contrast to the notion of enumerated set. Programming languages, on the other hand, having developed in an era of serial processors,

are usually better adapted to handle enumerated sets. Of course, every finite set

can be enumerated in some way or other (and if you believe the axiom of choice,

every set can be totally ordered), but if a problem does not mandate a particular

enumeration it seems rather ugly to have to choose one in order to program a

solution.

The fundamental datatype in Lua is the table. Any value except nil can be

a key in a table. We can use tables to represent both enumerated sets, using

integers starting from 1 as keys, and non-enumerated sets by taking as the

elements keys with value true. Lua has two iterator functions for tables: ipairs

iterates over integer keys in order starting from 1 and so is appropriate for

enumerated sets, and pairs iterates over all keys but in no predictable order,

and so is appropriate for non-enumerated sets. This unpredictability is the

serial processor’s apology, so to speak, to the parallel world of mathematical

sets (logicians may be thinking of permutation models at this point).

Although a component of a graph may have no natural enumeration, once

we have chosen a particular vertex in it we get a natural stratification of it. The

first stratum consists just of the chosen vertex. The n-th stratum consists of

those vertices having a shortest path of n − 1 edges to the chosen vertex. Each

stratum has no particular enumeration itself, and so should be represented by a

Lua table whose values we may take to be true. The collection of all the strata

does have an enumeration, and so is represented as an array of tables.

Listing 1 shows a module, graph, that calculates the array of strata, given

a graph and a vertex of it. A graph is a table with keys vertex and edge. The

value for vertex is a true-valued table. The value for edge is a Boolean valued

function on pairs of distinct keys of the vertex table. It should be symmetric,

that is to say if g is a graph then g.edge(x,y) and g.edge(y,x) should have the

same value for any keys x, y of g.vertex.

The function graph.component makes a copy of the table mygraph.vertex and

points the local variable vertex at it. It initializes the strata array to a single

stratum containing just the start node, and then removes the start node from

vertex. It defines a local function, more, which searches the nodes of vertex to

see if they are joined by an edge to the current stratum. If they are, they are

removed from vertex (which is why we had to make a copy of mygraph.vertex)

and put into the next stratum. This avoids redundancy. The more function is

called repeatedly until no more strata can be found.

151

-- graph

--[[component of start node in graph as a list of sets of nodes]]

local insert = table.insert

local pairs = pairs

module "graph"

component = function(mygraph,start)

local vertex,edge = {},mygraph.edge

for node,val in pairs(mygraph.vertex) do

vertex[node] = val

end -- for

local strata

= {{ [start] = true }}

vertex[start] = nil

local more = function()

local new,change = {},nil

for x,_ in pairs(strata[#strata]) do

for y,_ in pairs(vertex) do

if edge(x,y) then -- x cannot equal y

change = true

new[y] = true

end -- if

end -- for

end -- for

if change then

insert(strata,new)

for x,_ in pairs(new) do vertex[x] = nil end -- for

end -- if

return change

end -- function

repeat until not more() -- add new vertices while possible

return strata

end -- function

Listing 1. The graph module

152

14 · Word Ladders

The number of calls to the function mygraph.edge is given by

X x2

nm + n2 +

i

2

i

where n is the number of vertices in the component, m is the number of vertices

not in the component and xi is the number of vertices in the i-th stratum. This

number cannot be made smaller.

Word Ladder game

The Word Ladder game itself can be coded as in Listing 2. The program uses

a lexicon of words in a file whose pathname is passed in as a command line

argument, and a starting word which is input by the user. It outputs the words

in the strata in order.

The program asks for the start word to be input, and transforms it to lower

case. The lexicon is a file consisting of lower case words separated by white space

and newlines. The table called lexicon is set to have as keys the words found

in the lexicon file to be of the same length as the start word, as this version of

the game does not allow the length of words to change. A function string.vary

is added to the string library which creates a pattern matched by any word that

differs in only one letter from its first argument, whose position is the second

argument. The reason for adding it to the string library is a trivial aesthetic

one: it makes the pretty colon notation available. The function differby1 is

a Boolean-valued function that tells when words differ by only one letter. The

graph wordgraph is defined by the lexicon table and the differby1 function. A

function printout is defined to print out the contents of the strata; it converts

all words to upper case and first prints the stratum level. The graph library is

loaded and finally the result is output.

Summary

The graph module and the ladder program that uses it are tiny and straightfor-

ward, and in themselves unremarkable. They are simply pegs on which to hang

some observations. Readers are, of course, free to quarrel with my personal

preferences.

• I like to use comments, but not too many. It is pointless to overcomment if

meaningful variable names tell the story.

• I like to indent to make chunks easily identifiable.

• I like to comment the end keyword whenever feasible.

• I like to reduce the use of global variables as much as possible.

153

-- ladder

-- arg[1] holds pathname of lexicon

do

local read,lines = io.read,io.lines

print "Enter a word from the lexicon"

local startword = (read()):lower()

local n = #startword

local pat = ("%a"):rep(n) -- pattern for words of same size

local lexicon = {}

for line in lines(arg[1]) do

line:gsub(pat,function(word) lexicon[word] = true end)

end -- for

-- add vary to string library

string.vary = function(s,i)

-- pattern - vary i-th char

return s:sub(1,i-1).."."..s:sub(i+1,-1)

end -- function

local differby1 = function(x,y)

local n = #x

for i = 1,n do

if y:match(x:vary(i)) then return true end -- if

end -- for

return false

end -- function

local wordgraph = { vertex = lexicon; edge = differby1; }

local printout = function(strata)

for i,stratum in ipairs(strata) do

for word,_ in pairs(stratum) do

print(i,": ",word:upper())

end -- for

end -- for

end -- function

require "graph"

printout(graph.component(wordgraph,startword))

end -- do

Listing 2. The main program

154

14 · Word Ladders

• I am proud that in Lua functions are first-class values. For that reason

I make it clear that function definitions are assignments, and I do not use

the syntactic sugar provided to conceal this fact, lest it be misinterpreted

as apologetic. If you have got it, flaunt it!

The choice of data structures and the division of labour between main program

and required modules should follow from analysis of the abstractions thrown up

by the problem. In this case the analysis is the trivial fact that a vertex in a

graph stratifies the component in which it lives. It tells us loud and clear that

we need a function that returns a list of tables and not just their union, as a

superficial reading of the task might suggest.

15

Building Data Structures

and Iterators in Lua

Luis Carvalho

Besides being lightweight and fast, Lua is also highly regarded for being an

elegant and expressive language. The main purpose of this gem is to reinforce

this impression by showing how powerful and straightforward the concerted

application of tables, metamethods, and coroutines is in the implementation of

complex data structures and their iterators.

Here we implement a graph object module where graphs are modeled us-

ing a vertex set, a weighted edge/arc set and adjacency lists (set objects are as

described in “Programming in Lua” (PiL)). Graph vertex sets can be iterated

by depth first search (DFS), breadth first search (BFS), and topological sorting,

which illustrate well the application of closures and coroutines. Moreover, classi-

cal routines for shortest paths from a vertex and minimum spanning tree (MST)

are also provided. These routines require additional data structures in order to

achieve optimal time complexity: queues (similar to the ones implemented in

PiL) are used in the BFS iterator, while heaps and partition sets (both as trees

with special properties) are used in the MST routine. Each data structure, on its

own, comprises an individual pure Lua module whose methods satisfy the usual

colon calling convention for objects. A few examples and direct applications of

the routines are also presented.

156

15 · Building Data Structures and Iterators in Lua

Introduction

The study and development of algorithms and data structures are tightly cou-

pled together: appropriate data structures are the core of well designed, optimal

algorithms. Besides designing suitable data structures, it is also important to

use a programming language that allows easy specification and implementation

of a data structure and that is rich and expressive enough to favor the realiza-

tion of abstract concepts and possible future extensions. This gem aims to show

that Lua is such a language: with its many appealing resources — including

a powerful unique data structure building block, the table, metamethods, and

coroutines — we can implement many data structures effortlessly.

The main data structure portrayed in the text is a graph, but we also present

others in order to solve some classical graph problems efficiently. A complete

listing of all routines, which are encapsulated in modules, one per data struc-

ture, can be found in the gem repository.

Although we provide an explanation for each method in the text, we try to

be as terse as possible to keep the text short. Familiarity with data structures

and design of algorithms is highly desirable — this way you can concentrate in

enjoying the Lua code! — but not necessary; however, the reader should refer

to many excellent books that cover these topics for more details1. We should

mention the excellent “Programming in Lua” (PiL)2, by Roberto Ierusalimschy,

one of Lua’s creators: we make many references to it, and try to draw inspiration

from it whenever possible.

Queues

Our first data structure is a queue, a list in which elements are always inserted

to the front and retrieved at the rear, that is, in a first-in, first-out fashion.

Although simple, our queue implementation will be useful later on this chapter

when we talk about graph traversal and can serve as good warm-up exercise.

The implementation comprises a module Queue, very similar to the one in PiL: a

queue object is a table with two pointers, first and last, indicating the current

positions of the queue’s front and rear. A queue is then initialized by

local modenv = getfenv() -- module environment

function new ()

return setmetatable({first = 1, last = 0}, {__index = modenv})

end

where we use __index to enable colon call notation in our objects. Note that

getfenv takes 1 as default argument and returns the current, module, environ-

1We particularly recommend the classical “Design and analysis of computer algorithms” , by Aho, Hopcroft and Ullman, and the more modern “Introduction to algorithms” , by Cormen, Leiserson, Rivest, and Stein.

2PiL’s first edition is available online at http://www.lua.org/pil.

157

ment. The methods in the environment are the typical insert and retrieve, as

presented in PiL3,

function insert (Q, v)

assert(v ~= nil, "cannot insert nil")

local last = Q.last + 1

Q[last] = v

Q.last = last

end

function retrieve (Q)

local first = Q.first

assert(Q.last >= first, "cannot retrieve from empty queue")

local v = Q[first]

Q[first] = nil -- allow GC

Q.first = first + 1

return v

end

and the useful check:

function isempty (Q) return Q.last < Q.first end

Thanks to the __index metamethod we are able, for example, to simply issue

Q:retrieve() instead of Queue.retrieve(Q). We will be following this practice

of defining __index as the module environment for all objects in this chapter,

and so the queue module gives the reader a good opportunity to get familiarized

with this prototype-based convention4.

Heaps

We now implement a heap data structure, which is a binary tree stored in

an array. For each node in the tree we associate a record numeric field such

that the tree satisfies the (min) heap property: for every node v, record(v) ≤

min{record(l), record(r)}, where l and r are the left and right children of v. If a

tree is a heap, its root contains the smallest record over all nodes in the tree.

Heaps have many applications, but in this gem we focus on heaps as priority

queues, that is, as a data structure for keeping a set of objects ordered by their record field. Priority queues will be at the core of our solutions to the minimum

spanning tree and shortest path problems in a latter section about graphs. Since

records in a heap represent some object feature — such as the time of occurrence

of events in a simulation engine — it is then desirable to have a correspondence

between nodes in the heap and objects. We use two additional fields for this

3They are called push and pop in PiL.

4For a more thorough presentation, check PiL’s chapter on object-oriented programming, more specifically on the section about classes.

158

15 · Building Data Structures and Iterators in Lua

intent: key stores object labels and ref stores references from keys back to

heap nodes. Of course, key[ref[k]] = k and ref[key[n]] = n for all keys k

and nodes n. Since this correspondence between objects and the heap will be

useful for us later on, our heap methods are implemented with it in mind.

Heaps are similar to queues since we can insert elements, retrieve them,

and check the heap for emptiness. Insertions are performed at the leaves, filling

each level of the tree before going to the next, while retrieval is performed at

the root. Heaps differ from queues in that they also have an update operation

that rearranges the heap after some record is decreased. Still, our main concern

should be for the tree to keep the heap property after each of these operations.

We create heap objects with

function new ()

return setmetatable({record={}, key={}, ref={}}, {__index = modenv})

end

where our tree is represented by integer-keyed tables record, key and ref in the

following way: if i is a position representing a node in the tree, then 2i is the left

child of i, 2i + 1 is the right child of i, and thus ⌊i/2⌋ is the parent of i. Since we

are going to need parents of nodes a lot, it is convenient to provide:

local function parent (n) return (n - n % 2) / 2 -- floor(n / 2) end

The update method is listed below; we maintain the heap property by perco-

lating the node i with key k and new record value v < record[ref[k]] up the

tree until the property is again satisfied:

function update (H, k, v)

local record, key, ref = H.record, H.key, H.ref

local i = ref[k]

local p = parent(i)

while i > 1 and record[p] > v do -- climb tree?

-- exchange nodes

record[i], key[i], ref[key[p]] = record[p], key[p], i

i, p = p, parent(p)

end

record[i], key[i], ref[k] = v, k, i -- update

end

Note how we actually update three trees, one for each field in the heap, as we

exchange parent and child nodes, but use record exclusively for comparisons.

Insertions are now straightforward; we simply insert a new leaf and call update

to maintain the heap property:

function insert (H, v, k)

local ref = H.ref

assert(ref[k] == nil, "key already in heap")

ref[k] = #H.record + 1 -- insert reference

update(H, k, v) -- insert record

end

159

Updates and insertions are simple because there is a unique path from any

node to the root and so we just need to follow that path up looking for a suitable

position to place our new or updated object. Retrieval is a bit more complicated

since it comprises extracting the root and swapping the rightmost leaf (the last

position in the array) by the root: both left and right subtrees from the root are

still heaps, but now the root might be violating the heap property. The task of

fixing the tree in case this happens is handled recursively by heapify:

local function heapify (record, root, key, ref, n)

local left, right = 2 * root, 2 * root + 1 -- children

local p, l, r = record[root], record[left], record[right]

-- find min := argmin_{root, left, right}(record)

local min, m = root, p

if left <= n and l < m then min, m = left, l end

if right <= n and r < m then min, m = right, r end

if min ~= root then -- recurse to fix subtree?

record[root], record[min] = m, p -- exchange records...

key[root], key[min] = key[min], key[root] -- and keys

if ref ~= nil then -- fix refs?

ref[key[root]], ref[key[min]] = root, min

end

return heapify(record, min, key, ref, n)

end

end

Complementary to update, heapify percolates down any offending node root

at a subtree root until the whole tree becomes a heap. Note that heapify is

tail recursive, and so we should profit from Lua’s implementation of proper tail

recursion to allow an arbitrary number of calls. The use of a ref field is optional

in heapify, but we need to specify the size of the heap as a last argument; this

interface is more general, and will be justified when we talk about heapsort.

Finally, we can now extract the minimum record from the heap with

function retrieve (H)

local record, key, ref = H.record, H.key, H.ref

assert(record[1] ~= nil, "cannot retrieve from empty heap")

local n = #record -- heap size

local minr, mink = record[1], key[1]

record[1], key[1], ref[key[n]] = record[n], key[n], 1 -- leaf to root

record[n], key[n], ref[mink] = nil, nil, nil -- remove leaf

heapify(record, 1, key, ref, n - 1) -- fix heap

return minr, mink

end

while the check for emptiness is implemented by

function isempty (H) return H.record[1] == nil end

160

15 · Building Data Structures and Iterators in Lua

Heapsort

We could not talk about heaps and not mention one of their main applications:

to sort an array. The concept is an ingenious derivation from retrieve: we can

sort an array in-place by iteratively replacing the root by the rightmost leaf —

the first position by the last position in the array — and not actually removing

the root, but simply reducing the heap size. By performing the whole sorting

in-place, we do not need to insert elements either5. However, since we have a

min heap, we sort decreasingly; to sort increasingly we would need a max heap,

which can be trivially obtained by swapping record comparisons in heapify and

insert. Our heapsort routine does not use a ref field and needs to explicitly set

the size of the heap in heapify since no elements are removed from the heap:

function heapsort (t)

local n = #t -- heap size

local k = {} -- key: position in t

for i = 1, n do k[i] = i end

for i = parent(n), 1, -1 do -- build heap

heapify(t, i, k, nil, n)

end

for i = n, 2, -1 do -- sort in-place

k[1], k[i] = k[i], k[1] -- exchange keys...

t[1], t[i] = t[i], t[1] -- ...and records

heapify(t, 1, k, nil, i - 1)

end

return k

end

The motivation behind using the key field to store the original position of the

entries in the array to be sorted is to illustrate an interesting use of key: we can

recreate the original state of a sorted table, as in

function permute (t, o)

local p = {}

for i = 1, #t do p[o[i]] = t[i] end

return p

end

local k = heapsort(t) -- sort

-- do something with t

t = permute(t, k) -- restore

5You might ask: why not implement heapsort using insertions and retrievals? This is possible, of course, but less efficient: the “build heap” loop in heapsort has complexity O(n), n being the heap size, as opposed to O(n log n) if the heap is built using insert.

161

Partition sets

Consider now a set S and a partition {Si} over S, that is, the sets Si are disjoint,

∩iSi = ∅, and their union is S, ∪iSi = S. We want an efficient way to implement

two operations for partition sets: merge two sets into their union and, given an

element e ∈ S, find the set to which e belongs.

A good representation for partition sets is a forest where every set is repre-

sented by a tree and is identified by the element at the root of its tree. This way,

merging two sets S1 and S2 requires a simple attachment of the tree represent-

ing S1 as a subtree in S2. In addition, finding the set that contains a random

element involves a tree climbing routine from the element up to the root of the

tree representing the containing set. As we will see shortly, to guarantee effi-

ciency we also need to perform these operations according to some rules, but for

now let’s just assume that we need to keep track of the number of elements in

each set.

Our partition sets are represented by a table with two fields: card, a table

storing the cardinality of each set, and parent, a table storing the parent of

an element in its containing set. An element e is the root of a set S if and

only if card[e] contains the number of elements in S — e represents S — and

parent[e] == nil. New partition objects are created with

function new ()

return setmetatable({card = {}, parent = {}}, {__index = modenv})

end

while set takes a new element and creates a new set in the partition containing

only the argument,

function set (P, e)

assert(e ~= nil, "set cannot contain nil")

assert(P.card[e] == nil and P.parent[e] == nil,

"element already in partition")

P.card[e] = 1 -- e is root of new set

end

and sameset checks if two elements belong to the same set in the partition:

function sameset (P, e1, e2)

return find(P, e1) == find(P, e2)

end

Now we need to provide find. As stated before, to find to which set an

element e belongs, we just need to climb the tree from e up to the root. It is

clear that the shorter the tree the more efficient this task is; to favor this feature

in our sets, we then implement a compacting rule where all the internal nodes

in the path from e up to the root are collapsed into one level with the internal

nodes becoming children of the root:

162

15 · Building Data Structures and Iterators in Lua

function find (P, e)

local parent = P.parent

assert(P.card[e] ~= nil or parent[e] ~= nil,

"element not in partition")

-- climb tree up to root r

local r = e

while parent[r] do r = parent[r] end

-- compacting rule for tree paths

local u = e

while u ~= r do -- collapse nodes from e to r

u, parent[u] = parent[u], r

end

return r -- root identifies set

end

For merge we also try to keep the tree shorter by applying a weighting rule:

when merging two sets, attach the smaller set to the larger set. This rule

requires the use of card to keep track of set sizes. The best way to connect

the two sets is then to redefine the root of the smaller set as a child of the root

of the larger set.

function merge (P, e1, e2) -- merge sets containing e1 and e2

local s1, s2 = find(P, e1), find(P, e2)

if s1 ~= s2 then -- merge needed?

local card = P.card

-- weighting rule: merge with largest set

if card[s1] > card[s2] then s1, s2 = s2, s1 end

P.parent[s1] = s2

card[s1], card[s2] = nil, card[s1] + card[s2]

end

end

You might be asking: how much do we actually gain by incorporating the

path compression and weighting rules? After all, the path compression rule is

responsible for roughly half of find’s running time, while we need to allocate

extra space for card to implement the weighting rule. A rigorous analysis is

beyond the scope of this text, but the gains can be summarized, for O(n) merge

and find operations, as follows: with no rules it costs O(n2) time units, with only

the weighting rule it becomes O(n log n), and with both rules the complexity

becomes almost linear6. More details can be found in “Design and analysis of

computer algorithms”, by Aho, Hopcroft and Ullman.

6It is actually O(nG(n)), where G is a very slowly increasing function related to a functional inverse of the Ackermann’s function.

163

Graphs

There are many ways to represent a graph data structure, each being better

suited to a particular application. For our graph object we adopt a representa-

tion using adjacency lists — or better, tables — where the keys are vertices and

the values hold adjacency relations.

Vertices can be of any type but nil, and are stored in the graph’s vertex

set, vset; similarly, we store the edges or arcs of a graph, if it is undirected or

directed respectively, in its edge (or arc) set eset. To avoid redundancy, vset is

actually the adjacency list: vset[v] is a table where each key is a neighbor u

of v and the corresponding value is the edge {u,v}, also a table. The edge set

eset can be a regular set as presented in PiL — keys are edges and values are

true — or values can hold edge weights.

Graphs are created with our (now canonical) new method:

function new()

return setmetatable({vset = {}, eset = {}}, {__index = modenv})

end

As basic graph operations we offer the addition of vertices,

function addvertex (G, v)

assert(v ~= nil, "cannot add nil as vertex")

local vs = G.vset

assert(vs[v] == nil, "vertex already in graph")

vs[v] = {} -- new adjacency list

end

and edges to a graph,

function addedge (G, v1, v2, w)

assert(v1 ~= nil and v2 ~= nil, "cannot add nil as vertex")

local vs = G.vset

-- add v1 and v2 if not in G

if not vs[v1] then vs[v1] = {} end

if not vs[v2] then vs[v2] = {} end

-- update v1 and v2 adjacency lists

local e = {v1, v2} -- new edge

vs[v1][v2] = e -- v1 -> v2

vs[v2][v1] = e -- v1 <- v2

-- update edge set

G.eset[e] = w or true

end

Addition of arcs can be handled by a very similar method where the adjacency

lists are updated only according to the arc direction.

It is useful to have some iterators for traversing the vertex and edge sets,

and the neighborhood of a vertex v — the set of vertices adjacent to v. We can

derive such iterators by mimicking the stateless pairs iterator:

164

15 · Building Data Structures and Iterators in Lua

function vertices (G) return next, G.vset, nil end

function edges (G) return next, G.eset, nil end

function neighborhood (G, v)

local adjv = G.vset[v]

assert(adjv ~= nil, "vertex not in graph")

return next, adjv, nil

end

It should be noted that, even though pairs can be used instead of each

iterator, we explicitly define them to keep our following graph methods abstract,

that is, independent of our particular underlying graph implementation: if we

later decide to change the graph representation to solve some problem more

efficiently our higher level algorithms should require minimum modifications,

or even none at all.

In the next sections we treat a few classical graph problems that arise from

many applications. All problems have a theme in common, graph connectivity,

that allow us to exploit our chosen adjacency list representation.

Graph search

Many graph related problems involve a search through the graph’s vertex set in

order to, say, verify some property or compute some desired quantity. It is often

the case when the vertices should be visited in some specific order as required

by the task at hand. The most common orderings are provided by two graph

search procedures: depth-first search (DFS) and breadth-first search (BFS).

Both procedures take a vertex in the graph as a starting point. As the

names suggest, depth-first search follows some edge leaving the current vertex

in the search as deep as possible, backtracking to explore more vertices when

the options are exhausted, while breadth-first search explores the neighborhood

of the current vertex before attempting to branch the search further.

Let’s address DFS first; check Listing 1. Our DFS method is actually an

iterator wrapped around an auxiliary routine, search. Thanks to Lua’s corou-

tines, implementing such an iterator is simple even when it involves a recursive

routine since we can yield from it. In dfs, visited is a control variable that

keeps a set of already visited vertices. As an iterator, dfs is not as efficient

as neighborhood since a new closure is created for each search; however, even

though dfs is algorithmically more complex, it is, at the same time, semantically

as simple as neighborhood thanks to Lua’s generic for.

A subgraph S of a graph G is a graph such that V (S) ⊆ V (G) and E(S) ⊆

E(G), where V (G) and E(G) are the vertex and edge sets of G respectively. A

component C of a graph G is a maximal connected subgraph of G: any two

vertices in C have a path connecting them, that is, C is connected, and any

vertex not in C has no edges to a vertex in C, that is, if we add any other vertex,

C is not connected anymore. An important application of graph searches is to

165

local function search (G, v, visited)

visited[v] = true -- mark v as visited

coroutine.yield(v)

for u in G:neighborhood(v) do -- branch search

if not visited[u] then -- search deeper?

search(G, u, visited)

end

end

end

function dfs (G, s)

assert(G.vset[s] ~= nil, "vertex not in graph")

local visited = {} -- control variable (set)

return coroutine.wrap(function()

return search(G, s, visited)

end)

end

Listing 1. Depth-first search for G, starting at s.

identify the components of a graph; the following method is our first application

of dfs and returns a table containing vertex sets for each component of a graph:

function components (G)

local comp = {} -- holds components

local S = {} -- unvisited vertices

for v in G:vertices() do S[v] = true end

for v in pairs(S) do

local C = {} -- component set

for u in G:dfs(v) do -- or bfs

C[u] = true -- add to C

S[u] = nil -- remove from S

end

comp[#comp + 1] = C

end

return comp

end

We can also have true iterators, as discussed in PiL, by implementing a

factory that returns a DFS iterator starting at a provided vertex as a closure on

the graph, the control variable, and the auxiliary search routine; see Listing 2.

The iterator takes two functions as optional arguments, visit and finish, that

perform some action as the vertex is visited and finished, respectively. By a

finished vertex we mean a vertex that had all its neighbors traversed by DFS.

Using dfs from the listing above, we can, for example, print the order in which

the vertices are discovered by DFS from a vertex s by just creating an iterator

dfser = dfs(G, s) and then calling dfser(print). This last dfs has a strong

166

15 · Building Data Structures and Iterators in Lua

function dfs (G, s)

assert(G.vset[s] ~= nil, "vertex not in graph")

local dfsaux, visited

dfsaux = function (v, visit, finish)

visited[v] = true

if visit ~= nil then visit(v) end

for u in G:neighborhood(v) do

if not visited[u] then

dfsaux(u, visit, finish)

end

end

if finish ~= nil then finish(v) end

end

return function(visit, finish) -- true iterator

visited = {}

dfsaux(s, visit, finish)

end

end

Listing 2. DFS iterator factory.

functional flavor — it is a higher order function (HOF) that returns another

HOF! — that corroborates to its powerful semantics but increased complexity

when compared to previous iterators.

A more elaborate application is to sort a directed acyclic graph (DAG) topo-

logically, that is, to order its vertices such that there are no arcs from u to v if u is after v in the ordering. For example, if in a graph G the vertices represent

subtasks and the arcs precedence rules, a topological sort of G gives a valid way

of executing a procedure.

Listing 3 provides a simple routine for topologically ordering a graph G: we

just need to execute a depth-first search on G that computes the finishing times

of its vertices and then sort the vertices decreasingly by the finishing times. We

need to keep a notfinished set because dfs iterates only over one component at

a time, and we need an ordering over all vertices.

While depth-first search required a recursion to perform the search, breadth-

first search can be implemented iteratively using a queue, as presented in

Listing 4; note that bfs requires our Queue module.

The BFS iterator can still be viewed as a wrapper around an auxiliary

routine, an iterative one, nevertheless. Observe the similarities between dfs

and bfs: while the former performs the search as soon as it finds an unvisited

vertex, the latter stores the unvisited neighbors in a queue for later traversal.

As a matter of fact, we can construct an iterative version of DFS by using an

explicit stack to traverse the graph, and so the only difference between BFS and

DFS is how the neighborhood of a vertex is visited: if in first-in-first-out order,

as in BFS, or in last-in-first-out order, as in DFS.

167

function toporder (G, s)

local s = s or next(G.vset) -- optional start

local n = 0 -- number of vertices

local i = 0 -- finishing time

local f = {} -- vertices in decreasing time

local notfinished = {} -- set

for v in G:vertices() do

n = n + 1

notfinished[v] = true

end

local function finish (v)

if notfinished[v] then

notfinished[v] = nil -- finish v

i = i + 1 -- advance time

f[n - i + 1] = v -- store reverse order

end

end

while s do -- any unfinished vertex left?

dfs(G, s)(nil, finish)

s = next(notfinished) -- start at other component

end

-- return iterator

i = 0

return function ()

i = i + 1

return f[i]

end

end

Listing 3. Topological sort of graph G, optionally starting at s.

Minimum spanning trees

Given a weighted undirected graph G, the minimum (weight) spanning tree

(MST) problem asks us to find a spanning tree T of G — an acyclic connected

subgraph of G that covers all its vertices — such that the sum of the weights of

the edges in T is minimum over all possible spanning trees of G7.

The MST problem can be solved efficiently by Kruskal’s algorithm: initially

put each vertex of G in a component and tentatively add an edge to the tree if the

edge is not incident to vertices in the same component; when an edge is added to

the solution we merge the components the edge connects proceeding iteratively

until all edges have been visited. Of course, for the resulting spanning tree to

have minimum total weight we need to visit the edges in some fashion, and that

turns out to be in increasing weight order.

7In general, if G is not connected, we want the minimum spanning forest of G.

168

15 · Building Data Structures and Iterators in Lua

function bfs (G, v)

assert(G.vset[v] ~= nil, "vertex not in graph")

local visited = {} -- control set

local Q = Queue.new()

return coroutine.wrap(function()

Q:insert(v)

while not Q:isempty() do -- any vertex left?

local u = Q:retrieve()

if not visited[u] then

visited[u] = true

coroutine.yield(u)

end

for w in G:neighborhood(u) do

if not visited[w] then Q:insert(w) end

end

end

end)

end

Listing 4. Breadth-first search for G, starting at s.

Listing 5 presents an implementation of Kruskal’s algorithm where we use

Heap to keep the edges sorted by weight, and Partition to handle the compo-

nents. By using our optimized heap and partition operations, mst(G) achieves

the optimal complexity of O(m log m), where m is the number of elements in

G.eset8.

Shortest paths

Consider now two distinct vertices u and v in a connected graph G: there may be

many paths in G connecting them. However, if we attribute to each edge in G a

positive cost, we are usually more interested in a shortest path between u and v, that is, a path of minimum total cost. The cost of a path between u and v is

called the distance between u and v, and comprises the sum of the costs over all

edges in the path.

Dijkstra’s algorithm finds the shortest paths between a source v and all other

vertices in G9. His algorithm iteratively marks a vertex once its shortest path

to the source is known, and visits the unmarked vertices in order of increasing

distance to the source. Since the costs are positive, shortest paths leading to

unmarked vertices can only pass through already marked vertices. Thus, we can

compute the distance from an unmarked vertex to the source by only considering

the known distance from one of its marked neighbors to the source.

8You cannot sort a list of n elements with complexity less than O(n log n).

9Even if we are only interested in the shortest path between two specific vertices, an algorithm for that problem would not be more efficient in the worst case than the best single-source algorithm.

169

function mst (G)

local T = Graph.new() -- min spanning tree

local VS = Partition.new() -- component sets

local H = Heap.new() -- keep edges ordered by weight

-- fill heap with all edges

for e, w in G:edges() do H:insert(w, e) end

-- each vertex in a component

for v in G:vertices() do VS:set(v) end

while not H:isempty() do -- any edge left?

local w, e = H:retrieve() -- min weight edge

local v1, v2 = unpack(e)

if not VS:sameset(v1, v2) then

-- v1 and v2 in different components?

VS:merge(v1, v2) -- merge components

T:addedge(v1, v2, w) -- grow tree

end

end

return T

end

Listing 5. Kruskal’s algorithm: minimum spanning tree of a graph G.

We present shortestpath in Listing 6. A heap H is used to keep track of

the vertices to be visited in order of increasing distance; as we retrieve from H

we mark the vertex that minimizes the distance to the source. The previous

table stores information about the predecessor of a vertex as it is marked;

using previous we can construct the shortest paths from the source to all other

destinations.

The method returns dist, a table with vertices as keys and

shortest distances to the source as values, and previous.

Although previous contains all the information needed to construct the short-

est paths, it is desirable to have this information in a more convenient format.

As a matter of fact, it is not immediate which vertex was used as source to derive

previous. Alternatively, it would be nice to have, similarly to dist, a function

that takes a destination and returns the shortest path, as a table, from the

source to the argument. Here is one way to do it: we can define a factory,

function backtracker (previous)

return function (dest)

return function(p, v) return p[v] end, previous, dest

end

end

that creates stateless iterators that backtrack from the destination to the source

along the shortest path, and use one such iterator to implement a path builder:

170

15 · Building Data Structures and Iterators in Lua

function shortestpath (G, v)

-- initialize

local cost = G.eset -- alias

local dist, previous = {}, {}

local H = Heap.new() -- keep unmarked vertices ordered by dist

for u in G:vertices() do dist[u] = math.huge end

dist[v] = 0

for u, d in pairs(dist) do H:insert(d, u) end -- build heap

-- iterate

while not H:isempty() do -- any vertex left?

local du, u = H:retrieve() -- du = min_v dist[v], u marked

-- update distances

for w, e in G:neighborhood(u) do

-- dist[w] = min{dist[w], dist[u] + cost(w, u)}

local dw = dist[w]

local d = du + cost[e]

if dw > d then -- update w?

dist[w] = d

previous[w] = u

H:update(w, d)

end

end

end

return dist, previous

end

Listing 6. Dijkstra’s algorithm: single-source shortest paths in graph G from v.

function buildpath (btrack, dest)

local p = {dest} -- path

for v in btrack(dest) do -- dest -> source

p[#p + 1] = v

end

local c = #p

for i = 1, #p / 2 do -- reverse path

p[i], p[c] = p[c], p[i]

c = c - 1

end

return p

end

Of course, buildpath could have previous as direct argument, but backtracker

gives you the functionality to traverse shortest paths for purposes other than

building paths. Moreover, backtracker abstracts from previous and it is more

meaningful.

171

Conclusions

This gem highlights Lua’s simplicity and expressiveness through the implemen-

tation of data structures and iterators. Data structures are easily realized by

using tables and their __index metamethod for object-like notation, while iter-

ators can be build using Lua’s generic for loop, closures and coroutines. The

resulting code is simple, high-level and abstract, mostly composed of table ac-

cesses and object method calls.

One of the main motivations of this text is to show many possible ways of

achieving a goal in Lua. For instance, there are many ways to construct an

iterator in Lua according to PiL; we cover them all here. Another motivation is

to be true to one of Lua’s maxims in PiL — “Lua gives you the power, you build

the mechanisms” — by providing iterators instead of tables, for example, or ways

of generating a desired table instead of the table itself, as in the backtracker

factory.

16

A Primer of

Scientific Computing in Lua

Luis Carvalho

Lua is a fast, resourceful, easily embeddable and extensible programming lan-

guage; this set of features makes Lua very suitable for scientific computing

applications. This gem implements a simple interface to two cornerstones of

scientific computing: numerical linear algebra and discrete Fourier transforms

(DFTs). More specifically, one module for (two-dimensional) matrices is imple-

mented using some of Lua’s unique resources, namely: weak tables, coroutines,

metamethods, and environments.

The matrix module defines a matrix object and its methods. A matrix object

is a userdatum containing a lua_Number pointer as the data core and other de-

scriptive parameters: number of rows and columns and stride. Data cores are

allocated as new matrices are created and are then stored in weak-keyed table

as values, whereas the corresponding keys are matrices that reference them.

Matrix rows are objects where the data core is a pointer to the “parent” matrix’s

data core, and are lazily interned on the parent’s userdatum environment table

when a call to __index is made. Most routines, like addition and scalar multi-

plication, are thin wrappings around simple (level 1) routines in the ubiquitous

BLAS library. A routine to perform the discrete cosine transform of a vector is

also provided based on a routine from the — equally ubiquitous — FFTW library.

The C side of the matrix module is kept to the minimal necessary extent for

174

16 · A Primer of Scientific Computing in Lua

wrapping and performance sensitive routines, and it is further extended by Lua

code. Other examples with applications of the matrix module are presented.

Introduction

Lua tables are fast enough for many numerical applications. However, for more

specific applications performance is usually critical. One way to achieve a high

performance environment in Lua is to extend the language with suitable objects

and methods from efficient numerical libraries. Of course, these extensions

should profit from Lua’s expressiveness and resourcefulness.

In this gem we explore two common numerical extensions: matrices and dis-

crete Fourier transforms. These together allow us to implement many standard

scientific computing algorithms including numerical linear algebra, interpola-

tions, and quadratures. Moreover, with Lua we are able to devise very efficient

implementations that can beat even well-known scientific computing software!

We assume the reader is fairly familiar with Lua’s C API and feels comfort-

able managing the Lua–C virtual stack. Due to the numerical nature of the text,

a background in engineering or numerical analysis is desirable, but not needed;

the last section, however, deals with more mathematically sophisticated appli-

cations and relies on some knowledge of elementary calculus and linear algebra.

Matrices

In order to keep our interface simple our matrices are real two-dimensional

arrays stored in column-major order. Column-major order is essential because

we will be using Fortran libraries, and that is Fortran’s storage mode for arrays.

The first issue we need to address when representing matrices in Lua is

indexing. For a one-dimensional array (a vector) that is simple enough: just

create a userdatum with fields size and data and return data[i] for index i.

For matrices this is a bit more complicated since we have to go through two

dimensions to access an entry. One solution to accessing index (i, j) is to return

the i-th row as a matrix which in turn behaves like a vector and returns the j-th

entry.

Since matrices are stored in column-major order, rows in a matrix have an

offset from one entry to the next, that is, between consecutive columns. This row

offset is the number of rows from the matrix referenced by the row. We call this

offset stride, and we keep track of it in our data structure.

It is not uncommon to use a matrix without referencing its entries, that

is, by only using matrix operations. That is actually the most efficient way

to use our library: operate in the higher level, say, adding, multiplying or

transforming matrices, and leave the heavy-duty operations to the optimized,

architecture-dependent libraries under the hood; seasoned users of numerical

software usually refer to this strategy as “vectorizing your code”. This practice

suggests that we should adopt a lazy interning strategy and only create the rows

175

of a matrix when needed, that is, when some entry is requested. To intern the

rows of a matrix we can use the environment table of the matrix’s userdatum.

The straightforward association is then to store the i-th row at entry i in the

environment table.

The next issue is memory allocation and deallocation.

Suppose that we

decide to allocate a memory block to store both structure (number of rows,

columns, and stride) and data of a matrix; what would happen if we attribute one

row of this matrix to a variable and then garbage-collect the matrix? The data

would be collected along, and the variable would be left dangling. To avoid this,

we separate the structure and data of a matrix and allocate them individually. A

data memory block can only be collected once all the matrices referencing it have

been collected. To this end, we use a weak-keyed table where the keys are matrix

userdata holding both the structure and a pointer to a data memory block, and

the values are the referenced data memory blocks. To avoid extra table lookups

we do not create a dedicated table to hold the matrix to data associations, but

instead use the matrix userdatum metatable as a holder.

After all these considerations our matrix should have the following structure:

typedef struct {

int rows;

int cols;

int stride;

lua_Number *data;

} lua_Matrix;

To create a new matrix, we then use

static int matrix_new (lua_State *L) {

int r = luaL_checkinteger(L, 1);

int c = luaL_optinteger(L, 2, 1); /* vector as default */

int i, n;

lua_Number *data;

if (r < 1 || c < 1) luaL_error(L, "invalid size to matrix");

lua_settop(L, 2);

n = r * c; /* data size */

data = lua_newuserdata(L, n * sizeof(lua_Number)); /* data block */

for (i = 0; i < n; i++) data[i] = 0; /* initialize to zeros */

pushmatrix(L, r, c, 1, data, 3);

return 1;

}

where pushmatrix handles matrix userdatum allocation and matrix-to-data as-

sociations in Listing 1.

Note that we need to control the size of the stack

(lua_settop) in matrix_new, since the last parameter to pushmatrix is assumed

to be an absolute stack position.

The purpose of pushmatrix will be clearer shortly, but it should be intuitive

that we might want to create references to a data block that is already in the

176

16 · A Primer of Scientific Computing in Lua

static lua_Matrix *pushmatrix (lua_State *L, int rows, int cols,

int stride, lua_Number *data, int dataidx) {

lua_Matrix *m = lua_newuserdata(L, sizeof(lua_Matrix));

lua_pushvalue(L, -1);

lua_pushvalue(L, dataidx); /* data block */

lua_rawset(L, LUA_ENVIRONINDEX); /* env[matrix] = data */

m->data = data;

m->rows = rows;

m->cols = cols;

m->stride = stride;

lua_pushvalue(L, LUA_ENVIRONINDEX); /* metatable */

lua_setmetatable(L, -2);

if (rows > 1 && cols > 1) { /* not a vector? */

lua_newtable(L); /* new matrix environment */

lua_setfenv(L, -2);

}

return m;

}

Listing 1. Auxiliar routine to matrix new.

stack and that needed not to be previously allocated. It is important to observe

that all functions that call pushmatrix should have the matrix’s metatable as

environment. Also note that if a matrix is not a vector it receives a new table as

its environment to store rows.

One advantage of using a userdatum metatable as environment to its meth-

ods is that it is easy and efficient to check a userdatum in the stack: we just

need to verify the existence of a metatable and equality of this metatable to the

calling function’s environment first, as in checkmatrix below:

static lua_Matrix *checkmatrix (lua_State *L, int pos) {

lua_Matrix *m = NULL;

if (lua_isnoneornil(L, pos)

|| !lua_getmetatable(L, pos)) /* no MT? */

return NULL;

if (lua_rawequal(L, -1, LUA_ENVIRONINDEX)) /* MT == env? */

m = (lua_Matrix *) lua_touserdata(L, pos);

lua_pop(L, 1); /* lua_Matrix MT */

return m;

}

and then check for a null pointer:

lua_Matrix *m = checkmatrix(L, pos);

if (m == NULL) luaL_argerror(L, pos, "matrix expected");

This routine should be performed whenever we expect a matrix as argu-

ment. It assumes that the calling function has the appropriate environment,

177

as pushmatrix does. As a first example, matrix_size returns the dimensions of

a matrix userdatum:

static int matrix_size (lua_State *L) {

lua_Matrix *m = checkmatrix(L, 1);

if (m == NULL) luaL_argerror(L, 1, "matrix expected");

lua_pushinteger(L, m->rows);

lua_pushinteger(L, m->cols);

return 2;

}

From now on, to make the code conciser, we implicitly perform all matrix

checks and assume the arguments given to all functions are consistent: for

example, if we are adding two matrices, they should have the same number

of rows and columns1.

Metamethods

To access entries in our matrix we need to implement two metamethods: __index

and __newindex. Let’s start with __index. Since we want to enable colon call

notation for matrix methods — like m:method(...) — our __index should first

check if the key is a number, in which case an entry is requested, or otherwise

delegate to a metatable lookup. Next, if the key is numeric, we should check if

it is valid, that is, if it is positive and less than or equal to the size of the matrix.

Now, according to our previous discussion, we decided for lazily interned rows

when indexing a matrix and for a direct entry if the matrix is a vector; if the row

is interned, we should just get it from the userdatum’s environment, otherwise

intern it before returning. This discussion leads to matrix__index in Listing 2.

The first part of __index, when the key is a valid numeric index, is expected:

it returns an entry if the matrix m is one-dimensional or a row if not, taking care

to store the row if it is not interned yet. The row interning procedure comprises

three steps: getting the data block data associated with m from the function

environment; pushing a new row r such that r->rows = 1, r->cols = m->cols,

r->stride = m->rows, and r->data = data[k - 1] using pushmatrix, since the

r’s data block is only referenced, not allocated; and finally interning r as the k-th

entry in m’s environment.

If the key is not a number we resort to a metatable lookup. The class table

containing all matrix methods is the first upvalue in the __index closure; more

details on how this is set up will appear when we talk about luaopen_lmatrix in

the “Library setup” section.

Matrix entry attribution is accomplished through __newindex only on one-

dimensional matrices. The procedure is very similar, but simpler; it is listed in

Listing 3 for the sake of completeness.

We can also implement a __len metamethod,

1Don’t worry: the complete code in the repository contains all the consistency checks.

178

16 · A Primer of Scientific Computing in Lua

static int matrix__index (lua_State *L) {

lua_Matrix *m = (lua_Matrix *) lua_touserdata(L, 1);

if (lua_isnumber(L, 2)) {

int k = lua_tointeger(L, 2);

if (k < 1 || (m->rows == 1 && k > m->cols)

|| (m->cols == 1 && k > m->rows))

luaL_error(L, "matrix index out of range");

if (m->rows == 1 || m->cols == 1) /* vector? */

lua_pushnumber(L, m->data[(k - 1) * m->stride]);

else {

lua_getfenv(L, 1); /* matrix env */

lua_rawgeti(L, -1, k);

if (lua_isnil(L, -1)) { /* isn’t row k interned? */

lua_Number *data;

lua_pop(L, 1); /* nil */

lua_pushvalue(L, 1); /* matrix */

lua_rawget(L, LUA_ENVIRONINDEX); /* push data */

data = (lua_Number *) lua_touserdata(L, -1);

/* new row: */

pushmatrix(L, 1, m->cols, m->rows, data + k - 1, 4);

lua_pushvalue(L, -1);

lua_rawseti(L, -4, k); /* matrix env[k] = new row */

}

}

}

else { /* meta lookup? */

lua_pushvalue(L, lua_upvalueindex(1)); /* class */

lua_pushvalue(L, 2);

lua_rawget(L, -2);

}

return 1;

}

Listing 2.

index metamethod for matrices.

179

static int matrix__newindex (lua_State *L) {

lua_Matrix *m = (lua_Matrix *) lua_touserdata(L, 1);

int k;

lua_Number v;

if (m->rows > 1 && m->cols > 1)

luaL_error(L, "can’t assign to matrix row");

if (!lua_isnumber(L, 2) || !lua_isnumber(L, 3))

luaL_error(L, "wrong type to matrix assignment");

k = lua_tointeger(L, 2);

v = lua_tonumber(L, 3);

if (k < 1 || (m->rows == 1 && k > m->cols)

|| (m->cols == 1 && k > m->rows))

luaL_error(L, "matrix index out of range");

m->data[(k - 1) * m->stride] = v;

return 0;

}

Listing 3.

newindex metamethod for matrices.

static int matrix__len (lua_State *L) {

lua_Matrix *a = lua_touserdata(L, 1);

lua_pushinteger(L, a->rows);

return 1;

}

which is really useful only for column vectors since it only returns the number

of rows — we have matrix_size to retrieve both dimensions if needed — and a

__tostring metamethod for pretty printing:

static int matrix__tostring (lua_State *L) {

lua_pushfstring(L, "matrix: %p", lua_touserdata(L, 1));

return 1;

}

Since we are dealing with matrices, it is only natural to expect arithmetic

metamethods; these will be addressed in pure Lua in the “Lua side” section,

after we talk about library bindings.

Core methods

Now, true to our intention of pushing the heavy-duty work to the C side of our

library, we need to provide some basic methods that operate over all entries of a

matrix.

Our first routine, matrix_fill, sets all entries of a matrix to a number given

as argument:

180

16 · A Primer of Scientific Computing in Lua

static int matrix_fill (lua_State *L) {

lua_Matrix *m = checkmatrix(L, 1);

lua_Number s = luaL_checknumber(L, 2);

int i, n = m->rows * m->cols * m->stride;

lua_settop(L, 2);

for (i = 0; i < n; i += m->stride) m->data[i] = s;

lua_pop(L, 1); /* number */

return 1; /* matrix */

}

This routine is a good example of two important practices in our matrix methods:

the stride should always be used when traversing a matrix, as in the for loop;

and the input matrix should always be returned for notational convenience2 as

it allows consecutive colon calls in Lua, as in r,c = m:fill(1):size(). Note

that, since we are returning the first argument, we set the stack top and pop the

second argument.

A useful routine is scalar summation, where a matrix m is added to a number

s yielding a matrix m + s. Although this routine produces a new matrix by

definition, we can use an in-place version of it, which we call matrix_shift, by

simply changing the for loop at line 6 in matrix_fill to

for (i = 0; i < n; i += m->stride) m->data[i] += s;

In-place routines, where the entries of a matrix are updated, should always

be preferred. This guideline is justified, at this lower level of our implementa-

tion, as a way to avoid unnecessary memory allocations; if we really need to copy

a matrix, we should explicitly do so. To perform scalar summation on m and s, for

example, we would copy m to another matrix c and then apply c:shift(s). Scalar

summation will be treated shortly, when we discuss the __add metamethod. An

efficient implementation of a matrix copying routine is presented in the “Exter-

nal libraries” section.

We also need some operations, that is, routines that take two matrices with

same number of columns and rows and return another matrix consistent with

the arguments. An important operation is the element-wise multiplication3 of

two matrices:

static int matrix_ewmul (lua_State *L) {

lua_Matrix *a = checkmatrix(L, 1);

lua_Matrix *b = checkmatrix(L, 2);

int i, n;

lua_settop(L, 2);

n = a->rows * a->cols;

for (i = 0; i < n; i++) a->data[i * a->stride] *= b->data[i * b->stride];

lua_pop(L, 1); /* b */

return 1; /* a */

}

2Some might disagree and point out that such feature actually reduces code readability.

3Also known as Hadamard product, but .* should be more familiar.

181

Note that operation is in-place for the first argument, and that the strides of

both arguments are used in the update.

Functional facilities

In the previous section we managed to avoid loops that update entries in a

matrix by providing specialized routines. After all, we do not want to incur

in metatable overheads for calling __index and __newindex on loops like

for i = 1, #v do

v[i] = foo(v[i])

end

if foo is simple enough to be coded as a library core method. However, instead

of providing specific methods for every foo function, we should do so only if foo

is common enough to warrant a method on its own and such that the resulting

method is more efficient than a general method that takes a function like foo as

argument. This was the case for the methods in the previous section; now we

address the general case.

The best way to avoid the loop in the last listing is to provide a higher level

function that takes a matrix m, a function f, and maps each entry e in m to f(e)

if f(e) is a number:

static int matrix_map (lua_State *L) {

lua_Matrix *m = checkmatrix(L, 1);

int n, i;

luaL_checktype(L, 2, LUA_TFUNCTION);

lua_settop(L, 2);

n = m->rows * m->cols * m->stride;

for (i = 0; i < n; i += m->stride) {

lua_pushvalue(L, -1); /* function */

lua_pushnumber(L, m->data[i]); /* entry */

lua_call(L, 1, 1);

if (lua_isnumber(L, -1)) m->data[i] = lua_tonumber(L, -1);

lua_pop(L, 1);

}

lua_pop(L, 1); /* function */

return 1; /* matrix */

}

Our mapping method is a well-known functional facility. Another useful

one is a method that takes the same arguments as matrix_map, an optional

initial value for an accumulator a, traverses m in increasing column-major order

updating a to f(a, e), and returns the final value of a4:

4Exactly, a left fold.

182

16 · A Primer of Scientific Computing in Lua

static int matrix_fold (lua_State *L) {

lua_Matrix *m = checkmatrix(L, 1);

int n, i;

luaL_checktype(L, 2, LUA_TFUNCTION);

lua_settop(L, 3);

n = m->rows * m->cols * m->stride;

for (i = 0; i < n; i += m->stride) {

lua_pushvalue(L, -2); /* function */

lua_insert(L, -2); /* accumulator */

lua_pushnumber(L, m->data[i]); /* entry */

lua_call(L, 2, 1);

}

return 1;

}

Besides the obvious difference of returning an accumulation instead of a

mapped matrix, note that matrix_fold’s function f accepts two arguments in-

stead of one in matrix_map’s f. Also note the order in which the arguments to f

are pushed in both methods, and how we need to pop the result from lua_call

in matrix_map but not in matrix_fold.

A practical application of fold is to compute the sum of the entries of a matrix

function sum (m)

return matrix.fold(m, function(x, y) return x + y end, 0)

end

where zero should be given as initial value5. Functions of the form

function(x, y) return alpha * x + y end

are actually very common, and we refer to folds arising from their application

as linear folds. Since linear folds can be parameterized by alpha, we can define

a simpler version of matrix_fold:

static int matrix_linfold (lua_State *L) {

lua_Matrix *m = checkmatrix(L, 1);

lua_Number alpha = luaL_optnumber(L, 2, 1);

lua_Number x = luaL_optnumber(L, 3, 0);

int n, i;

lua_settop(L, 2);

n = m->rows * m->cols * m->stride;

for (i = 0; i < n; i += m->stride) x = x * alpha + m->data[i];

lua_pushnumber(L, x);

return 1;

}

5Not necessarily true: we can always test the first argument of the function against nil to provide a suitable initial value, but this is more efficient (not to mention traditional).

183

Now we can simply define sum = matrix.linfold. Another neat application

of linfold is to implement polynomial evaluation using Horner’s scheme: if c

is a vector where c[i] is the coefficient of xn+1−i in a polynomial P of degree n

(== #c - 1) on x, then

function poly (c)

return function(x) return c:linfold(x) end

end

returns an evaluator for P .6

External libraries

So far we have been able to provide efficient routines for simple methods in our

matrix library. For many other specialized and more complex matrix routines —

like computing the norm of a matrix, or solving a linear system, or inverting a

matrix — we can resort to optimized code from external libraries.

For our basic needs here we are going to use the high-quality ubiquitous

BLAS (Basic Linear Algebra Subprograms) library, or better, an optimized ver-

sion of it7.

Our first routine using BLAS is a method to copy matrices:

static int matrix_copy (lua_State *L) {

lua_Matrix *m = checkmatrix(L, 1); /* source */

lua_Matrix *d = checkmatrix(L, 2); /* dest [optional] */

int n;

lua_settop(L, 2);

n = m->rows * m->cols;

if (d == NULL) { /* no destination? create new matrix */

lua_Number *data = lua_newuserdata(L, n * sizeof(lua_Number));

int inc = 1;

dcopy_(&n, m->data, &m->stride, data, &inc);

pushmatrix(L, m->rows, m->cols, 1, data, 3);

}

else dcopy_(&n, m->data, &m->stride, d->data, &d->stride);

return 1;

}

The actual copying is done by the BLAS routine at lines 10 and 13, which

have signature

dcopy_(int *n, double *x, int *incx, double *y, int *incy);

6An easy polynomial object from c:

getmetatable(c). call = function(v, x) return v:linfold(x) end

7BLAS is in public domain and can be found at http://www.netlib.org/blas. There are many

optimized versions of BLAS, depending on the platform, but the most common open source version is ATLAS: http://math-atlas.sourceforge.net.

184

16 · A Primer of Scientific Computing in Lua

where x is to be copied to y, n is the size of x and y and incx and incy are

the strides of x and y respectively. Note that all arguments are pointers; since

BLAS’s natural implementation is in Fortran8 and Fortran passes arguments

by reference, we should provide variables by their memory addresses. Of course,

we are assuming that lua_Number is defined as double, as in vanilla Lua. We

should also always provide strides for our matrix arguments, similar to what we

did in the previous sections. As a matter of fact, all BLAS routines that we use

here have a common signature pattern: the size of the argument(s) comes first,

followed in some routines by a number meant for scalar multiplication, and then

the matrix argument(s) as a data block address and a stride.

We can provide an optional destination to matrix_copy as a second argument.

If no destination is specified, a data block is allocated and a fresh matrix is

pushed; otherwise, we just copy to the provided destination matrix — of course,

as in our previous routines, we are assuming the matrices are consistent and do

not set any checks in our prototype. The main reason for using a copy destination

is when a procedure performs a copy operation often and we can then use a

buffer to avoid new matrices being created at each operation. Also, note that we

return the copy destination matrix, as expected.

To scale a matrix m by a number s, that is, to multiply each element in m by a

scalar s, we use

static int matrix_scale (lua_State *L) {

lua_Matrix *m = checkmatrix(L, 1);

lua_Number s = luaL_checknumber(L, 2);

int n = m->rows * m->cols;

lua_settop(L, 2);

dscal_(&n, &s, m->data, &m->stride);

lua_pop(L, 1); /* scale */

return 1; /* matrix */

}

where dscal performs the hard work. The signature pattern should be already

familiar. As usual, we provide references as arguments to dscal and return the

scaled matrix.

Continuing with in-place linear operations, we have a routine that incre-

ments a matrix y by a * x, where a is a (not necessarily positive) optional num-

ber and x is consistent with y:

static int matrix_add (lua_State *L) {

lua_Matrix *y = checkmatrix(L, 1);

lua_Matrix *x = checkmatrix(L, 2);

lua_Number a = luaL_optnumber(L, 3, 1.0); /* defaults to 1.0 */

int n;

lua_settop(L, 3);

n = y->rows * y->cols;

8This also explains the ugly underscore after a Fortran routine’s name when calling from C.

185

daxpy_(&n, &a, x->data, &x->stride, y->data, &y->stride);

lua_pop(L, 2); /* x, a */

return 1; /* y */

}

Finally we can compute the dot product of two consistent column vectors with

static int matrix_dot (lua_State *L) {

lua_Matrix *x = checkmatrix(L, 1);

lua_Matrix *y = checkmatrix(L, 2);

int n = x->rows * y->rows;

lua_pushnumber(L,

ddot_(&n, x->data, &x->stride, y->data, &y->stride));

return 1;

}

where ddot_ returns a double.

Library setup

Now that we have all methods, we can register them in our library. For this pur-

pose we create two luaL_regs, one for class methods and other for metamethods,

static const luaL_reg lmatrix_func[] = {

{"new", matrix_new},

/* ... list other methods here ... */

{"dot", matrix_dot},

{NULL, NULL}

};

static const luaL_reg lmatrix_mt[] = {

{"__newindex", matrix__newindex},

{"__len", matrix__len},

{"__tostring", matrix__tostring},

{NULL, NULL}

};

where __index is not included in lmatrix_mt because it requires the class table

as an upvalue.

Our library entry point is in Listing 4. Recall that we need to complete three

tasks: set the environment for all our methods as the matrix class metatable

for pushmatrix and checkmatrix to run correctly; define the environment, which

holds matrix to data block associations, as a weak-keyed table; and set the class

table as an upvalue to matrix__index. We also need to fill the class table and

metatable, as it is usually done in entry point routines.

The comments in luaopen_lmatrix should guide the reader through all these

steps, and it is a good exercise to keep track of the stack as the code is executed.

Note that the resulting C library should be called lmatrix and linked against

-lblas; adjust your files and compiling targets accordingly.

186

16 · A Primer of Scientific Computing in Lua

int luaopen_lmatrix (lua_State *L) {

lua_newtable(L); /* class */

lua_newtable(L); /* new environment */

lua_pushvalue(L, -1);

lua_replace(L, LUA_ENVIRONINDEX); /* set as default environment */

/* fill class */

lua_pushvalue(L, -2); /* class */

luaL_register(L, NULL, lmatrix_func);

/* class as upvalue for __index: */

lua_pushcclosure(L, matrix__index, 1);

lua_setfield(L, -2, "__index");

luaL_register(L, NULL, lmatrix_mt); /* fill env/metatable */

/* set environment as weak-keyed table: */

lua_newtable(L); /* env metatable */

lua_pushstring(L, "k");

lua_setfield(L, -2, "__mode");

lua_setmetatable(L, -2);

/* return class */

lua_pop(L, 1); /* env/metatable */

return 1;

}

Listing 4. lmatrix entry point.

Lua side

Now that everything is set up on the C side, we can turn to Lua to enhance our

library, matrix.lua. First of all, we need to load all methods from lmatrix,

matrix = require "lmatrix" -- create global

module(...)

Since matrix is global and also the name of our library, the class table

returned by require"lmatrix" becomes the environment after the call to module.

As promised before, we now provide arithmetic metamethods for our matrix

userdata based on add, shift, and scale in Listing 5. All these metamethods

should return a new table, and so we use copy to explicitly copy the values and

apply any needed transformation in-place. Depending on the second argument

to __add and __sub, we either shift or add the copy s of the original matrix a.

Note that __unm uses consecutive calls in colon notation, which could also be

used for __add, for example,

mt.__add = function(a, b)

return type(b) == "number" and a:copy():shift(b) or a:copy():add(b)

end

but at the cost of being less clear.

187

local mt = getmetatable(matrix.new(1))

mt.__unm = function(m)

return m:copy():scale(-1)

end

mt.__add = function(a, b)

local s = a:copy()

if type(b) == "number" then s:shift(b) else s:add(b) end

return s

end

mt.__sub = function(a, b)

local s = a:copy()

if type(b) == "number" then s:shift(-b) else s:add(b, -1) end

return s

end

Listing 5. Remaining arithmetic metamethods for matrices.

Sometimes we need to fill a vector v of size n linearly starting from a number

a, v[1] = a, and finishing at a number b, v[n] = b. This could be done in-place,

but the need for a new vector is far more common, leading us to linspace

function linspace (a, b, n)

assert(type(a) == "number" and type(b) == "number",

"number expected")

local n = n or math_abs(b - a) + 1

assert(type(n) == "number" and n > 0, "unexpected number of steps") local s = (b - a) / (n - 1)

local l = a - s

return matrix.new(n):map(function() l = l + s; return l end)

end

where math_abs is a local for math.abs defined before the module call.

It is also handy — but not necessarily efficient — to have a method similar

to pairs for matrix traversal. The idea is simple: we just need to keep two

control variables as row and column indexes and update them as we traverse

the matrix. Our first try, in Listing 6, uses coroutines; local variables i and j

are controls for row and column indexes respectively, while v is used to cache

the i-th row and avoid metatable lookup overhead.

Vectors are treated differently because there is actually only one dimension

to traverse. We chose to assert this case here to save space, but it should

be simple to implement it by adapting from the matrix case9. As expected,

coroutine_wrap and coroutine_yield are locals to their respective methods,

coroutine.wrap and coroutine.yield.

9Or by drawing inspiration from ipairs. The complete version of entries can be found in the repository.

188

16 · A Primer of Scientific Computing in Lua

function entries (m)

local r, c = m:size()

assert(r > 1 and c > 1, "vectors not allowed")

local i, j, v = 0, c

return coroutine_wrap(function()

repeat

if j == c then -- next row

i, j = i + 1, 1

v = m[i]

else -- next column

j = j + 1

end

coroutine_yield(i, j, v[j])

until i == r and j == c

end)

end

Listing 6. Matrix traversal routine.

Although coroutines provide a flexible and easy way of implementing matrix

iterators, we can do better by simply providing a suitable closure on the control

variables that returns instead of yielding10:

function entries (m)

local r, c = m:size()

assert(r > 1 and c > 1, "vectors not allowed")

local i, j, v = 0, c

return function() -- closure on i and j

if j == c then -- next row

i, j = i + 1, 1

v = m[i]

else -- next column

j = j + 1

end

if i <= r then return i, j, v[j] end

end

end

Applications

Armed with our matrix module, we can now work on some applications. Since

the methods we implemented are basic, we need to illustrate their power with

some simple — but not elementary! — applications. In the next sections we

will first exercise our module with some straightforward tasks to gain more

10A good exercise is to provide a stateless iterator for vectors; check the repository for a solution.

189

familiarity, and then deal with variations on a theme: interpolation. We will talk

about the most common interpolation, Lagrangian interpolation, then discuss

discrete Fourier transforms, and finally apply another type of interpolation to

compute the quadrature of an arbitrary function.

This section, and in particular the latter part, is more mathematically in-

volved, but we try to keep the text as self-contained as possible without going

too much into details. The applications are standard in numerical computing,

and the interested reader can refer to a number of good books in the subject

to quench the curiosity for more details and the desire for more rigor. The last

application demands some degree of familiarity with more complex math, but

serves well to illustrate how versatile and powerful Lua is; it was largely in-

spired by some of Prof. Lloyd Trefethen’s works11.

Basic operations

Let’s first take our matrix module for a test drive in an interpreter session. We

start off by checking some basic operations:

$ lua

Lua 5.1.2

Copyright (C) 1994-2007 Lua.org, PUC-Rio

> require "matrix"

> n = 4

> a = matrix.linspace(1, n) -- [1, 2, ..., n]’

> b = matrix.new(n) -- [0, 0, ..., 0]’

> x = matrix.linspace(0, math.pi, n) -- [0, pi/(n-1), ..., pi]’

> x:copy(b):map(math.cos) -- b[i] = cos(x[i])

> s = -(a + 1 + b - 1 - b) -- __add, __sub, __unm

> for i = 1, n do print(a[i], x[i], b[i], s[i]) end

1

0

1

-1

2

1.0471975511966 0.5

-2

3

2.0943951023932 -0.5

-3

4

3.1415926535898 -1

-4

Everything works as expected. Now, as a warm up, let’s define a function

that returns the dot product of two vectors, that is, the sum of their entry-wise

product:

function dot(a, b)

local t = a:copy() -- t[i] = a[i]

t:ewmul(b) -- t[i] = a[i] * b[i]

return t:linfold() -- sum_i a[i] * b[i]

end

Our dot is clearly less efficient than matrix.dot since it needs a copy of the

first argument, but it provides a good exercise nonetheless. For instance, once

11In particular, “Is Gauss quadrature better than Clenshaw–Curtis?” and “An extension of Matlab to continuous functions and operators”.

190

16 · A Primer of Scientific Computing in Lua

we are more comfortable with the colon notation, we can drop temporary local

variables like t above and just concatenate operations — being careful not to

abuse notation at the cost of readability! — as this quick test shows:

> print(a:dot(b), dot(a, b), a:copy():ewmul(b):linfold())

-3.5

-3.5

-3.5

Next, we can compute the p-norm of a vector,

n

!1/p

X

kvkp =

|vi|p

i=1

by implementing norm directly from the definition:

function norm (v, p)

local s = 0

for i = 1, #v do s = s + math.abs(v[i]) ^ p end

return s ^ (1 / p)

end

A more generic version of norm could be devised using functional facilities

if we observe that we actually do not need to index the entries, but only a fold

(summation) over a map on the entries (p-th power of the absolute value):

function norm (v, p)

local f = function(a, e) return a + math.abs(e) ^ p end

return v:fold(f, 0) ^ (1 / p)

end

This last version is directly applicable to matrices as well. As a matter of

fact, we can define the Frobenius norm of a matrix, that is, the square root of

the sum of its squared entries, as

function frobenius (m) return norm(m, 2) end

As p grows, the norm converges to the inf-norm

kvk∞ = max |vi|

i=1,...,n

which we can compute similarly by

function infnorm (v)

local f = function(a, e)

local t = math.abs(e)

return a > t and a or t -- a = max(a, |e|)

end

return v:fold(f, 0)

end

191

Let’s make a quick check on norm and infnorm:

> for p=1,n do print(p, norm(a,p)) end; print("inf", infnorm(a))

1

10

2

5.4772255750517

3

4.6415888336128

4

4.3376131365334

inf

4

As another interesting simple application, we can write a pretty-printer for

matrices that returns a row per line and tab-separated column entries. To

simplify the implementation, we print a vector as a row even if it is a column

vector, and so to print a matrix we just need to iterate over its rows:

function pretty (m)

local r, c = m:size()

if r == 1 or c == 1 then -- vector?

local t, l = {}, r > c and r or c -- l = max(r, c)

for i = 1, l do

t[i] = string.format("%g", m[i])

end

print(table.concat(t, "\t"))

else

for i = 1, r do pretty(m[i]) end

end

end

Note the common Lua practice of storing strings to be concatenated in a table

and then using table.concat instead of using the concatenation operator ..

directly.

Finally, to test our pretty printer, let’s create a special type of matrix. A

Pascal matrix Pn is a symmetric matrix of order n where the elements pi,j in the

anti-diagonal where i + j = k + 2 are the binomial coefficients in the expansion

of (x + y)k, namely

i + j − 2

1,

i = 1 or j = 1

pi,j =

=

i − 1

pi,j−1 + pi−1,j, otherwise

that is, if we look at the anti-diagonals starting at the top-left corner of the

matrix we see the rows of Pascal’s triangle. The recursive definition above stems

from a well-known binomial coefficient identity12 and it is particularly useful for

our implementation:

12

k + 1

k

k

=

+

, where k = i + j − 3 and l = i − 1.

l

l

l − 1

192

16 · A Primer of Scientific Computing in Lua

function pascal (n)

local p = matrix.new(n, n)

local c, r = p[1] -- current and previous rows

c:fill(1) -- first row, i = 1

for i = 2, n do -- remaining rows

r, c = c, p[i]

c[1] = 1 -- first column, j = 1

for j = 2, n do -- i ~= 1 and j ~= 1: p[i][j] =

c[j] = c[j - 1] + r[j] --

= p[i][j-1] + p[i-1][j]

end

end

return p

end

An important point to observe in pascal is how we have used references r and c

to the previous, p[i-1], and current, p[i], rows respectively to avoid __index

lookup overheads. Before moving on to more elaborated applications, let’s check

pascal (and pretty):

> pretty(pascal(5))

1

1

1

1

1

1

2

3

4

5

1

3

6

10

15

1

4

10

20

35

1

5

15

35

70

Lagrangian interpolation

Given n points in the plane, (xi, yi), i = 1, . . . , n, with distinct xis, the interpola-

tion problem requires us to find an interpolating function f — the interpolant —

such that f (xi) = yi. Interpolants are usually expressed as a linear combination

of a set of basis functions b1, . . . , bn: f (x) = Pn

c

k=1 kbk(x), where the coefficients

ck are to be determined in order to satisfy the interpolation criterion

n

X

f (xi) =

ckbk(xi) = yi,

i = 1, . . . , n.

(16.1)

k=1

Since we have n points, it is reasonable to define f as a suitable polynomial

of degree not greater than n − 1. The first natural choice is to pick the mono-

mial basis, bk(x) = xk−1, and find ck by solving the linear system defined by

equation (16.1):



1

x

 







1

· · · xn−1

1

c1

y1



1

x2

· · · xn−1

c

y

2

 

2 



2 

BT c =  .

.

.

.

 

.  =  .  = y



.

.

. .

.

 

. 



. 



.

.

.

 

. 



. 

1 xn · · · xn−1

c

y

n

n

n

193

The matrix B above has a particular structure where the i-th row can be

obtained from the i − 1-th row by element-wise product with x; B is then said to

be a Vandermonde matrix with basis x and order n13. We can easily generate a

Vandermonde matrix with

function vandermonde (b, n) -- basis b, order n

local _, m = b:size() -- row vector

local v = matrix.new(n, m)

local r = (n > 1 and v[1] or v):fill(1)

for i = 2, n do

r = r:copy(v[i]):ewmul(b) -- v[i] = v[i-1] .* b

end

return v

end

Observe how we specify the copy destination in the for loop and use the colon

notation: r is v[i-1], and what gets multiplied by b is v[i], the copy destination.

For small n this method works fine, but even though B is not singular it

might get ill-conditioned as n grows — even singular to machine precision —

resulting in very sensitive coefficients. Moreover, we still need to solve the

system in equation (16.1) by providing more bindings to our matrix library from

other external libraries14.

Another option is to specify a different, more numerically stable, basis. For

instance, if we choose the Lagrangian basis

Y

x − x

b

j

k(x) =

,

k = 1, . . . , n,

xk − xj

j6=k

then ck = yk (B is the identity matrix), and we do not need to solve a linear

system anymore. Since f has a closed-form expression now,





n

X

Y

x − x

f (x) =

j



x

 yk

k − xj

k=1

j6=k

we can even implement it efficiently using tables. On the other hand, a version

using matrices should avoid explicit loops to be more efficient. Listing 7 presents

both versions in interp1t and interp1 respectively.

In interp1, b computes bk(z) by folding a closure on xk that accumulates the

product over x. We avoid the singularity when j == k by comparing xk directly

to xj and doing nothing, that is, passing t untouched, if they are equal15. After

evaluating bk(z) by mapping b to x, we apply a dot product to y to obtain the

13Some authors actually define the transpose of a Vandermonde matrix as Vandermonde.

14BLAS provides methods for solving linear systems only when B is triangular, and so we need to resort to BLAS’s big brother, LAPACK (http://www.netlib.org/lapack), to solve general linear systems. Unfortunately, this is out of our scope.

15Assure yourself that it is ok here to compare floating point numbers directly.

194

16 · A Primer of Scientific Computing in Lua

function interp1t(x, y)

local n = #x

assert(n == #y, "sizes differ")

return function(z) -- f

local p = 0 -- f(z)

for k = 1, n do

local xk = x[k]

local t = 1 -- b_k(z)

for j = 1, n do

if j ~= k then

local xj = x[j]

t = t * (z - xj) / (xk - xj)

end

end

p = p + t * y[k]

end

return p

end

end

function interp1(x, y) -- x, y: column vectors

local n = #x

assert(n == #y, "sizes differ")

local p = matrix.new(n) -- proxy

return function(z)

local b = function(xk) -- b_k(z)

return x:fold(function(t, xj)

return xk == xj and t or t * (z - xj) / (xk - xj)

end, 1)

end

return matrix.dot(x:copy(p):map(b), y)

end

end

Listing 7. Lagrangian interpolation using tables, interp1t, and matrices, interp1.

195

final linear combination. Since we would alter x by mapping b, we need to use a

proxy by copying x to p.

Despite using functional facilities, interp1 is still slower than interp1t,

approximately two times slower on my machine. However, since we often want

to interpolate many numbers at a time, we could provide a version of interp

that takes a vector z of abscissae, instead of a single one, to be interpolated and

returns a vector containing f (z). This version is illustrated in Listing 8.

function interp(x, y) -- x, y: column vectors

local n = #x

assert(n == #y, "sizes differ")

return function(z)

local m = #z -- column vector

local v, t, p = matrix.new(m), matrix.new(m), matrix.new(m)

for k = 1, n do

local xk = x[k]

t:fill(1) -- b_k(z), a vector

for j = 1, n do

if j ~= k then

local xj = x[j]

t:ewmul(z:copy(p):shift(-xj)) -- t = t .* (z - x[j])

t:scale(1 / (xk - xj)) -- t = t / (x[k] - x[j])

end

end

v:add(t, y[k])

end

return v

end

end

Listing 8. Vectorized Lagrangian interpolation.

We can think of interp as a vectorized version of interp1t. However, if we

add another loop in interp1t to compute interpolations for each entry in z, and

hence offer a facility similar to interp, we now observe that interp is faster

than its table version. As a matter of fact, it is close to five times faster on my

machine.

Fast Fourier transforms

While modeling cyclic data, it is often more appropriate to use trigonometric

functions — sines and cosines — as basis functions instead of polynomials, as we

did in the previous section. Also in contrast to common polynomial interpola-

tion, trigonometric interpolation presents a recursive structure and can be done

efficiently by an algorithm called the fast Fourier transform (FFT).

FFTs compute discrete Fourier transforms (DFTs): given a complex data

vector x = [x0, . . . , xn−1]T , its DFT y = [y0, . . . , yn−1]T , a consistent complex

196

16 · A Primer of Scientific Computing in Lua

vector, is such that

n−1

X

ym =

xkωmk

n

,

m = 0, . . . , n − 1,

k=0

where ωn = exp(−2πi/n). This is also known as the forward, or direct, trans-

form, and a similar expression exists for the backward, or inverse, transform

that maps y back to x.

Another important transform arises when x is real and even around x0 and

xn−1, that is,

x = [x0, x1, . . . , xn−2, xn−1, xn−2, . . . , x1],

where the actual size of x is 2(n − 1). In this case, we can save space by only

declaring the first n entries of x. Moreover, the DFT of x is also a real and

even vector, and so we can also save space by returning only the first n entries.

By considering only the non-redundant first half of x and computing its DFT

assuming evenness, we have a discrete cosine transform (DCT). DCTs can be

computed using an adapted version of FFT that deals with storage for reals only

and evenness: a fast cosine transform (FCT).

The most well-known library for FFT computing is FFTW (Fastest Fourier

Transform in the West)16, and that is the library we are going to use to imple-

ment a FCT for our matrix library. FFTW works through the concept of plan-

ners: it does not use a fixed, one-size-fits-all algorithm, but it rather learns the fastest way to compute the transform on the underlying hardware and then executes the learned plan to yield the desired transform.

Our new method, matrix_fct, is presented in Listing 9. Since the actual

size of x is 2(n − 1), we need to allocate it physically to perform the transform.

We use buf for this intent. Besides the (now hopefully familiar) routine of

allocating a matrix’s data block (data) and then pushing its structure with

pushmatrix, the DCT is computed by: copying the first half of x to buf using

dcopy (no need to initialize buf as it is updated in-place); creating a plan for a

DCT, which is specified as real-to-real one-dimensional (fftw_plan_r2r_1d) real-

even (FFTW_REDFT00) forward transform; executing and destroying the plan; and

finally copying back the first half of buf to data. Since the transforms computed

by FFTW are unnormalized, we fix this by scaling data by the inverse of x’s

actual size.

A nice optimization to this method would be the implementation of a buffer-

ing system with lua_Number units to avoid the creation and collection of buf at

every run of matrix_fct17. FCTs are the core of our next, and final, application.

Clenshaw–Curtis quadrature

Suppose now that we want to compute the integral of an arbitrary function f .

An approximative approach is to sample f at a number of distinct abscissae xk,

16FFTW is free software and can be found at http://www.fftw.org.

17In case you are wondering: yes, there is one such implementation in the repository, and it is very educative on its own.

197

static int matrix_fct (lua_State *L) {

lua_Matrix *x = checkmatrix(L, 1);

lua_Number s, *data, *buf;

fftw_plan plan;

int n, inc = 1;

lua_settop(L, 1); /* needed for pushmatrix */

n = (x->rows == 1) ? x->cols : x->rows; /* x is a vector */

/* fct(x)’s data block: */

data = lua_newuserdata(L, n * sizeof(lua_Number));

/* DCT */

buf = lua_newuserdata(L, 2 * (n - 1) * sizeof(lua_Number));

dcopy_(&n, x->data, &x->stride, buf, &inc);

plan = fftw_plan_r2r_1d(n, buf, buf, FFTW_REDFT00, FFTW_FORWARD);

fftw_execute(plan);

fftw_destroy_plan(plan);

dcopy_(&n, buf, &inc, data, &inc);

lua_pop(L, 1); /* buf */

/* renormalization */

s = 1.0 / (2 * (n - 1));

dscal_(&n, &s, data, &inc);

pushmatrix(L, n, 1, 1, data, 2);

return 1;

}

Listing 9. A fast cosine transform method for vectors.

obtain an interpolant ˆ

f as an approximation to f , and compute ˆ

f ’s integral,

which should be simpler. The whole process can be summarized by

n

n

!

n

Z b

Z b

Z b

Z b

X

X

X

f (x)dx ≈

ˆ

f (x) dx =

ckbk(x) dx =

ck

bk(x) dx =

ckwk

a

a

a k=1

k=1

a

k=1

which amounts to computing the cks and wks. This procedure is known as

interpolatory quadrature.

It turns out that if we choose the basis functions to be Chebyshev polynomi-

als and judiciously choose the sampling abscissae xk, than ck and wk are easy

to evaluate. The n-th Chebyshev polynomial is defined on the interval [−1, 1] by

T

18

n(x) = cos(n cos−1(x)), which hints at the oscillatory nature of Tn

. In fact,

Chebyshev polynomials have an important property known as equi-oscillation:

successive extrema of Tn are equal in magnitude and alternate in sign. Remark-

ably, if we choose the sampling points x = {xk} to be these extrema, known as

Chebyshev points,

kπ

xk = cos

,

k = 0, 1, . . . , n,

n

18cos−1(x) is arccos(x), not 1/ cos(x), and T is really a polynomial.

n

198

16 · A Primer of Scientific Computing in Lua

then ck can be obtained by a discrete cosine transform on f (x) and wk are simply:

Z 1

0,

k odd

wk =

Tk(x) dx =

2(1 − k2)−1,

k even

−1

The quadrature we obtain by using Chebyshev polynomials is known as

Clenshaw–Curtis quadrature. It has many attractive properties besides sim-

plicity: it is also stable, accurate, and progressive. By progressiveness we mean

that we do not need to re-evaluate f at every sample point in case we want to

enhance the precision by adding more sample points. For our particular choice

of xk, if we have n sample points and want 2n − 1, then only n − 1 new points

and function evaluations are needed.

Back to Lua, we can implement this discussion with Listing 10. cheb has

only one method, new, that returns an object to compute an approximate definite

integral using Clenshaw–Curtis quadrature. Method new takes a function f and

a number of points n as arguments, computes x, the Chebyshev points, and w,

the integral weights, using linspace, and returns a table storing f and n with a

__call metamethod to compute the quadrature.

To allow general, but finite, integration limits — since Chebyshev polynomi-

als are defined only on [−1, 1] — we apply a linear transformation:

Z b

b − a Z 1

b − a

b + a

f (t) dt =

f

x +

dx.

a

2

−1

2

2

There is still room for improvement in cheb. For one, we could provide a pre-

cision as argument instead of the number of sample points, and control the pre-

cision by selecting the appropriate number of points. Another improvement, or

better, extension, would be to implement an interpolation routine that computes

an approximation to f (x) at arbitrary x based on our Chebyshev interpolant.

By choosing a suitable transformation, we can even approximate improper

integrals. Consider the tangent transform,

Z b

π Z 2 tan−1(b)/π

πx

f (t) dt =

f tan

sec2 πx dx,

a

2 2 tan

2

2

−1 (a)/π

and its implementation based on cheb in Listing 11.

With ttintegral we can now implement, for example, pnorm, the cumulative

function for the standard normal distribution:

do

local f = function(x)

return 1 / math.sqrt(2 * math.pi) * math.exp(-x * x / 2)

end

local g = ttintegral(f, 1000)

pnorm = function(x) return g(-math.huge, x) end

end

199

require "matrix"

local linspace, dot = matrix.linspace, matrix.dot

local pi, cos, setmetatable = math.pi, math.cos, setmetatable

module(...)

function new (f, n)

local x = linspace(0, pi, n + 1):map(cos) -- Chebyshev points

local w = linspace(0, n, n + 1):map(function(i) -- weights

return (i % 2 == 0) and 2 / (1 - i * i) or 0

end)

return setmetatable({f=f, n=n}, {

__call = function(c, a, b) -- integral by linear transform

local s = (b - a) / 2

-- compute f(s * x + (b + a) / 2)

local p = x:copy()

p:scale(s):shift((b + a) / 2) -- p = s * p + (b + a) / 2

p:map(f) -- p = f(p)

-- compute interpolating coeffs

local c = p:fct():scale(2) -- c = 2 * dct(p)

c[1], c[n] = c[1] / 2, c[n] / 2

-- compute integral

return s * dot(w, c)

end

})

end

Listing 10. The cheb module.

require "cheb"

function ttintegral (func, n) -- tangent transform

local p2 = math.pi / 2

local f = function(x)

local v = x * p2

local c = 1 / math.cos(v) -- sec(pi * x / 2)

return func(math.tan(v)) * c * c

end

local c = cheb.new(f, n)

return function(a, b)

return p2 * c(math.atan(a) / p2, math.atan(b) / p2)

end

end

Listing 11. Improper integral through tangent transform.

200

16 · A Primer of Scientific Computing in Lua

We can also implement the gamma function:

gamma = function(t, n)

assert(t >= 1, "argument is lesser than 1: " .. t)

local n = n or 1000

local f = function(x) return x ^ (t - 1) * math.exp(-x) end

return ttintegral(f, n)(0, math.huge)

end

Conclusions

In this gem, we have implemented a matrix library and a few applications. Lua’s

resources, including userdatum environments, metamethods, function closures,

first class functions, and proper lexical scoping were invaluable for a simple

yet powerful implementation: our matrices have lazily interned rows, efficient

functional facilities, and proper arithmetic operators, while our Chebyshev in-

terpolants can compute the integral of any function with arbitrary precision and

limits of integration.

Thanks to the C API, it is almost straightforward to extend Lua by either

wrapping routines from high performance, specialized, external libraries or

providing your own routines. We hope this gem inspires the reader to create

new libraries — especially numerical ones! — by following the same approach.

One could also extend the current implementation as suggested in the footnotes,

or even by enhancing the matrix object to account for multidimensionality and

a typing system that would elect the best routine for a specific task if the

matrix were, say, triangular or symmetric. For an implementation of these

latter improvements and other scientific computing facilities, including random

deviates for a number of probability distributions, complex number support,

and specialized functions, the reader can refer to the Numeric Lua project at

http://numlua.luaforge.net.

17

Complex Structured Data Input

Julio M. Fern ández-D´ıaz

Lua is very good at describing (often complex) data structures through tables.

However, apart from spotting syntax errors, Lua cannot deal directly with the

logical structure of read tables.

This gem explains a small library which, in combination with data templates,

enables the user to: introduce complex, controlled structures to any depth; in-

clude test functions to check the validity of the input; declare optional values at

any level for missing fields, if desired; etc. An example, including the appropri-

ate driver, which runs in a convenient protected mode, is also shown.

The problem

As a developer, you are preparing a program (to be used by other people) which

requires (relatively general) data to be processed: control parameters in a chem-

ical plant, objects characteristics in an arcade game, etc.

You may also wish to develop programs that allow some external configura-

tion: sometimes for self use; at other times, the end users wish to adapt the

program to their particular needs (which always adds value to the product).

On other occasions, you are preparing a complex program with a graphical

user interface: windows, menus, buttons, radio-buttons, etc. A tool to facili-

tate the implementation would be welcome: a system that describes the menu

structure both well and clearly (with the corresponding actions) would be very

useful. Then, by adding only a general function for managing the structure, we

have nearly all the necessary parts for an operative program.

202

17 · Complex Structured Data Input

Let us show an example (for the sake of brevity, this is somewhat simplified

compared with a more realistic case). In a program to make book covers, a box

is described in the form:

box "box 1" "front" "c" 90 10 40 10 20 0.5 0.3 0

Our program manages it well, but the user will probably have to go to the

‘manual’ to understand the meaning of each item. It is evident that a descriptive

format (even with comments) is much better:

box {

-- this is a comment

id = "box 1",

place = "front",

adjust = "c",

angle = 90,

-- another comment

position = {x = 10, y = 40},

width = 10,

height = 20,

fill = {color = {r = 0.5, g = 0.3, b = 0}}

}

This second fragment, apart from data, is a chunk of Lua code, which has

to be processed using a function named box (defined in another place inside the

program). Some of the characteristics of the box may even be optional and have

default values. These improvements are easy to implement in the latter case,

but not in the former.

Lua is very good at describing data with a complex structure through tables

(see Section 10.1 in PIL2 for an example). However, the Lua interpreter is

not able to deal directly with the logical structure of those tables, to determine

whether a field must exist or not, to distinguish correct fields from wrong ones,

etc. For example, if the user types plaxe instead of place, the program should

throw an error. If the user inputs r = 2 (forbidden because the amount of

red must be between 0 and 1), the program must signal it. Therefore, a data

description ‘shell’ should be included.

Besides, the possibility of optional fields (with default values or not) should

also exist. Sometimes the program should hinder the use of fields different from

a given set, but at other times this restriction is inadmissible (because the data

file is also used by other programs with diverse necessities).

Thus, we have a problem which may be stated as follows:

To develop a Lua library that allows the input of complex structured

data, with complete control and validation of the contents.

The solution

The initial idea was to include in the program a table with the structure of the

desired input data and to compare the value type field by field. However, this

203

require "datatest"

local positive

= datatest.numrange(0, math.huge)

local purecolor = {VALUE = 1, TEST = datatest.numrange(0, 1)}

local rgbcolor = {r = purecolor, g = purecolor, b = purecolor}

local places = datatest.inset({"front", "back", "spine"},string.lower) template = {} -- to store data templates

template.box = {ALLSTRICT = true,

CONTAINS = {

id

= {VALUE = "a box"},

place

= {VALUE = "front", TEST = places},

fill

= {OPTIONAL = true,

CONTAINS = {color = {CONTAINS = rgbcolor}}},

adjust

= {VALUE = "l", OPTIONAL = true, DEFAULT = "l",

TEST = datatest.inset({"l", "c", "r"}, string.lower)},

angle

= {VALUE = 0, OPTIONAL = true, DEFAULT = 0},

position = {CONTAINS = {x = {VALUE = 0}, y = {VALUE = 0}}},

width

= {VALUE = 10, TEST = positive},

height

= {VALUE = 20, TEST = positive},

}

}

Listing 1. An example of a template for a box in a cover design. Other templates

(template.arc, template.isbn, etc.) could be developed for other elements.

does not allow other validations. For example, we could not distinguish between

different numbers (some correct, others not).

The solution found uses data templates as well, but also includes information

about the actual fields. Besides, the template structure must be related to the

data to be input, with the aim of facilitating their management from Lua.

Using the above example, we shall present and comment on a suitable tem-

plate for a box. Subsequently, we shall set out the design of a function to manage

the data. A version for the example is shown in Listing 1 (in a realistic case,

more fields could appear). This listing is almost self-descriptive. The template

is a table with sub-tables. The fields placed at an odd depth are control fields,

with information about the treatment of the fields placed at the next depth. A

level n in the data table corresponds to fields in CONTAINS (or VALUE) at level 2n

in the template. The possible control fields are described in Table 1.

In the example, only the shown fields are allowed, because in the first level

the control ALLSTRICT = true is declared. Some fields are optional, such as fill

and angle, although the latter has a default value of 0 if not given, due to its

control DEFAULT = 0. For some fields, the unique check is data type (i.e., id must

be a string), but other fields have testing functions (such as the color element

fill.color.r: the number must be between 0 and 1). The adjust field, which is

optional with a default value "l", is solely able to take the values "l", "L", "c",

"C", "r" or "R".

204

17 · Complex Structured Data Input

VALUE

Indicates a terminal datum; it is any value of the expected type.

CONTAINS

Indicates a non-terminal datum; it is a table with the sub-

structure of the branch.

TEST

A function to check the value in case of terminal data.

STRICT

true if the field cannot contain other fields not provided in

CONTAINS.

ALLSTRICT

true if the CONTAINS fields and all embedded ones cannot contain

other fields not given in the corresponding CONTAINS.

OPTIONAL

true if the field is optional.

DEFAULT

Default value for an optional field if not provided in data; both

terminal and non-terminal data can have default values.

Table 1. Control fields in templates. Table templates can be recursively nested

through CONTAINS fields to any depth. VALUE and CONTAINS cannot be used simulta-

neously at the same level.

The content of a field VALUE is any value of the expected type (even a function

or a table), which is used for type checking. The contents of the field CONTAINS

(‘non-terminal data’) must be a table with a sub-template.

Although the description of possible contents in a box is somewhat verbose,

the versatility is apparent. It should be borne in mind that the template is

designed once, and that the user ‘does not see it’.

The programmer should develop checking routines for the data. The follow-

ing, which appear in the example, are interesting:

• function datatest.numrange (x1, x2) returns a function that checks

whether a numerical value falls within a range. x1 and x2 are numeri-

cal values.

• function datatest.inset (t, f) returns a function that checks whether

a value falls within a set of values. t is a homogeneous table with values

(strings, numbers, etc.); f is a function to transform the target and the

fields of t before the comparison (such as string.upper, math.abs, etc.). If

f is not provided, no conversion is assumed.

These routines are included in the appropriate place in the Gems repository

(as part of a complete example). Other similar datatest.* routines can be

developed to check other entry types.

In Listing 1, we defined and used several variables (positive, purecolor,

and rgbcolor) based on the above functions, because the code is clearer.

The routines set out in http://lua-users.org/wiki/LuaTypeChecking could

also be useful for data checking (but be careful, as another box function is given

there which has no relationship with ours).

The recursive character of the data structure allows a relatively simple pro-

cess via a recursive function: datatest.main (template, data, label) verifies

205

whether table data agrees with template. The argument label is used to dis-

play the table name if an error is detected. The function returns true and a void

string "" if all is correct, and false plus an error message otherwise. An outline of this function is shown in Listing 2. (In the repository, the library datatest is

provided as a module.)

Justification and explanation

Although the ensemble datatest.main/template now appears somewhat sim-

ple, its development underwent several improvement stages. The first improve-

ment, from the initial idea set out at the beginning of Section 17, consisted of

a more advanced type checking through test functions. One possibility which

was initially analyzed was to add an auxiliary table to these. However, some

fields do not require more than a type checking (such as some text labels or un-

bounded numbers), and this is not convenient. Finally, the tests were added to

the template as control fields (three initially, CONTAINS, VALUE and TEST).

After the idea of control fields was ‘discovered’, others were included. STRICT

is of major usefulness: by means of this, errors in the field names are detected,

since we can enforce their uniqueness at a given level in the template. This

control gave rise to ALLSTRICT, which extended the same idea to all the sub-

tables from a given depth.

However, the user should sometimes not be forced to give data for all fields.

This was achieved by means of the control OPTIONAL. On other occasions, it is of

interest for the program to give some values for optional, not provided fields in

data. This was achieved using the control DEFAULT, which contains the default

value.

For the sake of brevity, the explanation of datatest.main, following Listing 2,

is simplified. The reader can explore the complete routine in the repository.

The main characteristic of the routine is that it is a wrapper, which hides some

necessary variables inside it, when treating the successive template levels.

In the first lines, the function prepares two containers for possible error

messages and stores a pointer to the original table (to use it in default value

assignment inside the recursive function that follows).

Then it defines the local function maintest, which actually does the work

and calls itself if necessary. As arguments, it has: a template (or sub-template)

template, the target table (or sub-table) data, a Boolean variable allstrict to

indicate whether all branches of the target table could have more fields than the

template or not, and finally the name of the field to process, field.

After the definition of maintest, the first invocation of it appears, with nil

for the last two arguments. Finally, datatest.main returns with the appropriate

messages if an error is detected.

We shall now describe the function maintest. At its beginning, some local

variables are defined from the information in the control fields of template

(VALUE, OPTIONAL, etc.). Also, a variable to store the error status of the execution

and a label for displaying purposes are prepared.

206

17 · Complex Structured Data Input

function datatest.main (template, data, label)

-- the wrapper

label = label or ""

local cart, precart, torig = "", "", data

local function maintest (template, data, allstrict, field)

-- prepare some local variables:

field = field or ""

local ok = true

local contains, value = template.CONTAINS, template.VALUE --etc.

<<check if one (and only one) of CONTAINS and VALUE is present>>

<<check if a field is not provided and it must be>>

<<set the DEFAULT value if an OPTIONAL field is not given

and put and adequate label in precart>>

if value ~= nil then

-- terminal data

<<check if type of value is correct>>

<<check with TEST function>>

else

-- non-terminal data

<<check if data is a non-void table>>

<<check if contains has extra fields than allowed>>

for key, val in pairs(contains) do

local name = field .. (field == "" and "" or "." )

.. tostring(key)

ok = maintest(val, data[key], allstrict, name)

if not ok then return ok end

end

end

precart = "" -- reset it after a DEFAULT field was analyzed

return ok

end -- of maintest

local ok = maintest(template, data)

return ok, cart == "" and "" or precart .. cart

end

Listing 2. Outline of datatest.main. Parts of the code summarized inside <<· · ·>>.

207

Then the function checks CONTAINS and VALUE fields, which are incompatible,

but one of which must appear. After this, the routine analyzes whether the field

exists in the data table. If the field is not optional and is not provided, an error

message is prepared and the error is returned. If the field is optional and is

not given, but it has a DEFAULT field, this is assigned to the corresponding one

in the original table torig. (We use setvar from http://lua-users.org/wiki/

SetVariablesAndTablesWithFunction in this part.) A message is placed in an

auxiliary container (precart) to indicate the analysis of the default value (this

might have been incorrectly input in the template). It is evident that an error in

a template DEFAULT field is more a problem of programming than of the user: the

latter could inform the developer of it. Nevertheless, the user should check the

template with an extensive set of testing data before deployment of the program.

After that, the routine tests whether the item is terminal. If so, the data type

is checked versus the one in the template. Then the corresponding TEST function

is called to verify whether the value is valid, returning an error otherwise.

When data are not terminal, the field in data and template must contain

tables. If this is so, the presence of more fields than allowed (if STRICT or

ALLSTRICT are true) is analyzed, returning an error in that case. Otherwise

all the fields in the sub-table are analyzed by recursively invoking maintest for

each one with the appropriate arguments.

How to use

A driver for the above example is shown in Listing 3. Data processing is per-

formed in a protected environment, which allows assignments, calling functions

from the libraries math, string, table, and the ones defined in process (like box).

This methodology is known as ‘sandboxing’. The use of a protected environment

is very important in the present case: it avoids undue use of some dangerous

functions (like os.execute and os.remove) by the user.

Employing this method, variables can be included in the data file for their

use in some parts of the data. For example, we can define a color:

reddish = {r = 0.8, g = 0.4, b = 0.4}

to be used subsequently, in a consistent way, in any place in which we need

this color by putting color = reddish.

If auxiliary variables are used, we

also recommend strict.lua to avoid problems with the variables used and not

defined by mistake.

On the other hand, as the data file is a chunk of Lua, we can program inside

it. We can obtain some effects with loops and conditionals; for example:

for k = 1, 10 do

w = k; h = 11-k

if k ~= 3 then box{width = w, height = h, <<other fields>>} end

end

208

17 · Complex Structured Data Input

require "datatest"; require "template"

local process = {} -- to include other processes: isbn, text, etc.

function process.box (b)

local ok, msg = datatest.main(template.box, b, "box")

if msg ~= "" then print(msg) end

if not ok then return end

<<the appropriate code for make a box with ’b’>>

end

-- fname stores the name of the data file taken from argument #1

local fname = arg[1]

local proc, msg = loadfile(fname)

if not proc then

print(msg); os.exit(1)

end

setfenv(proc, {math = math, string = string,

table = table, box = process.box})

local ok, msg = pcall(proc)

if not ok then

print(msg); os.exit(2)

end

Listing 3. A simple driver (in protected mode) for data processing.

In this case, we could also export pairs, ipairs, tonumber, tostring, and per-

haps os.time and os.date by means of setfenv.

In the data file, we can also define functions to be applied to the data before

processing them inside box (in the example). Any variable definition, whether

local or global, used in the data file (using the driver shown above) remains

internal in it and does not pollute other environments of the program.

Weaknesses and suggested improvements

The main weakness of the proposed system is that the user might define a

substitution for the function box inside the data file, for example:

local function box (b) <<some code>> end

From this point on, our (correct) function box is no longer accessible inside

the data file. This problem is impossible to overcome because a local variable

can always be defined without limitations inside the data file1. However, this

1In contrast, a global table can be defined read-only with a metatable; see PIL2, p. 127.

209

problem is general in character: we cannot ensure the correctness of user input

for every bit in the data. Actually, if we want the great versatility in inputting

provided by the use of Lua chunks as data, we have to bear the potential

misuses.

Another limitation of the proposed method (not of the datatest/template

system, but of the use of functions like box) is that it is not possible to share

pieces of tables, unless they have been previously defined as variables with

values.

To avoid this, we can directly use assignments to tables instead of function

callings. Security is also easily controlled by using setfenv to process the data

file. In this case, different boxes can be handled as sub-tables; for example:

box[1] = {<<some data>>}

box[2] = {<<some other data>>}

This method allows the direct re-use of data, because we can do:

box[2] = box[1]

box[2].angle = 45

and the second box is equal to the first, except for angle. The checking of data is

now:

local ok, msg = datatest.main(template.box, box[1], "box 1")

local ok, msg = datatest.main(template.box, box[2], "box 2")

With respect to enhancements, more control fields can be defined. We could

add others similar to VALUE, STRICT, etc., bearing information related to the

process of the actual fields. This obviously implies the subsequent modification

of datatest.main. For example, a control field NOCASE could be added: if true,

the program would not distinguish lower and upper cases in the field names

(converting, for example, all names to lower case). Another possibility is to

include a control field FIXED which assigns a fixed value to the corresponding

data field, irrespective of user input (the corresponding field is even forbidden

in the data file).

As described above, in the case of error datatest.main promptly returns

control to the calling function or chunk, along with an error message. Therefore,

if several errors are scattered in the data file, they will be spotted in successive

runs of the program. This keeps error control simple. One possible (though

difficult) improvement would consist in storing all error messages, returning

them after the whole data table has been analyzed. In this case, however, some

errors would interfere with one another, thus confusing the user, and we prefer

the simple approach used here.

Conclusions

Sooner or later programmers using Lua are confronted with often complex data

input, which is frequently not under their control (because other people use the

210

17 · Complex Structured Data Input

program). A method that allows management of the data, with contents control

and validation, is thus convenient.

This gem presented a solution to this problem, using table templates and

combining these with a function that does the checking. The templates are

tables with control fields placed at odd depth levels, whereas even levels are

used for the proper data field names and values. The template is actually a

qualitative-quantitative description of the input data to be treated.

Control fields are used to define characteristics of data fields: whether these

are optional or not, whether they have default values, whether other fields

not present in the template are allowed in the data, and to define data check

functions.

From a programming point of view, the solution (mainly) uses a function

that behaves as a wrapper and includes a recursive function that performs the

checking of the data table level by level. The number of code lines is small due

to the facilities provided by Lua: tables, recursivity, and the first-class character

of functions (among other questions).

An example driver is included accompanying this main function. This works

in an appropriate environment to protect the data processing from possible

undue uses.

Finally, due to its versatility, this library can be used to neatly develop

programs for user-friendly data input, adaptive configuration of programs and

graphical user interface design, among other tasks.

18

Lua Implementations of

Common Data Structures

Matthew M. Burke

In Lua, as with any programming language, one should learn the style and

idioms that best take advantage of the language’s features, rather than utilizing

techniques from other languages. Jung and Brown state “[b]ecause of tables’

flexibility, you often don’t need a customized data structure. Just ask yourself

how you most often want to access your data — usually an associative table or

an array will do the job” [7, pg. 157].

Lua provides a single built-in data structuring mechanism, the “table,” which

combines the functionality of (re-sizable) arrays and hashes. Any of the fun-

damental data structures and their associated algorithms can be implemented

using Lua tables, but it is not always clear how best to do so. After a brief

discussion of how tables are implemented in Lua, this article describes imple-

mentations of several of the most important and common data structures: lists,

stacks and queues, trees, graphs, and sets. Next, several specialized data struc-

tures, including dictionaries and multisets, are presented. Finally, a few tips are

provided on how to structure data in concordance with the spirit of Lua.

Code examples

Good Lua practice dictates that the functions associated with a data structure

be collected in a (meta)table associated with the data structure. Not only does

this reduce pollution of the global namespace, but it allows one to program in an

object-oriented style.

212

18 · Lua Implementations of Common Data Structures

In order to focus on implementation techniques, the code examples in this

article do not include error checking. More robust implementations of the data

structures that follow Lua best practices can be found at the book’s web site.

In discussing the data structures presented in this article, runtime charac-

teristics of the implementations are noted. While these runtime characteriza-

tions are correct, they should not be taken too seriously for two reasons. First,

although Lua is an interpreted language, Lua scripts are byte-compiled and

run on a virtual machine that is quite fast. The VM is most likely fast enough

for your purposes and, as Donald Knuth states, “premature optimization is the

root of all evil” [8, pg. 268]. While proper programming technique suggests one

choose effective implementations, it is the author’s experience that Lua’s speed

is usually not a limiting factor. Second, in those cases in which Lua’s speed is

limiting, one is almost always better off extending Lua with a C library rather

than worrying about optimizing Lua code.

Abstract data types and common functionality

The following sections discuss several common data structures. In each case the

data structure is presented by means of its Abstract Data Type (ADT) definition.

An ADT definition is a description of the functions which make up a data type’s

public interface.

All ADTs are assumed to have the methods presented in Table 1. These

methods will not be repeated in the various ADTs.

isEmpty()

Return a Boolean indicating if the data structure is empty

size()

Return the number of items in the data structure

Table 1. Functions common to all ADTs.

One easily implements isEmpty as

function DS:isEmpty()

return (DS:size() == 0)

end

It is tempting to make use of table.getn for implementing size. But there are

three points to consider: First, table.getn only returns the size of the array

portion of a table. Not only does it not count entries in the hashed portion of the

table, but if there is a gap in the sequence of integer keys, the indices following

the gap are not counted1. Second, table.getn is O(n). Finally, and, perhaps,

obviously, implementations which use more than one table, particularly linked

data structures, cannot be sized using table.getn.

For these reasons, it is preferable to store the size explicitly in the data

structure.

Doing so yields an O(1) implementation, although care must be

taken to ensure that this value is properly updated as the data structure is

manipulated. Implementations will accompany the discussions of the various

data structures in this article.

1This follows from the fact that non-sequential integer keys are stored in the hash part of the table.

213

Lua tables

Lua tables function as a combination of (adjustable-size) arrays and associative

arrays depending on what kinds of values are used as keys. In this article, a Lua

table that has exclusively integral keys is referred to as an array, provided the

table also satisfies the condition that there are no gaps in the sequence of key

values. Tables that do not qualify as arrays will be referred to as mixed tables,

or simply, tables.

Lua table implementation

The implementation of tables in Lua 5.x is described in [3]. Of course, the full

details are always available in the source code. The following is a brief exposition

of the most important points.

Tables have a hash portion and an array portion, either (or both) of which can

be zero bytes in size. Both portions are resized as necessary. Whenever possible,

items with integer keys are stored in the array portion. This eliminates the

need for explicitly storing the key and thus reduces memory requirements. The

hash portion of tables is implemented using chained scatter table with Brent’s

variation [3, pg. 18]. This algorithm performs well even at 100 percent load.

Assigning a value to an array index is typically considered to be O(1). But in

the case of Lua tables, this is correct only if one is not performing insertions and

deletions on the table. Insertions and deletions may cause the table to adjust the

sizes of its array and hash parts. When one of these re-sizings occurs, memory

is allocated and items are copied; this is not a constant-time operation. So, to be

more precise, access has an amortized cost of O(1).

The fact that table access does not have a hard upper bound can be significant

if one has tight time constraints on individual actions. If it is assumed that a

high startup cost is acceptable, one could create a table and prefill it with a

number of entries whose values are some sentinel value.

Lists

There are two common ADTs for Lists. The first is the Array List2, whose ADT

is presented in Table 2. The second is the Node List with ADT in Table 3.

get(i)

Return the element of the list with index i

set(i, e)

Replace with e and return the element at index i

add(i, e)

Insert a new element e into the list to have index i

remove(i)

Remove the element at index i

Table 2. Array List ADT.

2which does not necessarily imply an array implementation.

214

18 · Lua Implementations of Common Data Structures

first()

Return the first node

last()

Return the last node

next(p)

Return the node following node p

prev(p)

Return the node preceding node p

set(p, e)

Replace node p’s current value with e

addFirst(e)

Insert a new first node with value e

addLast(e)

Insert a new last node with value e

addBefore(p, e)

Insert a new node with value e into the list before node p

addAfter(p, e)

Insert a new node with value e into the list after node p

remove(p)

Remove and return node p

Table 3. Node List ADT.

Array lists

Array lists are most easily implemented as Lua tables with integer indices. The

functions in the Array List ADT are implemented below. The functions get and

set have (amortized) costs of O(1) and the costs of add and remove are O(n)

because elements must be shifted up or down.

List = {};

List.__index = List

function List:new()

return setmetatable({ __size = 0 } , self)

end

function List:get(i)

return self[i]

end

function List:set(i, e)

self[i] = e

end

function List:add(i, e)

table.insert(self, i, e)

self.__size = self.__size + 1

end

function List:remove(i)

table.remove(self, i)

self.__size = self.__size - 1

end

function List:size()

return self.__size

end

215

Node lists

Since variables store references to tables, it is easy to create linked lists using

one table per node. Moreover, these linked lists can very easily be multiply-

linked lists. Two example implementations follow. The first is a doubly-linked

list. The second is an excerpt from an implementation of a Skip List. In these

examples, a node’s data is accessed with an index value. The other (string)

indices are used for the links. The motivation for Skip Lists is outside the scope

of this article. Interested readers should consult [2].

Note that using a doubly-linked structure makes implementing the Node List

easier, although it can be implemented with a singly-linked structure.

NList = {};

NList.__index = NList

function NList:new()

local l = { head = {}, tail = {}, __size = 0 }

l.head.__next, l.tail.__prev = l.tail, l.head

return setmetatable(l, self)

end

function NList:first()

if self.__size > 0 then

return self.head.__next

else

return nil

end

end

function NList:last()

if self.__size > 0 then

return self.tail.__prev

else

return nil

end

end

function NList:set(node, elem)

node.value = elem

end

function NList:next(node)

if node.__next ~= self.tail then

return node.__next

else

return nil

end

end

216

18 · Lua Implementations of Common Data Structures

function NList:prev(node)

if node.__prev ~= self.head then

return node.__prev

else

return nil

end

end

function NList:addFirst(elem)

local node =

{ __prev = self.head, value = elem, __next = self.head.__next }

node.__next.__prev = node

self.head.__next = node

self.__size = self.__size + 1

end

function NList:addLast(elem)

local node =

{ __prev = self.tail.__prev, value = elem, __next = self.tail }

node.__prev.__next = node

self.tail.__prev = node

self.__size = self.__size + 1

end

function NList:addBefore(node, elem)

local new_node =

{ __prev = node.__prev, value = elem, __next = node }

new_node.__prev.__next = new_node

node.__prev = new_node

self.__size = self.__size + 1

end

function NList:addAfter(node, elem)

local new_node =

{ __prev = node, value = elem, __next = node.__next }

node.__next = new_node

new_node.__next.__prev = new_node

self.__size = self.__size + 1

end

function NList:remove(node)

node.__prev.__next = node.__next

node.__next.__prev = node.__prev

self.__size = self.__size - 1

end

function NList:size()

return self.__size

end

217

The Skip List example below is included here to demonstrate the ease with

which one can build structures with arbitrary linkages. A Skip List is imple-

mented using several doubly-linked lists. Each list has copies of some of the

nodes in the previous list. A node in a list points to its predecessor and succes-

sor in the list. It also points to its copies (if they exist) in the previous and next

lists. Therefore, a node in a Skip List has four pointers: next, prev, below, and

above. The example code shows an implementation of the find function.

function SkipList:find(k)

local p = self.TopLeft

while p.below do

p = p.below

while k >= p.next.key do p = p.next end

end

end

Stacks, queues, and dequeues

Stacks, queues, and double-ended queues (dequeues) can be implemented quite

simply with the insert and remove functions from Lua’s standard table library.

The ADTs for these three data structures are presented in Tables 4, 5, and 6.

As mentioned before, if amortized costs are acceptable, one can rely on the

automatic resizing of tables to relieve the need to explicitly manage memory.

push(e)

Insert element e as the new top of stack

pop()

Remove and return the top element of the stack

top()

Return the top element of the stack without removing it

Table 4. Stack ADT.

enqueue(e)

Insert element e at the end of the queue

dequeue()

Remove and return the object at the front of the queue

front()

Return the object at the front of the queue without removing it

Table 5. Queue ADT.

addFirst(e)

Insert element e at the beginning of the dequeue

addLast(e)

Insert element e at the end of the dequeue

getFirst()

Remove and return the object at the front of the dequeue

getLast()

Remove and return the object at the end of the dequeue

first()

Return the object at the front of the dequeue without removing it

last()

Return the object at the end of the dequeue without removing it

Table 6. Dequeue ADT.

218

18 · Lua Implementations of Common Data Structures

Stacks

The default value for table.insert’s position parameter is n + 1, where n is the

length of the table. For table.remove, the default value for position is n. Thus,

we can implement push as table.insert(stack, value). One implements pop

as table.remove(stack). The cost of these functions is interesting because it

depends on whether the table needs to be resized, whether getn compatibility

was enabled when Lua was compiled, and whether the sizes table has been

created (see getsizes in lauxlib.c). In some cases, these function calls may

invoke the internal function luaH getn in ltable.c, which costs O(log n), to

determine the correct value for the insertion/removal position.

Queues and dequeues

Queues and dequeues are slightly problematic.

While we could implement

dequeue as table.remove(q, 1), this involves shifting all of the table elements

down to fill the gap. Thus, this implementation has a runtime of O(n). There are

two ways to avoid this cost penalty. An implementation which explicitly tracks

the left and right indices of the dequeue is discussed in [5] and sketched below.

Alternatively, one can make use of the Node List ADT described above.

Dequeue = {};

Dequeue.__index = Dequeue

function Dequeue:new()

seturn setmetatable({ __first = 0, __last = -1 } , self)

end

function Dequeue:addFirst(elem)

self.__first = self.__first - 1

self[self.__first] = elem

end

function Dequeue:addLast(elem)

self.__last = self.__last + 1

self[self.__last] = elem

end

function Dequeue:first()

return self[self.__first]

end

function Dequeue:last()

return self[self.__last]

end

219

function Dequeue:getFirst()

if self.__first > self.__last then return nil end

local result = self[self.__first]

self.__first = self.__first + 1

return result

end

function Dequeue:getLast()

if self.__first > self.__last then return nil end

local result = self[self.__last]

self.__last = self.__last - 1

return result

end

function Dequeue:size()

return (self.__last - self.__first + 1)

end

require ’NList’

NDequeue = {}

NDequeue.__index = NDequeue

function NDequeue:new()

local l = { nlist = NList:new() }

return setmetatable(l, self)

end

function NDequeue:addFirst(elem)

self.nlist:addFirst(elem)

end

function NDequeue:addLast(elem)

self.nlist:addLast(elem)

end

function NDequeue:getFirst()

local result = self.nlist:first()

if result then

self.nlist:remove(result)

return result.value

else

return nil

end

end

220

18 · Lua Implementations of Common Data Structures

function NDequeue:getLast()

local result = self.nlist:last()

if result then

self.nlist:remove(result)

return result.value

else

return nil

end

end

function NDequeue:first()

return self.nlist:first().value

end

function NDequeue:last()

return self.nlist:last().value

end

function NDequeue:size()

return self.nlist:size()

end

Trees

The Tree ADT is presented in Table 7. In the case of binary trees, additional

functions are typically implemented. These are described in Table 8. Trees are

typically implemented in one of two fashions: either collections of linked nodes

or arrays with a protocol for making use of entries. Lua works well for both

implementations. Linked node implementations will be discussed first.

root()

Return the tree’s root

addRoot(e)

Create a root node with value e

parent(v)

Return the parent of v

children(v)

Return a list containing the children of node v

insertChild(v, i, e)

Create a new node with value e and

insert it as the i-th child of node v

isInternal(v)

Test whether node v is internal

isExternal(v)

Test whether node v is external

isRoot(v)

Test whether node v is the root

replace(v, e)

Replace with e and return the element stored at v

remove(v)

Remove the subtree rooted at node v

Table 7. Tree ADT.

221

left(v)

Return the left child of node v

right(v)

Return the right child of node v

insertLeft(v, e)

Create a new node with value e and

insert it as node v’s left child

insertRight(v, e)

Create a new node with value e and

insert it as node v’s right child

Table 8. Additional functions for binary trees.

Linked-node trees

A linked-node implementation of a Tree can be accomplished using the same

techniques described above for Node Lists. Each node of the tree is represented

by a table. This table has an entry with index value for the element and an entry with index children for the references to the node’s children. The children entry is itself a table whose elements are references to the child nodes. For binary

trees, a slight optimization is to have entries indexed with left and right to hold the references for the node’s children. The following code implements the binary

tree interface.

NTree = {};

NTree.__index = NTree

function NTree:new()

return setmetatable({ __size = 0 }, self)

end

function NTree:root()

return self.__root

end

function NTree:addRoot(elem)

self.__root = { value = elem }

self.__size = 1

return self.__root

end

function NTree:parent(node)

return node.__parent

end

function NTree:left(node)

return node.__left

end

function NTree:right(node)

return node.__right

end

222

18 · Lua Implementations of Common Data Structures

function NTree:insertLeft(node, elem)

local new_node = { value = elem, __parent = node }

node.__left = new_node

self.__size = self.__size + 1

return new_node

end

function NTree:insertRight(node, elem)

local new_node = { value = elem, __parent = node }

node.__right = new_node

self.__size = self.__size + 1

return new_node

end

function NTree:isInternal(node)

return node.__left or node.__right

end

function NTree:isExternal(node)

return not self:isInternal(node)

end

function NTree:isRoot(node)

return (node == self.__root)

end

function NTree:replace(node, elem)

node.value = elem

end

function NTree:remove(node)

local parent = node.__parent

local i_am_left_child = (parent.__left == node)

if i_am_left_child then

parent.__left = nil

else

parent.__right = nil

end

self.__size = self.__size - 1

end

function NTree:size()

return self.__size

end

223

Array-based trees

An array-based implementation of a Tree can be accomplished by numbering

the nodes of the tree. For a binary tree this can be done as follows:

n(v) = 1

if v is the root node

n(v) = 2i

if v is the left child of node u and n(u) = i

n(v) = 2i + 1

if v is the right child of node u and n(u) = i

More generally, we can number the nodes of a k-ary tree as follows:

n(v) = 1

if v is the root node

n(v) = ki − k + j + 1

if v is the j-th child of node u and n(u) = i, 1 ≤ j ≤ k

With this numbering scheme at hand, implementing a tree with an array simply

requires storing node v at n(v). Note that, with this implementation, it is not

possible to store nil as a tree entry.

Implementations in other languages typically use zeroth array entry to hold

the number of elements in the tree. This implementation, however, uses the

index size to store the number of elements in the tree.

ATree = {};

ATree.__index = ATree

function ATree:new()

return setmetatable({ __size = 0 }, self)

end

function ATree:root()

return self[1]

end

function ATree:addRoot(elem)

self[1] = elem

self.__size = 1

end

function ATree:parent(node)

return math.floor(node/2)

end

function ATree:left(node)

return 2 * node

end

function ATree:right(node)

return 2 * node + 1

end

224

18 · Lua Implementations of Common Data Structures

function ATree:insertLeft(node, elem)

self[2 * node] = elem

self.__size = self.__size + 1

end

function ATree:insertRight(node, elem)

self[2 * node + 1] = elem

self.__size = self.__size + 1

end

function ATree:isInternal(node)

return (self[2*node] ~= nil or self[2*node+1] ~= nil)

end

function ATree:isExternal(node)

return not self:isInternal(node)

end

function ATree:isRoot(node)

return (node == 1)

end

function ATree:replace(node, elem)

self[node] = elem

end

function ATree:subsize(node)

if self[node] == nil then

return 0

else

return 1 + self:subsize(self:left(node))

+ self:subsize(self:right(node))

end

end

function ATree:remove(node)

local count = self:subsize(node)

self[node] = nil

self.__size = self.__size - count

end

function ATree:size()

return self.__size

end

225

Maps and dictionaries

The Map ADT is presented in Table 9. Maps are usually defined so that each

object has a distinct key. Relaxing that assumption yields the Dictionary ADT,

presented in Table 10.

get(k)

Return the value with key k

put(k, v)

Insert value v with key k

remove(k)

Remove the value with key k

keys()

Return a list of keys

values()

Return a list of values

Table 9. Map ADT.

find(k)

Return a value with key k

findAll(k)

Return a list of all values with key k

insert(k, v)

Insert value v with key k

remove(k, v)

Remove value v with key k

keys()

Return a list of keys

Table 10. Dictionary ADT.

Note that almost all of the Map ADT is already provided by Lua tables. Below

are implementations of the functions keys and values. One can implement the

Dictionary ADT by storing a table of values at each index in a Lua table. See

the following implementation for details.

Dictionary = {}

Dictionary.__index = Dictionary

function Dictionary:new()

local l = { __size = 0 }

return setmetatable(l, self)

end

function Dictionary:find(k)

local matches = self[k]

if (matches == nil) then

return nil

else

local _, match = next(matches)

return match

end

end

function Dictionary:findAll(k)

return self[k]

end

226

18 · Lua Implementations of Common Data Structures

function Dictionary:insert(k, v)

local tab = self[k] or {}

table.insert(tab, v)

self[k] = tab

self.__size = self.__size + 1

end

function Dictionary:remove(k, v)

local tab = self[k]

if tab ~= nil then

for k, val in pairs(tab) do

if val == v then

tab[k] = nil

self.__size = self.__size - 1

end

end

if next(tab) == nil then

self[k] = nil

end

end

end

function Dictionary:keys()

local keys = {}

for k, _ in pairs(self) do

if k ~= ’__size’ then

table.insert(keys, k)

end

end

return keys

end

function Dictionary:size()

return self.__size

end

Sets

The mathematical definition of a set is a collection of distinct objects. In partic-

ular, sets do not allow for duplicates, and there is no notion of order amongst the

elements. In addition to functions that manipulate set elements, several binary

operations are defined on sets. The most common of these operations are union

(members of the new set are all objects that are members of either of the input

sets), intersection (whose members are only those objects that belong to both

input sets), and difference (all members of the first set that are not members of

227

the second set). An ADT for a set is presented in Table 11.

add(e)

Inserts object e into the set

remove(e)

Removes object e from the set

member(e)

Returns a Boolean indicating whether e is in the set

union(A, B)

Replaces the set A with the union of A and B

intersect(A, B)

Replaces the set A with the intersection of A and B

difference(A, B)

Replaces the set A with the difference of A and B

Table 11. Set ADT.

An example use of sets follows. Suppose one wants to create an index of a

collection of documents. This is easily accomplished by first generating a list

of all distinct words that occur throughout the collection of documents. Then,

for each word in this list, a set is created whose elements are the names of the

documents in which this word occurs. These sets are stored in a table indexed

by the words in the list.

The index is used as follows. Obviously, if one wants to find all documents

containing a particular word, one simply retrieves the set stored at that word’s

index. Suppose, however, that one is interested in finding the set of documents

containing several words. The desired set is the intersection of the sets for each

individual word. If, however, one desires the set of all documents containing any

one of several words, the desired result is the union of the sets for the individual

words.

Lua tables allow for a straightforward implementation of sets. A set can

be represented by a Lua table whose keys are the objects which belong to the

set. The value stored at each key is the Boolean true. Thus, the functions for

manipulating set elements are implemented as follows:

Set = {}

Set.__index = Set

function Set:new()

return setmetatable({ __size = 0 }, self)

end

function Set:add(elem)

local result = self[elem]

self[elem] = true

if not result then

self.__size = self.__size + 1

end

end

228

18 · Lua Implementations of Common Data Structures

function Set:remove(elem)

local result = self[elem]

self[elem] = nil

if result then

self.__size = self.__size - 1

end

end

function Set:member(elem)

return (self[elem] or false)

end

function Set:size()

return self.__size

end

The following implementations of the binary set operations are all destructive

in that they modify the first set rather than return a new result set. They are

all implemented by iterating over the keys of one set and modifying the entries

of the first set as appropriate.

function Set:union(b)

for k, _ in pairs(b) do

if k ~= ’__size’ then

if not self[k] then

self.__size = self.__size + 1

end

self[k] = true

end

end

end

function Set:intersect(b)

for k, _ in pairs(self) do

if k ~= ’__size’ then

if not b[k] then

self.__size = self.__size - 1

self[k] = nil

end

end

end

end

229

function Set:difference(b)

for k, _ in pairs(b) do

if k ~= ’__size’ then

if self[k] then

self.__size = self.__size - 1

end

self[k] = nil

end

end

end

The run times of these three functions are all O(n). Note that the implementa-

tion of these functions can be simplified by using the (functional programming)

map. A non-destructive implementation can be achieved by copying the first set

to a new set before proceeding with the rest of the operation.

Multisets

A common variation of a set is the so-called multiset (also known as a bag). A

multiset discards the restriction that each contained object be distinct. One

can easily adapt the prior implementation of set operations to multisets by

storing an integer count, rather than the Boolean value true, at each key. The

Multiset ADT has two functions in addition to the functions in the Set ADT.

These functions are described in Table 12.

count(e)

Returns how many e objects are in the multiset

removeAll(e)

Removes all e objects from the multiset

Table 12. Multiset ADT.

Additionally, the semantics of the binary set operations are modified to take

into account the possibility of elements occurring multiple times in a multiset.

The Multiset ADT is implemented as follows:

MSet = {}

MSet.__index = MSet

function MSet:new()

local l = { __size = 0 }

return setmetatable(l, self)

end

function MSet:add(elem)

self[elem] = (self[elem] or 0) + 1

self.__size = self.__size + 1

end

230

18 · Lua Implementations of Common Data Structures

function MSet:remove(elem)

local current = self[elem] or 0

if current > 0 then

current = current - 1

self.__size = self.__size - 1

end

self[elem] = current

end

function MSet:member(elem)

return ((self[elem] or 0) > 0)

end

function MSet:count(elem)

return self[elem] or 0

end

function MSet:removeAll(elem)

local rcount = self[elem] or 0

self[elem] = nil

self.__size = self.__size - rcount

end

function MSet:size()

return self.__size

end

function MSet:union(b)

for k, v in pairs(b) do

if k ~= ’__size’ then

local bcount = v or 0

self[k] = (self[k] or 0) + bcount

self.__size = self.__size + bcount

end

end

end

function MSet:intersect(b)

for k, acount in pairs(self) do

if k ~= ’__size’ then

local bcount = b[k] or 0

self[k] = math.min(acount, bcount)

self.__size = self.__size - math.abs(acount - bcount)

end

end

end

231

function MSet:difference(b)

for k, bcount in pairs(b) do

if k ~= ’__size’ then

local acount = self[k] or 0

local reduced = acount - bcount

if acount >= bcount then

self[k] = acount - bcount

self.__size = self.__size - bcount

else

self[k] = nil

self.__size = self.__size - acount

end

end

end

end

Partitions

A partition of a set S is a collection of subsets of S with the following property: Each element of S is a member of exactly one set in the partition. For example,

consider S = {1, 2, 3, 4, 5, 6}. One partition is the collection of singleton sets: {1},

{2}, {3}, {4}, {5}, {6}. Another partition consists of the following sets: {1, 2},

{3, 4}, {5, 6}. As an interesting aside, one can define an ordering on partitions

where a partition A is a refinement of partition B if every set in B is composed

of a union of sets of A. Note this is not a total ordering because it is possible to

have two partitions, neither of which is a refinement of the other. An ADT for

partition is presented in Table 13. An implementation of this ADT follows.

makeSet(x)

create a singleton set containing the element x

merge(A, B)

Merge the sets A and B, destroying the old B

find(x)

find the set containing x

get set(x)

return the elements in the same set as x

Table 13. Partition ADT.

Partition = {}

Partition.__index = Partition

function Partition:new()

local l = { __size = 0 }

return setmetatable(l, self)

end

function Partition:make_set(x)

self[x] = self.__size

self.__size = self.__size + 1

end

232

18 · Lua Implementations of Common Data Structures

function Partition:merge(a, b)

local a_idx = self:find(a)

local b_idx = self:find(b)

for k, v in pairs(self) do

if k ~= ’__size’ and v == b_idx then

self[k] = a_idx

end

end

self.__size = self.__size - 1

end

function Partition:find(x)

return self[x]

end

function Partition:get_set(x)

local idx = self:find(x)

local res = {}

for k, v in pairs(self) do

if k ~= ’__size’ and v == idx then

table.insert(res, k)

end

end

return res

end

Graphs

A graph is composed of a set of vertices and a set of edges which connect

the vertices. It is important to remember that a graph is a topological object

rather than a geometric one, i.e., it is the connections between vertices that

is important, not the precise picture used to illustrate a graph. Normally in a

graph, if a node u is connected to a node v, this implies that node v is connected to

node u. In a directed graph, however, it is possible for a node u to be connected

to a node v while node v is not connected to node u. A good analogy to help

understand the difference between directed and undirected graphs is to think

of a road map. A directed graph is a network of one-way streets, whereas an

undirected graph is a network of two-way roads. A graph, either directed or

undirected, may have weights assigned to the edges. Again, if one considers the

road map analogy, weights would correspond to the distances between cities.

There are two common representations of graphs: the adjacency matrix and

the vertex list. Adjacency matrices are simple but waste space in the cases

of non-directed graphs and sparse graphs. When using Lua tables, however,

adjacency matrices, even for sparse graphs, are memory efficient. Therefore, the

vertex list representation of graphs is not discussed in this article.

233

Although a directed graph with n nodes requires an n × n adjacency matrix,

a non-directed graph only requires a triangular portion of the full n × n matrix.

Lua tables can be used to store shaped multi-dimensional matrices. One can

take advantage of this feature to reduce by half the storage requirements for

non-directed graphs. The (undirected) Graph ADT is presented in Table 14. An

implementation of undirected graphs follows. An implementation of directed

graphs may be found at the book’s web site.

vertices()

Return a list of the vertices of the graph

edges()

Return a list of the edges of the graph

incidentEdges(v)

Return a list of all edges where one endpoint is vertex v

areAdjacent(v, u)

Test whether vertices v and u are adjacent

insertEdge(v, u)

Make vertices u and v adjacent

removeEdge(v, u)

Make vertices u and v non-adjacent

Table 14. Graph ADT.

Graph = {};

Graph.__index = Graph

function Graph:new(n)

local l = { __size = n }

local vertices = {}

for i = 1, n do

table.insert(vertices, i)

end

l.__vertices = vertices

local graph = {}

for i = 1, n do

table.insert(graph, {})

end

l.__graph = graph

return setmetatable(l, self)

end

function Graph:vertices()

return self.__vertices

end

function Graph:incidentEdges(v)

local result = {}

for i = 1,v-1 do

if self.__graph[i][v] then table.insert(result, i) end

end

for i, _ in pairs(self.__graph[v]) do

table.insert(result, i)

end

return result

end

234

18 · Lua Implementations of Common Data Structures

function Graph:areAdjacent(v, u)

return ( (self.__graph[v][u] or self.__graph[u][v]) or false)

end

function Graph:insertEdge(v, u)

if u < v then v, u = u, v end

self.__graph[v][u] = true

end

function Graph:removeEdge(v, u)

if u < v then v, u = u, v end

self.__graph[v][u] = nil

end

Text processing

Most books on data structures contain a chapter discussing text processing,

particularly pattern matching and compression. This article does not cover these

topics, but the interested reader is directed to read the documentation for Lua’s

string library [4]3, Reuben Thomas and Shmuel Zeigerman’s rex library [13],

and Roberto Ierusalimschy’s LPEG library [6].

Augmented data structures

In many circumstances, an operation on a data structure can be made more

efficient by including additional information. This technique is known as using

augmented data structures. For example, threaded trees use additional pointers

so that it is possible to find the pre-order successor to a given node in O(1) time.

A particularly elegant example of an augmented data structure is the use of

an augmented binary search tree to enable determination of k-th order statistics

in O(log n) time. An implementation based on the discussion in [11] follows.

require ’NTree’

OrderTree = {}

OrderTree.__index = OrderTree

function OrderTree:new()

local l = { }

l.tree = NTree:new()

return setmetatable(l, self)

end

3and, of course, refer to the source code.

235

function OrderTree:addRoot(elem)

elem = { data = elem, subtree_size = 1 }

return self.tree:addRoot(elem)

end

function OrderTree:increment_ranks(node)

while (node ~= nil) do

node.value.subtree_size = node.value.subtree_size + 1

node = self.tree:parent(node)

end

end

function OrderTree:decrement_ranks(node)

while (node ~= nil) do

node.value.subtree_size = node.value.subtree_size - 1

node = self.tree:parent(node)

end

end

function OrderTree:insertLeft(node, elem)

elem = { data = elem, subtree_size = 1 }

local new_node = self.tree:insertLeft(node, elem)

self:increment_ranks(node)

return new_node

end

function OrderTree:insertRight(node, elem)

elem = { data = elem, subtree_size = 1 }

local new_node = self.tree:insertRight(node, elem)

self:increment_ranks(node)

return new_node

end

function OrderTree:rank(x)

local root = self.tree:root()

return self:_rank(x, root)

end

function OrderTree:select(k)

if k > self.tree:size() then

error(’Tree does not contain that many items.’)

end

local root = self.tree:root()

return self:_select(k, root)

end

236

18 · Lua Implementations of Common Data Structures

function OrderTree:_rank(x, node)

local key = node.value.data

local left_child = self.tree:left(node)

local right_child = self.tree:right(node)

if x == key then

return 1 + ((left_child and left_child.value.subtree_size) or 0)

elseif x < key then

if left_child then

return self:_rank(x, left_child)

else

error(’Key is not in tree.’)

end

else

if right_child then

return

1 + self:_rank(x, right_child) +

((left_child and left_child.value.subtree_size) or 0)

else

error(’Key is not in tree.’)

end

end

end

function OrderTree:_select(k, node)

local left_child = self.tree:left(node)

local right_child = self.tree:right(node)

local left_tree_size =

(left_child and left_child.value.subtree_size) or 0

if k == left_tree_size + 1 then

return node

elseif k <= left_tree_size then

return self:_select(k, left_child)

else

return self:_select(k - 1 - left_tree_size, right_child)

end

end

Homogeneous data structures

In Lua, values are typed, but variables are not. In other words, a particular

variable, say my variable, may refer to a string at one point during a program’s

execution, a table at a later point, and, perhaps, a number later on. Thus, the

data structures discussed here are all heterogeneous, i.e., they can simultane-

ously store values of different types.

Often, however, it is necessary to restrict a data structure to values of a

particular type. For example, suppose one is collecting a list of expenses with the

intent of calculating an average expense. If a non-numeric value is inadvertently

inserted, the result would either be an erroneous result or a run-time error.

237

One can easily modify the data structures presented here so that the inser-

tion routines check the type of the value before inserting it into the structure.

A simple approach would be to use the type function, although in many circum-

stances that is not sufficient. A more robust approach would be to make use

of metatables, particularly since most object-oriented systems in Lua rely on

metatables to implement classes.

Below is the implementation of a factory for creating functions to verify types.

function make_typechecker(spec)

if type(spec) == "string" then

return function(v) return (type(v) == spec) end

else -- spec is a table

return function(v) return (getmetatable(v) == spec) end

end

end

One can use the type-checker factory as follows:

local isNumber = make_typechecker("number")

-- Stack which only holds numeric values

function push(stack, v)

if not isNumber(v) then error("This stack only holds numbers!") end

table.insert(stack, v)

end

Working with Lua

To paraphrase an anonymous saying, “any programmer can write a C program

in Lua.” For the most part, the data structures presented here were developed

in response to the constraints of languages such as C and Fortran. In some

circumstances, these data structures can be the best choice in Lua programs.

Whenever possible, however, it is preferable to develop data structures that take

advantage of the strengths of Lua. A few heuristics to aid one in this task follow.

1. Take advantage of table resizing.

2. Consider using values as keys (such as in the Set implementation).

3. Remember that objects of any type can be table keys (except nil).

4. Make use of both the array portion and hash portion of a table.

Resources

There are a number of resources available to Lua programmers to aid in the

design of data structures. These include both existing libraries and reference

238

18 · Lua Implementations of Common Data Structures

material. Good texts on Lua programming include Programming in Lua [5], the

Lua Reference Manual [4], and Beginning Lua Programming [7].

There are also several existing libraries which contain a range of data struc-

tures and algorithms. These include Reuben Thomas’s stdlib [12] and Paul

Chisano’s Sano Library [1]. Thomas’s work contains a number of useful exten-

sions to Lua’s standard libraries and has a particularly functional programming

style. Chisano’s library contains implementations of almost all the data struc-

tures described in this article.

Complete implementations of the data structures discussed in this article are

available at the book’s web site. And, of course, there are a number of examples

of data structure implementations available at the Lua Wiki [9]. Finally, the

Lua mailing list [10] has an excellent signal-to-noise ratio.

References

[1] Chisano, Paul. Sano Library. http://luaforge.net/projects/sano/.

[2] Goodrich, Michael T. and Roberto Tamassia. Data Structures and Algo-

rithms in Java. 4th Ed. John Wiley and Sons, 2006.

[3] Ierusalimschy, Roberto, de Figueiredo, Luiz H. and Waldemar Celes. “The

implementation of Lua 5.0,” Journal of Universal Computer Science, Vol.

11, No. 7, 2005.

[4] Ierusalimschy, Roberto, de Figueiredo, Luiz H. and Waldemar Celes. Lua

5.1 Reference Manual, Lua.org. 2006.

[5] Ierusalimschy, Roberto. Programming in Lua. 2nd Ed. 1006.

[6] Ierusalimschy, Roberto. lpeg. http://luaforge.net/projects/lpeg/.

[7] Jung, Kurt and Aaron Brown. Beginning Lua Programming. Wiley, Indi-

anapolis, IN, 2007.

[8] Knuth, Donald. “Structured Programming with go to Statements,” ACM

Computing Surveys, Vol 6, No. 4, Dec. 1974.

[9] Lua Community. Lua Users’ Wiki. http://www.lua-users.org/wiki.

[10] Lua Community. lua-l. http://www.lua.org/lua-l.html.

[11] Ottmann,

Thomas.

http://electures.informatik.uni-freiburg.de/

catalog/chapter.do?courseId=advancedAD2005&chapter=9

[12] Thomas, Reuben. stdlib. http://luaforge.net/projects/stdlib/.

[13] Thomas, Reuben and Shmuel Zeigerman. lrexlib. http://luaforge.net/

projects/lrexlib/.

19

Tic-Tac-Toe and the

Minimax Decision Algorithm

Rafael Moreira Savelli

and Roberto de Beauclair Seixas

We present a way to implement the minimax algorithm from scratch in Lua.

First, we start with some game-playing concepts. Then, we explain how could

tables handle with required data structures specifications such as trees and

nodes, and present a Lua implementation of the minimax algorithm, showing

how those tables are manipulated. Its performance is evaluated based on the

worst case execution time defined by the game-playing concepts. In addition,

in order to confirm the given approach, we also implemented the well-known

tic-tac-toe and checkers games as case studies. At the end practical results are

commented proving that, like many other languages, Lua can easily handle this

sort of adaptive search problems.

Introduction

Humans started to play games a long time ago and just after the firsts civiliza-

tions have existed. The main purpose of them were basically for entertainment.

Since then, more and more games have been created so that future generations

still spend their time with game playing.

240

19 · Tic-Tac-Toe and the Minimax Decision Algorithm

depth

nodes

time

memory

0

1

1 millisecond

100 bytes

2

111

0.1 seconds

11 kilobytes

4

11.111

11 seconds

1 megabyte

6

106

18 minutes

111 megabytes

8

108

31 hours

11 gigabytes

10

1010

128 days

1 terabytes

12

1012

35 years

111 terabytes

14

1014

3500 years

11.111 terabytes

Table 1. Time and memory requirements for breadth-first search. The figures shown

assume branching factor b = 10 ; 1000 nodes/second; 100 bytes/node.

Adding this fact with computing, many of this games were inserted into

computers and thus some artificial intelligence became indispensable. In fact,

game playing is one of the oldest areas of endeavor in artificial intelligence,

turning up around 1950 when computers just became programmable [3].

Search problems

In order to solve search problems we build a tree that is superimposed over the

problem space and find a solution by searching node-after-node until reaching

the desired goal node. However, in some cases, building such tree is not an

easy task. Even a simple problem can generate a huge amount of data and,

in this case, the resulting tree becomes intractable even when submitted to

modern computers. This can be observed by looking at Table 1 which shows

a hypothetical computer times and memories usage for a given number of tree

depth.

Even with this exponential growth, the most common difficulties related

to problem solving is finding a good strategy for searching among those trees.

This is almost always the majority of work in the area of search problems.

The strategy can be classified in term of four different criteria as presented on

following:

Completeness. Is the strategy guaranteed to find a solution if one exists?

Time Complexity. How long does it take to find a solution?

Space Complexity. How much memory does it need to perform that search?

Optimality. Does the strategy find the highest-quality solution when there are

several different solutions?

Games

An important consideration when designing games is to treat them as search

problems. However, the presence of an opponent makes the decision problem

241

somewhat more complicated when compared to simple search problems. The

opponent introduces uncertainty, because one never knows what he or she is

going to do. So, in essence, all game playing programs must deal with the

contingency problem as well [1].

As mentioned before, the best way for solving search problems is to build

a tree for that problem. Here, the idea is applied according to a generic two-

players game as shown in the following:

• The initial state (root), which includes board position and an indication of

whose move it is.

• A set of operators (transition), which defines the legal moves that a player

can make.

• A terminal set (leaves), which determines when the game is over.

• A utility function (node values), which gives a numeric value for the out-

come of the game. For example, we can assign to leaves the values +1, 0 or

−1 respectively for winning, drawing, or losing the game.

In order to understand some of approaches used in this work, first we need

to introduce some of Lua table concepts. In Lua, a table is a special type that

implements associative arrays. An associative array is an array that can be

indexed not only with numbers, but also with strings or any other value of the

language (except nil). Moreover, tables have no fixed size. You can add as many

element as you want to a table dynamically. The most surprising fact is that

tables are the main and also the only data-structuring mechanism. We can use

tables to represent ordinary arrays, symbol tables, sets, records, queues, and

more [2].

In our case we used tables for storing trees nodes. Each node must have

information of the current board position. For example, Figure 1 shows how

a possible search tree for the tic-tac-toe game.

For instance, the given left

positioned block of code in Figure 2 illustrates how to build each node in Lua.

The corresponding tic-tac-toe board situation appear on the right yet in the

same figure. In addition, positions marked as “X” or “O” are both players while

positions marked “ ” denote only empty position.

The minimax algorithm

Having defined the main game concepts and its data structure, we are now ready

to understand the essence of minimax algorithm. The minimax algorithm is

designed to determine the optimal strategy for one player at a time informing

the best move for that player. The entire algorithm consists of four steps:

1. Generate the whole game tree, all the way down to the terminal states.

2. Evaluate the utility function to each terminal state to get its value.

242

19 · Tic-Tac-Toe and the Minimax Decision Algorithm

Figure 1. A partial search tree for the Tic-Tac-Toe game.

Figure 2. A partial search tree for the Tic-Tac-Toe game.

243

3. Recursively, use the evaluated utilities to determine the utilities of the

nodes one level higher.

To do that, apply minimal or maximal utility

alternately until reach the root by taking a maximal utility for last. [XXX]

4. Finally, choose the move that leads to the highest utility value.

For programming purposes, imagine the computer playing CIRCLE and the

human playing CROSS. The code in Listing 1 shows a Lua implementation of

the minimax algorithm. The top function, MINIMAX DECISION, starts the decision

process and selects the computer best move from all legal ones, which are

evaluated in turn by the MINIMAX VALUE recursive function.

When computer’s turn comes, the CROSS player leaves it a board with a

given game situation. Then, MINIMAX DECISION is invoked with the current game

board as argument. After that, all legal moves are taken by using getLegalMoves.

From the resulting table, every legal move must be played in separate boards

and passed to MINIMAX VALUE, which calls itself recursively until it reaches the

maximum predefined hMax depth or when the game is over for that board state.

In both cases, the recursion stops and the UTILITY function is evaluated. Other-

wise, the recursive function continues alternating each player’s turn, maximiz-

ing or minimizing the utilities.

Comparing Lua to other programming languages, such as C or Java, we can

spot some particular differences. The first one is related to Lua tables. They

are simple, easy to use, and avoid any additional memory management. As an

example, take this line from MINIMAX VALUE

newBoard = move( gameBoard, v )

This line means that a function move receives a game board (a Lua table) and

a move v to play on this board. The resulting situation leads to a new game

board named newBoard. If we were using C, one of the best ways would be

treating boards as integer vectors. However, special memory managements

would be required in that case because vector copying in that language is not as

simple as copying tables in Lua. This will surely require user care and attention

with pointers. And every programmer knows that just a single, small memory

manipulation mistake can affect the whole application integrity. This sort of bug

does not happen in Lua programs.

Case study

Up to this point, we have been using the tic-tac-toe game in our explanations.

In fact, this game was chosen to be our case study for several reasons. First, it

is a well-known and popular game with dismiss any rules explanations. Second,

other games like checkers (also known as draughts) and chess are much more

complex than tic-tac-toe. For instance, tic-tac-toe has a maximal branching

factor of 9 while in chess the average is about 35. Moreover, tic-tac-toe has

exactly 623530 nodes in its search tree but chess can easily achieve 35100.

244

19 · Tic-Tac-Toe and the Minimax Decision Algorithm

function MINIMAX DECISION( gameBoard )

local bestMove, newBoard, value

local maxValue = -999

local legalMoves = getLegalMoves( gameBoard, CIRCLE )

for i,v in pairs(legalMoves) do

newBoard = move( gameBoard, v )

value = MINIMAX VALUE( newBoard, 1, CROSS )

if value > maxValue then

maxValue = value

bestMove = v

end

end

return bestMove

end

function MINIMAX VALUE( gameBoard, h, player )

local legalMoves, newBoard, value

local maxValue = -999, minValue = 999

if gameOver(gameBoard) or h == hMax then

return UTILITY(gameBoard)

end

legalMoves = getLegalMoves( gameBoard, player )

if player == CIRCLE then

for i,v in pairs(legalMoves) do

newBoard = move( gameBoard, v )

value = MINIMAX VALUE( newBoard, h+1, CROSS )

if value > maxValue then

maxValue = value

end

end

return maxValue

else

for i,v in pairs(legalMoves) do

newBoard = move( gameBoard, v )

value = MINIMAX VALUE( newBoard, h+1, CIRCLE )

if value < minValue then

minValue = value

end

end

return minValue

end

Listing 1.

245

While implementing minimax in Lua, we have noticed how important and

powerful a table is. Alone or combined with others tables, we can build complex

data structures which make Lua easy to use and much better than many other

programming languages. In this work, we used combined tables to form matri-

ces. Those matrices were used to store tic-tac-toe boards which stands for valid

game positions. This table was defined as shown in Figure 2.

However, the generic Lua algorithm described in the previous section works

not only for simple games like tic-tac-toe but basically for every other two-player

board games. The key is to know how to combine tables to form the desired game

board. For example, if we wish to store checkers board information, instead of

doing code shown in Figure 1, we simply store the necessary 32 dark squares of

checkers board. By checkers rules, the pieces must remain only in these squares.

So, a table for this game might be:

-- Defining valid board positions

local EMPTY = 0, CROSS = 1, CIRCLE = 2

local checkersBoard = {EMPTY, CROSS, CROSS, ... , CIRCLE}

Of course this is the main change, but not the only one.

The functions

getLegalMoves, move, and gameOver must also be adapted to match with checkers

rules. Once these changes are made, the main algorithm structures are still

valid and solve the search problem now for the checkers game.

Conclusions

The first conclusion is that implementing the minimax strategy in Lua is an

easy task. The Lua combine tables facility to form matrices speeds up users

implementation skills keeping the source code simple and clear. The memory

manipulation abstraction also contributes to make Lua an easy programming

language.

Another important Lua approach consists in collecting any memory garbage

with frees programmers from this responsibilities.

Yet Lua is a powerful programming language and also provides extensions.

One future work could be implementing other games concepts such as alpha-

beta pruning.

Alpha-beta pruning is a technique where a large amount of

nodes are securely removed from search tree and for this reason they are not

processed. Avoiding unnecessary processing means increasing the computer

answer velocity.

References

[1] J. D. Funge. Artificial Intelligence for Computer Games. Peters Corp., 2004.

[2] R. Ierusalimschy. Programming in Lua. Lua.org, 2006.

[3] S. Russel and P. Norvig.

Artificial Intelligence:

A Modern Approach.

Prentice-Hall, 1995.

Part IV

Game Programming

20

Using Lua in

Game and Tool Creation

Konstantin Sokharev and Vadim Groznov

Nowadays, when development costs are skyrocketing, selection of tools and

instruments for creating and designing your game is of utter importance. This

article discusses the use of Lua as a language for writing game and its game play,

and as a platform for the creation of a game editor. Writing high quality code is

impossible without a good convenient IDE. In this article we discuss struggle for

higher performance and lower memory consumption, and issues of readability

that arise when the volume of Lua code grows significantly.

Programmable access via Lua

Our earliest architectural solutions used Lua to script events in game missions,

allowing designers to realise a lot of ideas and make missions sufficiently di-

verse. Usage of Lua also allows to differentiate tasks between scripters and

programmers (in spite of scripter being a programmer too — but a programmer

of less complex constructions), which is extremely useful from the production

process viewpoint (Listing 1).

Actually, this approach allowed quick prototyping and production of missions

(of an RTS/Action game in production at that time). Later, we researched Lua’s

capabilities further and understood that usage of such architectural features as

250

20 · Using Lua in Game and Tool Creation

tables allowed for a convenient storage of game structures. Here is an example

of our organisation of structured database of game entities (Listing 3).

It can be seen that we are creating a table named Prot.Characters with sub-

tables named Defaults, where default properties of an object are stored — like

its radius (space it occupies on terrain in game), HP (health points), ability to

cast and receive shadows, physic characteristics (information used by a physics

engine), material type (for drawing hit effects etc.). All the actual game char-

acters follow, also in sub-tables (such as TestCharacter1), where model for its

representation, class and other individual characteristics are indicated. Previ-

ously, we used XML for these, and it was quite handy, but Lua syntax turned out

to be simpler and easier understand and it does not require knowledge of XML

markup. Also, while using XML, one will need to find a parser with required

functionality and optimal speed. Lua parser is very fast (comparable with the

fastest XML parsers such as tinyxml and pugxml). If need be, Lua database can

even be compiled to Lua bytecode for instant loading.

Early successes led us to the idea that all game logic can be written in Lua

(this beauty and convenience fooled us, as will be described later in the “Exces-

sive flexibility of Lua” section). This way, game entities’ classes (for example,

base functions of a main hero, mainhero_class.lua), Finite State Machine of a

game class (mainhero_behaviours.lua/mainhero_goals.lua), Events Manage-

ment System (Listing 2) were fully transferred to Lua, while C++ retained only

low-level functions such as pathfinding. As can be seen, Object Oriented pro-

gramming paradigm is used here. (It can be added that later, during the devel-

opment of a third person shooter game, a system for management of multi-axis

blending of animations was also written in Lua.)

Lua has a built-in system of threads/coroutines and their management. We

first thought to use this feature of Lua and wrote a test system, but realised

that for an RTS game usage of FSM is more important, and debugging of such

system will be too difficult.

IDE (debugger)

Soon we understood that, with all the convenience and flexibility of Lua, at

large volume of code (over 10 Lua files at several thousand lines each) means of

debugging are lacking, so we decided to write full-featured Lua script debugger.

Such a debugger was written and built into the game. It has a convenient

IDE with game files navigation tree, search, multitabs, syntax highlighting,

and all the functionality for debugging: breakpoints, step in, step out, edit

and continue, watches, expressions.

It significantly simplified and sped up

development, because earlier we had to debug code by logging.

It is worthy of note that for text representation we used Scintilla middleware,

while Lua middleware has all the functionality for the creation of a good debug-

ger (i.e., we didn’t have to change anything in Lua source code), so the creation

of a debugger took only one man-month.

251

Excessive flexibility of Lua

Despite seeming simplicity and beauty of code, Lua conceals some dangers. A

sudden problem arose — excessive flexibility of Lua. For example, access to any

member of any object at any place in the script (as an example) is a straight

way to “hacks” — often forgetting to write “local” inside a function you create a

global. A conclusion that can be made — strict coding standards are a must, i.e.:

• All structural definitions must always have the same format (Lua allows

to do one thing by several means). For example, function declaration:

function Running() or Running = function() — you must choose most con-

venient.

• Names of different entities must be created by one mnemonic rule — or

the code will become very difficult to understand. For example, signal

enumerator: SGN_USE1, SGN_CAMERA_ROTATE, SGN_MOVE.

• Despite Lua being a type-less language, type shall be pointed in variable

name. For example, vPos (Vector), fRadius (float), blsCovered (boolean).

To conclude this topic, good advice: frequently make a code review to solidify

structures (we advise to include it into the weekly development cycle).

Performance issues

One of the most important questions with script language usage is “Will it be

fast enough?”. We conducted preliminary synthetic tests and found that perfor-

mance was sufficient even with several hundred units operating simultaneously

in game. In reality, it turned out to be slower — in massive combats with many

game entities (30–60 fighting units on-screen), the game began to lag signifi-

cantly. An important conclusion was made: game designers’ vision of battles

in game and other scripted events happening simultaneously on the map must

be taken into account beforehand. Then most critical (performance-wise) stuff

must be exposed and it should be decided whether to keep it in C++ or transfer it

to Lua. Regrettably, we made this too late, so we had to fix these problems dur-

ing production. We decided to integrate a Lua profiler (from the Kelper project),

which did its job well and showed up bottlenecks, and did it in runtime, with-

out stopping game in needed game situations, which is very important. That

way we found out that significant amount of processing time was consumed by

Lua-to-C++ data transfer (we will elaborate on that later), and next function

used for looking through table elements. A conclusion can be made: thoroughly

plan structure of data that you will store in script (use massives extensively and

tables where you really need them.) Remember that performance impact when

looking through larger number of elements is significantly heavier in Lua then

in C++. Also, there is sense in using custom containers, exported from C++, with

faster look through elements.

252

20 · Using Lua in Game and Tool Creation

Later we stumbled upon a problem of excessive memory allocation — the fact

that Lua automatically manages memory is very useful and actually simplifies

coding, but several things must be remembered.

Profiling showed that excessive memory allocation had to do with Lua hash

tables (i.e., if your game entity has 129 elements in table, this table will be of

256 element size — which is a substantial overhead). Regrettably, this problem

was solved only partially by code review and better architecture.

Also you need to remember that table element of type “byte”, that would take

1 byte in C/C++ will take 40 bytes in Lua (not counting the table itself).

Another problem connected with memory was the garbage collector — as we

used Lua 5.0.2, we did not have many alternatives. Mixed approach such as

reference counting + garbage collection can be used — this can be a good solution

(LuaRC can be used as a foundation). Or, if you have little amount of runtime

data, make garbage collection every frame. Today, Lua 5.1 has incremental

garbage collection, which is more progressive and has wide functionality for

management and budgeting of memory, but can’t solve all the problems, because

it has a substantial overhead at large volumes of data. Perhaps, fine tuning and

low volumes of game data may provide the needed result. Also, an integration

of a custom memory manager (Lua allows this to be done easily) may help. Such

manager can be adapted for size and quantity of memory chunks allocated by

Lua code.

As a conclusion, these are several useful solutions that have to do with Lua

being used as a game script. The most used types (Vector, Quaternion, Matrix)

can be made native for Lua — this significantly increases performance. If Lua

execution speed is not enough (event though it is higher than in other script

languages such as Python, Perl, and Ruby), you can try to use LuaJIT. We tested

it — LuaJIT actually speeds program several times in certain circumstances and

brings execution speed closer to native C.

Lua–C++ data transferring

One of the important moments in integration of C/C++ code and Lua is data

transfer and class exporting. For this we first used a proxy class that made

Lua and C++ integration less solid and allowed usage of another scripting

language, but profiling showed that significant memory overhead and excessive

calls happened. Our latest solution is based totally on #defines and works

directly with Lua stack. See Listing 4. You can see pseudo-C++ code where

we have bindings macros part, example Obj class method (GetAnimations) and

so called “exportmap” below separated by comments.

Lua and .NET

Our second case of Lua usage was as a script language for creation of tools in

a game editor. It was decided to use .NET platform because of its capabilities

253

and flexibility. As a link, we used the fantastic LuaInterface, which we used as

a foundation and wrote our own link in C++ for Lua and .NET. That decision

allowed us also to export needed methods of engine and import them into .NET,

so Lua became a sort of glue between our code and Managed. Problematic were

two garbage collectors (.NET and Lua), which began to conflict each other at

large volumes of data and executed operations. The solution was to force .NET

garbage collection every frame.

For interface and controls we used weifenluo.dll. That way, after we added

ability of comfortable editing and edit and continue, we got a powerful tool for

rapid prototyping and realisation of handy editing tools.

Conclusion

Our experience showed that utilization of Lua as a convenient database and a

means for rapid creation of game entities is quite possible, provided you have a

convenient IDE for debugging and navigation of code.

Well thought-out architecture of Lua allowed for good integration with .NET

and usage of Lua as a tool for creating plug-ins for a game editor.

When using Lua, one should consider memory consumption, execution speed

and “excessive” flexibility of Lua code — overlooking these peculiarities you may

hinder all your efforts to improve your system.

254

20 · Using Lua in Game and Tool Creation

function MapLogic:LocationSubscription()

local loc

g_Player:GetActiveTroop():Subscribe("Die", self, "OnActiveTroopDie")

loc = getObj("loc_street_enter" )

if ( loc ) then

loc:Subscribe("GE_OBJECT_ENTERS_LOCATION", self,

"OnUnitEnterLocationStreetEnter" )

end

loc = getObj("loc_yard_enter" )

if ( loc ) then

loc:Subscribe("GE_OBJECT_ENTERS_LOCATION", self,

"OnUnitEnterLocationYardEnter" )

end

end

function MapLogic:OnUnitEnterLocationStreetEnter(Sender, UnitId)

if ( not self:IsOurUnit(UnitId) ) then

return

end

local pos

local index = 1

for _, pos in self.Constants.StreetPos do

local obj = self.ProtEnemy[index]()

index = index + 1

if index > table.getn(self.ProtEnemy) then

index = 1

end

if ( obj ) then

obj:SetPosForced(pos)

obj:SetBelong(1002)

end

end

end

Listing 1.

255

-- PostEvent member

function g_ProcessManager:PostEvent(Recipient, EventName, SenderId, Params)

local CurEventsQueue = self.EventsQueue[ self.EventsQueueNum ]

CurEventsQueue[ self.EventsQueueCurIndex ] =

{ Recipient, EventName, SenderId, Params }

self.EventsQueueCurIndex = self.EventsQueueCurIndex + 1

end

-- ProcessEvent member

-- Calls recipient ( if object) with OnEvent( EventName, SenderId, Params )

-- Calls function by itself with ( EventName, SenderId, Params )

function g_ProcessManager:ProcessEvent( Evn )

local Recipient = Evn[1]

if type( Recipient ) == "function" then

Recipient( Evn[2], Evn[3], Evn[4] )

else

-- Exception handler

local obj = getObj( Recipient )

if not obj then

LOG( "warning: attempt to deliver event "..Evn[2]..

" to non-exist object with ObjId="..tostring(Recipient) )

return

end

if obj.OnEvent then

obj:OnEvent( Evn[2], Evn[3], Evn[4] )

end

end

end

Listing 2.

256

20 · Using Lua in Game and Tool Creation

Prot.Characters = {

Defaults = {

PhysicAnimDeath = true

HP = 500

Material = "FleshDead"

FullInstanceCopy = true

StandAnimationCount = 1

Radius = 0.5

HardRadius = 0.5

RunSoftRadius = 1

StandSoftRadius = 1

TimeBeforeRemove = 10

EngineProperties = {

CastShadow = false

ReceiveShadow = true

DynamicShadow = true

}

}

TestCharacter1= Prot {

Class = Obj

ModelFile = "data/Models/NPC/Women/Pols_1/model.xml"

Material = "FleshAll"

RotationSpeed = 2 * math.pi / 3

HP = 2000

Damage = 70

CaptureTargetRadius = 20

AttackRadius = 2

AttackAngle = math.pi / 4

AttackEnergy = 10

MaxEnergy = 50

RestoreEnergyPerSecond = 7

EmaciationTIme = 3

DamageTimeInAttackAnimation = 0.4

MoveInUpdate = true

EngineProperties = {

DefaultAnimationSpeed = 0.78

CastShadow = true

DefaultPhysicState = "STATIC"

Physic = {

ObjectWeight = 7500

RigidBodyWeight = 3000

DynamicTimeout = 10

NoSleepingCheckTimeout = 10

DynamicSleepingTimeout = 10

}

}

}

}

Listing 3.

257

//Bindings stuff

/**

The types of exports supported by the binding system.

*/

enum eExportType

{

METHOD,

NATIVE_METHOD,

PROPERTY,

//CONST_INT,

//PROP_INT,

//PROP_FLOAT,

//PROP_BOOL,

//PROP_VECTOR,

};

/**

Information about a class entry point.

*/

struct ExportInfo

{

const char* name; // The name of the entry point

eExportType type; // METHOD, NATIVE_METHOD, etc.

void* addr1; // The memory address of the entry point.

void* addr2; // A second address, used by properties

/// these are optional export descriptors

const char* returns; // e.g. "void" or "int"

const char* params;

// e.g. "int a, const CStr& s"

const char* desc;

// e.g. "returns number of chicks avail"

};

Listing 4.

258

20 · Using Lua in Game and Tool Creation

//Bindings stuff (continued)

#define RETURNS_TABLE int retCount = 1; _Lua_pushTable( L,

#define CLASS_METHOD(className, method, returnType) \

static int _export_##className##_##method( lua_State* L ) { \

className* _this = (className*)_Lua_popObject(L, 1, RT_CLASS_LOCAL(className)); \

returnType( _this->method(

#define METHOD_DESCRIBED(className, method, returnsDesc, paramsDesc, shortDesc) \

{ #method, ::engine::METHOD, \

(void*)_export_##className##_##method, 0, returnsDesc, paramsDesc, shortDesc },

#define BEGIN_EXPORT_MAP(className) \

static ::engine::ExportInfo _exports_##className[] = {

#define END_EXPORT_MAP {0}};

//Example Function( in c++ )

Lua::Table Obj::GetAnimations()

{

Lua::Table tbl;

for ( int i = 0; i < this->GetNumAnimations(); i++ )

{

tbl.SetNum(float(i + 1), sArg::FromS(this->GetAnimationName(i)));

}

return tbl;

}

//Begin export stuff

CLASS_METHOD(Obj, GetAnimations, RETURNS_TABLE ) END_METHOD

BEGIN_EXPORT_MAP(Obj)

METHOD_DESCRIBED(Obj, GetAnimations, "", "", "" )

END_EXPORT_MAP

Listing 4. (continued)

21

Leveraging Lua and C++ to

Create a Dynamic and Flexible

Event System for Script-Driven

Games

Robert Oates

Lua boasts several features that make it an attractive option for use in event-

driven applications such as video games.

These programs are made much

easier to program through judicious application of the subscriber pattern, and

can be given a lot of runtime flexibility through the strategy pattern. In this

article I will explain what these patterns are, how they are beneficial to game

development, and finally how we can implement and optimize these patterns

easily with Lua.

The subscriber

The subscriber pattern is absolutely integral to event-driven programming. Ob-

jects will subscribe to events through an event manager, and the event manager notifies the subscribers when the event occurs. There is a lot of room for creative

implementation with this pattern, so let’s step back a moment and consider the

requirements we have for our application and see if that helps narrow down the

possibilities.

260

21 · A Dynamic and Flexible Event System for Script-Driven Games

Since we’re going for an event-driven application here, most of our objects

need to be able to subscribe to and receive events. Characters, UI components,

even entire systems must have a consistent way of interacting with event man-

agers. The goal here is for objects of any type to subscribe to events from multi-

ple event managers and handle them in one place. We can accomplish this goal

by leveraging Lua’s first-class functions. To illustrate the usefulness of such

functions, let’s look at some example code for an imaginary game:

function LeftPlayerKeyHandler(receiver, eventData)

if eventData.key == "A" then

--move receiver left

elseif eventData.key == "D" then

--move receiver right

end

end

function RightPlayerKeyHandler(receiver, eventData)

if eventData.key == "J" then

--move receiver left

elseif eventData.key == "L" then

--move receiver right

end

end

local Player1 = ThisLevel:NewActor("PlayerPrototype")

local Player2 = ThisLevel:NewActor("PlayerPrototype")

Player1:Subscribe("OnKeyPress", LeftPlayerKeyHandler)

Player2:Subscribe("OnKeyPress", RightPlayerKeyHandler)

The first thing we did here was define two slightly different functions to be

used as event callbacks. Second, we created two instances of some predefined

“Player” object. These instances then subscribed to the same event with different

callback functions. Player1’s OnKeyPress callback will cause it to move left and

right when the ‘A’ and ‘D’ keys are pressed. Player2’s OnKeyPress callback uses

the ‘J’ and ‘L’ keys instead. We can see that regardless of what our object,

event, and callback functions are the act of subscribing to an event will be

consistent with this interface. But the use of callbacks in event systems is

certainly not groundbreaking and definitely not Lua-exclusive, so what’s the big

deal? The “big deal” is a combination of things. While callback functions are

not exceptional on their own, when they also happen to be first-class functions

it opens up some interesting possibilities.

Subscribers can define simple callbacks inline:

Player1:Subscribe("OnDeath", function()

error("Player Died!")

end)

261

We can take full advantage of Lua’s closures:

local counter = 0

function DoStuff()

--do stuff

counter = counter + 1

print("DoStuff has been called " .. counter .. " times.")

end

Player1:Subscribe("OnDeath", DoStuff)

--the variable "counter" will be properly incremented when

--DoStuff is called by the event handling system even though

--it’s not defined in the function. Powerful!

We can create data that is used to quickly subscribe an object to events

common of its prototype. This creates a more data-driven approach.

--In the player prototype data somewhere

DefaultPlayerEvents = {

OnCreate = PlayerPrototype.PlayerCreateFunc,

OnKeyPress = PlayerPrototype.LeftPlayerKeyHandler,

OnKilled = PlayerPrototype.DrawFrownyFace,

}

--In the NewActor function somewhere

for event, callback in pairs(PlayerPrototype.DefaultPlayerEvents)

do

somePlayer:Subscribe(event, callback)

end

The other reason why callbacks in Lua are so useful is highlighted in the

“inline callback” example above. Since Lua discards arguments if passed into a

function that does not expect them, we can use callback functions that do not

take a receiver or event data table. Notice also that we have not specified a

return type for our callbacks. In general Lua’s type flexibility sets our event

system apart from similar systems in languages like C++ where all callback

functions would be required to match a specific function signature. Callbacks

in C# for different event types frequently have different function signatures!

In our Lua-based system we can use the same callback to subscribe to several

very different events and, as long as the event data table contains the entries

expected in the callback function, everything will work out great. We can even

create a callback function that will describe the event data of any event we

subscribe to!

function EventDataWriter(receiver, eventData)

for key, val in pairs(eventData) do

print(key .. " - " .. val)

end

end

262

21 · A Dynamic and Flexible Event System for Script-Driven Games

MyGame:Subscribe("OnKeyPress", EventDataWriter)

MyGame:Subscribe("OnWindowResize", EventDataWriter)

MyGame:Subscribe("OnUDPRecv", EventDataWriter)

MyGame:Subscribe("OnKilled", EventDataWriter)

Now that we have a subscription and callback interface that satisfies our

requirement for consistency and flexibility, we must turn our attention towards

posting events and the interface thereof.

To get the most out of our event

manager we will need an equally flexible interface for sending events as we

created for receiving them. Here again we will take advantage of Lua’s flexibility

with respect to data types. Consider the following call to send an event:

PostEvent("OnKilled", someEventDataTable )

All we’ve done here is tell some nebulous event managing system that the

OnKilled event has fired and whatever is in the event data table should be

passed along to any objects subscribed to that event. What’s wonderful about

this system is that there’s no need for predefined events or event data types.

We can post an OnKilled event without any object ever being subscribed to it,

and we can put whatever we want in the event data so long as the subscriber’s

callback function handles (or ignores) it. Most importantly, the call to post an

event looks the same no matter what the event is, who’s getting it, or what data

is associated with the event.

All that’s left is to decide how the internals of our event manager should

work. What happens when we subscribe to an event? What happens when

we post one? In general, an event manager needs to keep a list of subscribed

objects and their callbacks for each event. Such a structure could be visualized

in a manner similar to picture below:

Let’s examine how the subscription and posting of events works in better

detail to provide some insight. Subscribing to an event would add it to the list

of subscribed events above. Once the event name has been found or added, the

subscriber and callback are inserted into the corresponding list as shown below:

function Subscribe(self, eventName, callback)

table.insert( EventManager.Events[eventName],

{ Subscriber = self, Callback = callback } )

end

263

This is event subscription at its simplest. A more robust system would need

to add functionality for unsubscribing from events, as well as error checking to

avoid duplicate registrations and similar cases. We must also consider events

that add/remove other events, iterator invalidation, and all of the headaches

that brings. The simplest way to handle these issues is with “pending queues”.

Rather than inserting new event handlers immediately when the function is

called, it should add them to a separate list that is added en masse at the end

of the frame. Removing events should flag them for deletion at the end of the

frame.

Posting events is also remarkably simple. If the list of subscribed events

contains the event being posted, then we iterate over the associated list and

execute the callback for each subscriber. This example too eschews robustness

for brevity:

function PostEvent(eventName, eventDataTable)

for _, pair in ipairs(EventManager.Events[eventName]) do

pair.Callback(pair.Subscriber, eventDataTable)

end

end

Strategy

I mentioned earlier that one of the great things about Lua is that its functions

are first-class. This language feature immensely facilitates the strategy design

pattern. The strategy pattern creates differences between objects at runtime

by changing their function references. We can leverage the event manager we

created above as a way to implement strategy in our application by specifying

different callbacks for different objects (as in the very first code example), but we

can also override functions directly when it suits us. Consider the code below:

function Fly()

--Do stuff

end

function Walk()

--Do Stuff

end

local enemy1 =

ThisLevel:NewActor("BlobMonster")

local enemy2 =

ThisLevel:NewActor("BlobMonster")

enemy1.MoveFunc = Fly

enemy2.MoveFunc = Walk

Now if we have a list of similar enemies we can simply iterate through it and

call MoveFunc for each one. This is similar to inheritance, with the exception

that you can actually alter the behavior during runtime.

264

21 · A Dynamic and Flexible Event System for Script-Driven Games

function Icarus(receiver, eventDataTable)

--If we’re flying right now then this is bad news. :-(

receiver.MoveFunc = Walk

end

enemy1:Subscribe("OnTooCloseToSun", Icarus)

Now when our flying enemy gets too close to the sun, his movement function

is changed to make him a walking enemy instead.

Extra credit

Serialization of this event data is very straightforward, and even allows us to

leverage our existing event system for sending network messages. Consider the

following example:

local MyChatWindow = GUI:NewWindow("ChatWindow")

MyChatWindow:Subscribe("OnNetChatMessage", NetChatMsgFunc)

Where will the “OnNetChatMessage” event be generated? From the other com-

puter we’re connected to, of course! What we need to do to make this work is cre-

ate a method for remote systems to tell our event manager to broadcast the On-

NetChatMessage event — with proper data and all. This gets a little involved,

but it’s definitely worth it. This section was written with winsock and C++ in

mind, but the theory should apply to any network API you can use through Lua.

The remote system wants to send an event to our system. The data will need

to be serialized into a stream of bytes by the remote system, sent across the

network, and then deserialized into a meaningful event on our system. How

should we approach this given that we do not predefine event templates? With

a little trickery, of course! We will give events a predictable header that tells the

receiver how to decode them on the fly. So what sort of information do we need

to send so the receiver can decode our byte string?

• We should send the length of the serialized data as the first integer.

• We will need to store the name string of the event.

– All strings will require an integer prefix to describe their length.

• We need to define the event table data.

– We should consider adding a “sender” member to the table with our

IP address or a connection ID, so the receiver knows who sent this

message.

– For all tables we need to specify the table size, along with each ele-

ment’s type.

– We need to differentiate between (string) keyed tables and array- type

tables.

265

· For keyed tables, we will need to store each key string as well as

its value.

Taking all of this into consideration, a call and viable serialization might look

like this:

SendNetEvent("PlayerData",

{

Name = "Robert Oates",

Age = 23,

Single = false,

Color = {255, 0, 0},

})

• Line 1 (header):

– byte unsigned integer for the buffer length in bytes

– 1 byte for size of event name string

– 10 bytes for event name string (no null terminating character)

– 1 byte for number of parameters in the event table

– 4 bits to describe type of each value in event table (×2 = 2 bytes in

this case)

• Line 2 (first event table entry):

– 1 byte for key string length

– 4 bytes for key string (Name)

– 1 byte for value string length

– 12 bytes for value string (Robert Oates)

• Line 3 (second event table entry):

– 1 byte for key string length

– 3 bytes for key string (Age)

– 4 bytes for float number (23)

266

21 · A Dynamic and Flexible Event System for Script-Driven Games

• Line 4 (third table entry):

– 1 byte for key string length

– 6 bytes for key string (Single)

– 1 byte for Boolean (false)

• Line 5 (fourth and last event table entry)

– 1 byte for key string length

– 5 bytes for key string (Color)

– 1 byte for number of elements in array

– 4 bits to describe the type of each element in the array (3 numbers × 4

= 12, which rounds up to 2 bytes in this case)

• Line 6 (members of Color array)

– 4 bytes for number (255)

– 4 bytes for number (255)

– 4 bytes for number (255)

Now that our data has been serialized, it is ready to be streamed across the

internet to the waiting event handler on some other computer. The benefit to

self-describing data such as this is that the receiver does not need to know about

an event in order to receive and decode it.

As you can see, this setup allows for the nesting of tables and all sorts of other

neat stuff. The drawback it suffers is bloat from all of the strings used to describe

the various bits of data. Fortunately if the format of our event data is not going

to change much, we can take advantage of caching. Once this message has

been successfully sent, both sides can remember “Ok. There’s an event called

PlayerData, and its event structure looks a certain way.” The event information

will then be cached and assigned a unique number for future use. Next time the

event gets sent across the network, it only needs to supply the values and none

of the description data (keys, type info). The example I’m about to show makes

the assumption that types will remain consistent even inside of nested tables.

Your implementation may vary. In the event that I needed to send my player

data again the serialized version would now look like this:

Notice that the buffer length in picture below is now a negative number,

which I’ve decided to use as a flag to indicate that this is a cached message

type. The following byte (you may wish to use a short or uint instead) with

value 1 is the unique cached message type number. With the type information

cached, we’re able to send only the value data and reduce subsequent messages

from 72 bytes to 35 bytes — cutting the message length by more than half while

preserving the ability to send events over the network as easily as we send them

to objects in our own game. When the PlayerData message is received on the

other side, it will be reconstructed and then sent to an event handler which will

then forward it to any objects (such as the game object, a level, a user interface,

or an enemy) that have subscribed to the message.

267

Closing

Hopefully the simple examples I have provided are adequate to illustrate the

power and flexibility of these patterns. When combined with Lua’s advanced

features (closures, first-class functions) they can be used to build a solid founda-

tion for any script-driven game.

22

Lua for Game Programming

Steve Gargolinski

The goal of this article is to describe a breadth of ways that Lua can be used to

supplement a traditional (C++) game engine. The three areas that we will take

a look at specifically are data representation, adding an extensible structure

for providing dynamic in-game challenges to the player, and supplementing our

game world with Lua-driven artificial intelligence.

When video games were young, all pieces of a typical game were coded

directly into the engine. Maps, sprites, the user interface, game logic, and AI

were all represented in assembly code or C, and later C++. This approach was

cumbersome and inflexible, requiring a build step in between changing any sort

of game data and being able to view it in the game. Before long, the concept

of separating the game engine from its external components became possible.

Maps are now created by editors as external files, Non-Player Character (NPC)

dialogue is stored in a text file rather than C++ code, and sprites are stored as

textures. It is no longer necessary to perform a costly build step after changing

one line of NPC dialogue.

After moving game assets out of the engine and into data, the next step is

to create a separation between our engine code and game logic, which can be

achieved by exposing select areas of the game engine to a scripting language

such as Lua. This separation allows programmers to define clean interfaces

between select areas of the game and the chosen scripting solution. Also, less

technically proficient designers can work at a higher level and tinker with game

systems without needing to write actual C++ code, compile, or fully understand

the engine. Much of the game experience can be tweaked and tuned without

requiring the game to be rebuilt or even restarted.

270

22 · Lua for Game Programming

Keep in mind that games are performance-intensive applications. It is neces-

sary to make careful choices about which layers are implemented in script and

which layers stay in the (faster) C++ game engine. It’s up to the game program-

mers to decide just which areas of the engine will provide maximum advantage

when exposed to a scripting solution.

In this article we will discuss several potential high-leverage areas ripe for

Lua exposure. Lua is a particularly strong choice for a scripting solution in

any game engine. It is fast and lightweight. It is open source — in case you

decide it needs to be extended — and also completely free. It has a reasonably

simple syntax while remaining powerful. This will come in handy considering

that designers who do not write code every day for a living will often be the ones

to use Lua scripts.

Example game

To clearly illustrate the goals of this article, we will refer to a sample game.

Hopefully you all remember the classic game Adventure (or Zork). The frame-

work game we are going to use is a very simple version of these “text adven-

tures”.

In this sample game there is a World made up of Locations. Locations are

connected to each other in a sparse graph. There are Items in this World which

can either be at a Location or in the possession of an Actor who can be either

the Player or an NPC. Actors are able to pick up items, drop them, and move

between connected Locations in the World. It is as simple as that.

Note that this article assumes an existing C++/Lua binding solution.

A

description of the different available techniques is beyond the scope of this

article, but Celes et al.1 provide a solid discussion.

Data representation

Games use huge amounts of data. In a typical game there are models, ani-

mations, maps, entities, and sounds, each with its own data format. Figuring

out the best way to represent this data is an involved decision with many im-

plication details including: platform-specific issues, memory limitations, and

designer/artist workflow patterns.

Lua can be used to efficiently and flexibly handle loading designer-defined

data. Assume that in the realm of our example game, designers are in charge

of creating each Location in the World. The code in Listing 1 describes a way to

load in two Locations by defining them as a Lua table.

LoadAllLocations is responsible for building up a locationTable and pass-

ing it along to LoadLocationsFromTable where the information is extracted and

used to add Locations to the World through a minimal number of exposed C++

1W. Celes, L. H. de Figueiredo, and R. Ierusalimschy, “Binding C/C++ Objects to Lua”, Game Programming Gems 6, Charles River Media: pp. 351–355, 2006.

271

function LoadLocationsFromTable(locationTable)

local world = LPGWorld.GetInstance()

for i = 0, table.getn(locationTable) do

world:AddLocation(locationTable[i].name, locationTable[i].desc)

end

end

function LoadAllLocations()

local locationTable = {}

local index = 0

locationTable[index] = {}

locationTable[index].name = "FOREST"

locationTable[index].description = "A lush forest."

index = index + 1

locationTable[index] = {}

locationTable[index].name = "SWAMP"

locationTable[index].description = "A dark swamp."

LoadLocationsFromTable(locationTable)

end

Listing 1.

functions. Allowing game data to be defined this way is very flexible; it’s easy

to cut and paste, add, and delete Locations with just a few clicks. In the real

world of game development, however, game data is far more complicated than

simple Locations with names and descriptions. Specifying our data explicitly

in a Lua table is not optimal. It is important for designers to have the power

to organize data in a cleaner way — in spreadsheet form, for example. No mat-

ter which external format we decide to use, the result will be a locationTable

passed to LoadLocationsFromTable. Whenever designers want to use an exter-

nal data format (.xml, .csv, etc.) we need to write a bit of code to turn this data

into a properly formatted locationTable. Lua has a decent set of string ma-

nipulation utilities which makes writing these functions easy. If we want to

represent our data above in a spreadsheet, we will end up needing to parse a

.csv (comma separated value) file, as follows:

FOREST,A lush forest.

SWAMP,A dark swamp.

Extracting this data into a locationTable equivalent to the one defined as

an explicit Lua table above can be done with a simple bit of code (Listing 2).

We can now update our loading process to read in a .csv file:

LoadLocationsFromTable(BuildLocationTableFromCSV("map.csv"))

Loading data in this fashion allows us to store the actual data in any format we

want, while still passing through a common function (LoadLocationsFromTable)

during the loading process. This allows load-time designer defined validation of

272

22 · Lua for Game Programming

function BuildLocationTableFromCSV(csvFile)

local locationTable = {}

local index = 0

for csvEntry in io.lines(csvFile) do

local _, _, locationName, locationDesc =

string.find(csvEntry, "(.+),(.+)")

locationTable[index] = {name = locationName, desc = locationDesc}

index = index + 1

end

return locationTable

end

Listing 2.

data on top of the constraints implemented in our game engine. We can expose

this functionality by adding a single line:

function LoadLocationsFromTable(locationTable)

local world = LPGWorld.GetInstance()

for i = 0, table.getn(locationTable) do

ValidateLocationData(locationTable[i]) -- ***

world:AddLocation(locationTable[i].name, locationTable[i].desc)

end

end

ValidateLocationData becomes an opportunity for designers to define vali-

dation requirements for the data they are specifying. A designer could decide

that Locations should always specify a description that is not an empty string.

It would be simple to add this check without requiring any change to the game

code. This is most useful for designer-desired guidelines, with more strict re-

quirements enforced in the engine:

function ValidateLocationData(loc)

if string.len(loc.desc) == 0 then

print("DATA ASSERT - Empty description for location: " .. loc.name)

end

end

With this structure in place, it is simple to expose the ability to add Locations

at run time.

All that we need to do is add a Lua function to build up a

locationTable to pass through the same loading procedure used above. Hooking

a game debug console into this function allows designers to modify the game’s

data easily at runtime, giving them the ability to test out new Locations without

even needing to restart the game.

function AddSingleLocation(locationName, locationDesc)

local locationTable = {}

locationTable[1].name = locationName

273

locationTable[1].description = locationDesc

LoadLocationsFromTable(locationTable)

end

Dynamic challenges

The goal of this section is to add a mechanism to present the player with chal-

lenges — focused, mix-in situations with risk/reward structures to drive and con-

trol the overall flow of the game. We will be creating a simple example challenge

called “Water The Forest”, in which the player must bring the “Water Jug” Item

to the “Forest” Location in the World. This is a very simple challenge based on

our example game, but the mechanism is powerful and can be applied to many

different types of games. The idea here is that high-level control of the chal-

lenges (initialization, updating, etc.) is handled in the game engine, but the

content is completely controlled through Lua scripts. Each challenge is defined

in terms of a single Lua file. For the purposes of this article we will keep things

very simple — Lua file only needs to implement four functions: EvaluatePre-

Reqs(), Update(), Success(), and Failure(). EvaluatePreReqs() is responsible for

controlling when a particular challenge is given out. This function returns a

boolean value indicating true whenever the pre-requisites for this challenge are

met, and false otherwise. We will use the results of EvaluatePreReqs() when

deciding which challenge to present to the player. Here is some simple example

(engine level) pseudocode for using EvaluatePreReqs() to choose a valid chal-

lenge based on the current game state:

Array<Challenges> validChallenges;

for i = 1; i < allChallenges.size(); ++i

{

if (allChallenges[i]->TriggerEvaluatePreReqs() == true)

validChallenges.push(allChallenges[i]);

}

activeChallenge = validChallenges[RandInt(0, validChallenges.size)];

This piece of code will loop through all of our challenges, building up an

array of the ones which pass our EvaluatePreReqs() test. We then set our active

challenge to a random entry in this array. Something important to note here

is how the allChallenges array gets filled in. Since each of our challenges is

contained within a Lua file, we can simply iterate on all the .lua files in a

specified challenges directory, adding each one to the allChallenges array. This

discovery mechanism is a very useful property since it does not require a list

of challenges to be stored anywhere. Adding a new challenge only requires the

addition of a new file. If we decide to release an expansion pack, downloadable

content, or some combination of the two, there is no need to coordinate an index

file of challenges between these permutations. Each expansion pack simply

needs to drop a few files in the challenges directory. After the engine has chosen

an active challenge, the next responsibility of the engine is to trigger an update

274

22 · Lua for Game Programming

on this challenge. The result of an update can either be success (1), failure (–1),

or no resolution (0).

int challengeResult = activeChallenge->triggerUpdate();

if (challengeResult == 1)

{

activeChallenge->TriggerSuccess();

completedChallenges.push(activeChallenge);

activeChallenge = NULL;

}

else if (challengeResult == -1)

{

activeChallenge->TriggerFailure();

failedChallanges.push(activeChallenge);

activeChallenge = NULL;

}

These lists can be used to present the player with a history of the challenges

they have attempted, or to filter allChallenges to prevent giving the player

a challenge they have already completed.

For our example challenge these

functions are simple. The goal of “Water the Forest” is for the player to bring

the Water Jug into the Forest. We do not want to give out this challenge unless

three preconditions are met:

• The Water Jug Item exists somewhere in the World.

• The Forest is a Location in the World.

• The Water Jug is not already in the Forest.

The EvaluatePreReqs() implementation is very simple. As long as these

three conditions are satisfied, we want to give out the challenge. Here is some

example code to handle evaluating the prerequisites (assume that the world and

itemManger are passed into EvaluatePreReqs() by default):

function EvaluatePreReqs(world, itemManager)

local forest = world:GetLocationFromIDString("FOREST")

local waterJug = itemManager:GetItemFromIDString("WATERJUG")

if (forest == nil) or (waterJug == nil) or (forest:HasItem(waterJug))

then

return false

end

return true

end

In Update() we only need to perform one check: Does the Forest currently

contains a Water Jug? When it does, we provide feedback for the player indicat-

ing that the challenge has been completed. In a full game we would also give

out some gold or experience points. Here is an example Update() function for

our challenge:

275

function Update(world, itemManager)

local forestLoc = world:GetLocationFromIDString("FOREST")

local waterJugItem = itemManager:GetItemFromIDString("WATERJUG")

if forestLoc:HasItem(waterJugItem) then

return 1

else

print("Still waiting for that Water Jug.")

end

return 0

end

When the player drops the Water Jug in the Forest, our Update() function

will return 1, causing this challenge’s Success() function to trigger. In this simple

case we will print some feedback for the user, but in a more complicated example

it could display some fancy UI, update the hero’s experience points, or give out

some gold pieces. In this simple example there is no failure case.

function Success()

print("Successfully brought the Water Jug to the Forest!")

end

The above example displays a very useful pattern. We’ve created a fully

data-driven challenge system. Each challenge is contained completely within a

single Lua file. Aside from the C++ function exposure, the only responsibility of

the game engine is to decide which challenge to run, and to know which script

functions to call when updating. The rest of the logic is contained completely

within the Lua code.

AI and state machines

AI programmers have long used Finite State Machines (FSMs) in all (ok, most)

of your favorite games to successfully create the appearance of enemy/opponent

intelligence. There are a number of ways to implement an FSM structure with

Lua. It can be done completely in Lua code (and for certain situations this

might make sense), but for this example we’re going to stick with the pattern of

supplementing a traditional FSM structure with Lua instead of moving over the

entire implementation. This gives us the power to set breakpoints and monitor

variables in C++ while retaining the flexibility of Lua.

It can make a lot of sense to expose your FSM functionality to a script-

ing solution such as Lua. This choice carries with it a lot of familiar bene-

fits/drawbacks of exposing a game system to scripts. A data-driven AI solution

can provide some serious power. With script-controlled game agents, it is quick

and easy to make widespread changes. New AI logic can be quickly tested and

iterated on. After exposing the necessary sections of the AI system to Lua, it is

possible to hand off the state machine implementation to designers — allowing

them complete control over the characters in game with no C++ knowledge re-

quired. However, with this amount of power comes the natural ability to screw

276

22 · Lua for Game Programming

things up big time. FSMs can become complicated, assuming control over a

huge number of rich character behaviors. Designers (or programmers) unfamil-

iar with the subtleties of FSM design may have trouble keeping full view of the

‘big picture’ when modifying the AI. Changes in one state may have undesired

side-effects on other states in the machine. These state to state relationships

are not always obvious, and exposing them to more people (instead of just AI

programmers) is a risk increase. Performance is another important factor to

keep in mind when implementing a FSM solution. State logic implemented in

Lua will never be as fast as state logic implemented in C++. Make sure that the

majority of expensive calculations are taking place on the C++ side and develop

metrics to keep an eye on where processor cycles are being used up in your game

AI. As your FSMs near completion, consider moving the more expensive states

into C++ code, trading flexibility for performance.

Here is the basic skeleton of our FSM:

class FSMMachine

{

void UpdateMachine();

void ChangeState(FSMState* newState);

void AddState(FSMState* newState);

FSMState* m_currentState;

}

FSMMachine is responsible for aggregating the states in our FSM. It keeps

track of the active state, provides a mechanism for switching states, and is the

entry point for updating our FSM.

class FSMState

{

void Begin();

void Update();

void End();

}

Each FSMState requires three functions: Begin(), End(), and Update(). Be-

gin() is called whenever the state machine transitions to this state, End() is

called whenever the state machine transitions away from this state, and Up-

date() is called each tick on the active state. This functionality is captured in

the engine as FSMMachine::ChangeState(), which looks like this:

void FSMMachine::ChangeState(FSMState* newState)

{

m_currentState->End();

m_currentState = newState;

m_currentState->Begin();

}

The base class FSMState can be extended to implement a specific state in

C++ code. For example, we could add a new state FSMStateHunt, which extends

277

FSMState and implements the logic necessary to send an NPC out on a hunt. In

order to facilitate a Lua-driven FSM, we’re going to provide the LuaFSMState

class:

class LuaFSMState : public FSMState

{

LuaFunction m_beginFunc;

LuaFunction m_updateFunc;

LuaFunction m_endFunc;

void Begin();

void Update();

void End();

}

This class replaces arbitrary C++ logic in Begin(), Update(), and End() with

Lua function calls in this fashion:

void LuaFSMState::Begin()

{

CallLuaFunction(m_beginFunc);

}

Our FSMMachine is now free to mix and match Lua driven states with C++

driven states.

Now that we’ve got the basic transition structure of state machine logic set

up, how do we actually use it? First we need to define the state machine of

an NPC. We’ll expose a LuaFSMState factory function, and provide a startup

method for each NPC to seed their state machine.

function SetupFSM(fsm)

local idleState = LuaFSMState.Create("Idle", "npc0.lua",

"Idle_Begin", "npc0.lua", "Idle_Update", "npc0.lua", "Idle_End") local wanderState = LuaFSMState.Create("Wander", "npc0.lua",

"Wander_Begin", "npc0.lua", "Wander_Update", "npc0.lua", "Wander_End") fsm:AddStateToFSM(idleState)

fsm:AddStateToFSM(wanderState)

fsm:ChangeState("Idle")

end

This block of code starts off by creating “Idle” and “Wander” states, specifying

which Lua functions to call for the Begin(), Update(), and End() of each. It then

adds these states to our NPC’s FSM, and finally starts him off in the “Idle”

state. Note that it is possible to follow the pattern we defined in the “Data

Representation” section to move this data into a .csv file.

For this example, we only need to define the Update() functions for our two

FSM states (Assume that the FSM, the player, and our NPC are passed into

these functions when called from the engine):

278

22 · Lua for Game Programming

function Idle_Update(fsm, player, self)

if player:GetLocation() == self:GetLocation() then

fsm:ChangeState("Wander")

end

end

function Wander_Update(fsm, player, self)

if rand() < 0.5 then

fsm:ChangeState("Idle")

end

self:SetLocation(GetRandomConnectedLocation())

end

This code will cause the NPC to hang out in the Idle state until the player

comes onto the same location as him, which triggers a transition into Wander.

This NPC will spend the next few updates wandering randomly through the

world before returning to Idle.

We’re not doing anything in the Begin() or End() functions in a simple exam-

ple like this, but we can show some useful trace code here:

function Idle_Begin(fsm, player, actor, world)

print(actor:GetName()..": Let’s hang out here for a while.")

end

function Idle_End(fsm, player, actor, world)

print(actor:GetName()..": Time to move!")

end

This is a simple example that shows the basic structure behind a flexible and

powerful FSM. We’ve given developers the option to move any or all AI decision-

making code into Lua, separating it completely from engine code. We retain

the option to implement each state in either Lua or C++ as we see fit. We can

define and iterate on many states quickly in Lua, while preserving the ability

to implement states in C++ when speed is a priority or the potential exists for

significant debugging.

Generic Lua function exposure

A powerful and simple use of Lua in a game engine is to expose arbitrary Lua

script execution at run time via your game’s command console or debug tools.

All that you need to do is give the development team access to something as

basic as: runluascript file.lua function.

This simple addition will be beneficial for everyone on the development team

from programmers to artists to designers to QA. The ability to quickly, easily,

and repeatedly run a set of actions contained in a script has advantages on many

different levels. Designers can force situations they want to mess around with

279

in game to see how they play out, QA can attach a Lua script to a bug report

to provide more accurate reproduction, and programmers can use the script for

anything from general code testing to performance evaluation.

Making generic Lua function execution available has the added benefit of

increasing the development team’s exposure to your chosen scripting solution.

The more comfortable your team gets with Lua, the more use you will get out of

it.

Conclusion

Hopefully this article has given you some ideas on how to supplement your game

engine by exposing key areas to a Lua scripting solution.

The techniques discussed here barely scratch the surface of potential uses of

Lua for game programming. Before you delve in, analyze the problems that your

particular game is trying to solve and identify areas which lend themselves well

to a data-driven, scripted solution.

23

Designing an Efficient Lua

Driven Game Scripting Engine

Nicolas Peri

When designing a modern game engine, able to handle hundreds of “intelligent”

objects with various complex behaviors, you need to deal with two major issues:

runtime speed and ease of development. Using traditional C/C++ programming

to define your objects behaviors implies recompiling your game code each time

you make a modification in order to test it, which is a big time wasting. On

the other hand, using a script driven game code will offer you a huge gain

of development time, allowing you to perform “in game programming”, but

a bad integration can quickly result in poor runtime performance. The ease

of use and the execution speed of Lua are not to prove anymore. However,

a well-done integration is not an easy thing to do, if you do not understand

exactly how Lua works and how to use it efficiently. This article explains how

to take advantage of Lua to design a powerful and flexible game engine with

autonomous object behaviors, using scripted event handlers, and how to avoid

the common performance pitfalls.

The problem — motivation and statement

We can consider a game as a collection of scenes, also known as levels. Each

scene represents an area in the universe, containing objects like flowers, trees,

birds, and so on. Many objects do not need to have an artificial life, like flowers

or dead trees, but some must have particular dynamic behavior, like birds flying

282

23 · Designing an Efficient Lua Driven Game Scripting Engine

from tree to tree or evil monsters trying to attack you. When designing a game

engine, you must provide your future users an easy and flexible way to define

those behaviors. Nowadays, scripted artificial intelligence is the best solution to

do that. The problem is that scripts will always be slower than native code; that’s

a fact. When making a game engine, performance is one of the most important

constraints. We thus must find a way to reduce the CPU cost of scripts execution,

to stay under an acceptable threshold.

The solution — description

Use an object local solution

Every “intelligent” object in the scene must have its own behavior. We will call

an AIModel the data structure defining an object’s behavior: it can be compared

to a C++ or Java class, with the main difference that all the code is written

in Lua. An AIModel basically contains member variables and functions. Each

instance of an AIModel has his own set of member variables, with or without

initial values, allowing every instance to be independent and quite different

from others. Each object will have one or more AIModel instances controlling it.

For example, an object representing a soldier could have an AIModel dedicated

to the path finding and another one to the attack.

Use frame-by-frame execution

Ideally, autonomous behaviors should be done by using one thread for each

AIModel instance. For performance reasons, this is just impossible in a game

engine. The execution of each AIModel instance must thus be designed frame by

frame, with the aim to emulate multithreading: each AIModel must for example

implement a runOneFrame function, written in Lua, that will be in charge to

step for each frame the desired behavior. In pseudo code, the engine loop would

basically be:

while ( gameIsRunning )

do

for each Object o in the Scene

do

for each AIModel instance ai controlling o

do

ai.runOneFrame ( )

done

done

scene.draw ( )

done

Member variables are used to maintain the state of an AIModel instance,

frame to frame. Nothing remains at the end of the frame but those.

283

while ( gameIsRunning )

do

for each Object o in the Scene

do

for each AIModel instance ai controlling o

do

if ( ai.isTimeToRunOneFrame ( ) )

then

ai.runOneFrame ( )

end

done

done

scene.draw ( )

done

Listing 1.

Allow different update rate for the AIModel instances

Some behaviors do not need to be updated every frame: consider for example a

radar that has to check at a regular time interval if there are ships around. It

would thus be useful to provide to the user a way to specify that an AIModel only

needs, for example, to be updated every second. This will avoid Lua to execute a

script that spends, in average, most of its time in a waiting code. Because of the

object local and frame-by-frame design, it will be possible to pause the execution

of any AIModel instance when it is required. Our engine loop, in pseudo code,

would now look like Listing 1.

Use an automatic activation process

Big scenes can contain thousands objects with one or more behaviors each. To

limit the number of simultaneous running AIModel instances, we have to use an

automatic mechanism dedicated to identify if, for a given frame, an object really

needs to be updated. A fairly simple but good criteria would be to consider the

distance to the camera: far objects, which are not even visible, certainly do not

need their behavior to be updated. By limiting only nearest objects to be active,

we will only execute Lua calls for a little subset of the objects in the scene. Our

engine loop, in pseudo code, would then look like Listing 2.

Use an event based communication schema between AIModel

instances

Now that we have autonomous behaviors for our scene objects, we would like

them to be able to communicate with each other. We will add to AIModels some

sort of public functions called “event handlers”, and give to the user a way to

284

23 · Designing an Efficient Lua Driven Game Scripting Engine

while ( gameIsRunning )

do

for each Object o in the Scene

do

if ( o.isActive ( ) )

then

for each AIModel instance ai controlling o

do

if ( ai.isTimeToRunOneFrame ( ) )

then

ai.runOneFrame ( )

end

done

end

done

scene.draw ( )

done

Listing 2.

send “events” from an AIModel instance to another one. A solution will be to

provide a function similar to the following one:

object.sendEvent ( hObject, "Pathfinder", "Goto", x, y, z )

This function takes at least three parameters: an identifier of the object, the

name of the AIModel instance and the name of the event we want to send. All op-

tional remaining parameters will be passed to the event handler onGoto, of the

instance of the AIModel Pathfinder, that is controlling the object represented by

hObject. By this way, objects in the scene will have their own local behavior and

will be able to send or receive stimuli from others, thus taking some decisions

or just changing their state. In addition, this schema guarantees that no objects

will be strongly dependent of another one: if an object is not active, it will simply

not react to incoming events.

Use high level function packages

Written in C/C++ and accessible from the Lua code, it is recommended to provide

high level dedicated packages of functions, avoiding long calculations to be done

in script. As an example consider a simple object behavior: the object must

always look at the camera. This requires tens of multiplications and additions. If

there are tens of active objects running, this will result in hundreds of arithmetic

operations per frame to be done in Lua. We thus must provide a function to

do this in one Lua call, all the calculations being done in C/C++ compiled and

optimized code. In our case this could be something like:

object.lookAt ( hObject, x, y, z )

285

where hObject would be an identifier of the object we want to control, and x, y,

and z the coordinates of the point to look at. In a game engine it would be a good

idea to provide, in addition of common packages like math or system, a ded-

icated function package for each domain of application: animation, dynamics,

navigation, sound, sfx, . . .

Use handles to exchange data structures between Lua and C/C++

runtime

Lua provides an easy way to access your C/C++ data structures through the

lightuserdata type. This is nothing but a pointer, that cannot be used directly

from script, but that should be used like that:

-- Always look at the origin of the scene.

function MyAIModel:runOneFrame ( )

local hObject = self.getObject ( )

if ( hObject ~= nil )

then

object.lookAt ( hObject, 0, 0, 0 )

end

end

In this example that defines a simple behavior, the self keyword is the

Lua sugar that represents the current AIModel instance that is executed: its

members are set up by the C/C++ just before calling the Lua function. It allows

to get a handle to the object controlled by this AIModel instance. Once we get

this handle, it will be possible to use it in every high level functions that take

an object handle in parameter. A more complex behavior using could be for

example:

-- If the user pressed the space key, then jump.

function MyAIModel:runOneFrame ( )

if ( input.isKeyPressed ( input.kKeySpace ) )

then

local hObject = self.getObject ( )

if ( hObject ~= nil )

then

dynamics.addImpulse ( hObject, 0, 100, 0 )

end

end

end

In this example, input and dynamics are Lua function packages, dedicated

to input devices management (keyboard, mouse, gamepad...), and dynamics

subsystem access.

286

23 · Designing an Efficient Lua Driven Game Scripting Engine

Limit the use of complex types

Strings, tables, or metatables used for temporary variables are not recommended,

because they will fire the memory allocator. Prefer using simple types like

booleans, numbers, or lightuserdatas. For example using a vector class imple-

mented in Lua with metatables will surely be very elegant, but not encouraged

if you are looking for the best performances. In general all syntax sugars are

not good for speed, or must be preprocessed. So for example to perform a dot

product between two vectors, prefer using this solution:

local x1, y1, z1 = 1.0, 0.0, 0.0

local x2, y2, z2 = 0.0, 1.0, 0.0

local dot

= math.dotProduct ( x1, y1, z1, x2, y2, z2 )

Than this more elegant but slower one:

local v1

= Vector3:new ( 1.0, 0.0, 0.0 )

local v2

= Vector3:new ( 0.0, 1.0, 0.0 )

local dot = v1:dotProduct ( v2 )

Separate values of the first version will be stored on top of the Lua stack,

but tables used for the syntax sugar of the second one, will require at least two

memory allocations. Multiplied by the number of such operations needed to

represent your behavior, and then by the number of active AIModel instances at

a time, those two versions of the same code will have a small but sometimes not

negligible difference, in term of execution speed.

Encourage the use of local variables

When using multiple times an AIModel instance member variable, you can

obtain a significant performance gain by storing it in a local variable. Lua local

variables reside in registers, and are accessible by index, as opposed to global

variables, that reside in a table, and are accessible by a hash lookup. Here is an

example of a local variable usage:

function MyAIModel:runOneFrame ( )

local hObject = self.getObject ( ) -- Store the global into a local

if ( hObject ~= nil )

then

dynamics.enableGravity

( hObject, true )

dynamics.setLinearDamping ( hObject, 0.5 )

dynamics.addImpulse

( hObject, 0, 100, 0 )

end

end

Use preallocated memory pools

Lua allows you to define your own memory allocation functions. If you do that,

you will see that Lua allocates a lot of small temporary buffers that will be freed

287

at the next garbage collection. All of this will fragment your system memory

and will also be time consuming. In an embedded environment or a gaming

console, with a limited and fixed amount of memory, it is vital to avoid as much

as possible dynamic allocations at runtime. A solution is to use a preallocated

pool of small buffers (16, 32, and 96 bytes are good values, but it should depend

on your implementation) that will be dedicated to the Lua runtime. Bigger or

unpredictable allocations are done through your main allocator.

Explanation and justification

Taken one by one, each of the recommendations done just before can look neg-

ligible, but if all are respected, the speed improvements will make it possible

to use Lua as a runtime language for fast applications such as games. Indeed,

we passed from a naive solution to an optimized one by reducing Lua runtime

intrinsics, scripts execution, C/C++ communication and the number of calls.

Weaknesses and suggested improvements

Even with all of that, the same work done with C/C++ compiled behaviors will

always be faster. A solution would be to allow Lua code to be translated to

C/C+ code then compiled, at product releasing time, scripting being used just for

development time. If you follow the recommendations written in this gem, and

strictly use nothing but Lua simple types and loops, this should be a quite easy

thing.

Conclusion

Making a well done integration of Lua in your game engine finally implies

two major things: finding a way to execute only the scripts that are really

needed (but every engine should already do that for all other sub systems like

animation or dynamics, so doing it for AI should not be a big deal), and reducing

the overheads of Lua execution by avoiding time consuming operations and

providing high level dedicated function packages. Lua scripted behaviors would

also be useful for remote programming and debugging: it is possible to change

the behavior of one object in a game running on a game console, by simply

uploading a string containing the new script, which will automatically be taken

in account at the next frame.

Part V

Embedding and Extending

24

Enhanced Coroutines in Lua

Patrick Rapin

This article describes a method to increase the power of coroutines by using

preemptive native threads together with standard collaborative Lua threads.

This way coroutines are able to perform blocking operations, like reading data

from a opened pipe, without freezing all the other coroutines. This is done

by having a dynamic pool of preemptive threads executing commands on the

background, while the associated coroutine is suspended from the Lua point of

view.

History

Our company, Olivetti, is active in the ink-jet printer industry. We have always

written specific programs in order to drive our printers for testing and produc-

tion. The most recent one is a very powerful tool, called LuaDura, based on

Lua 5.1. This program can send low level commands to the printer, through a

dedicated communication channel. The protocol used here is similar to the Mass

Storage Class of USB: the host sends a binary command, optionally followed by

data; the printer always sends back a status, along with additional response

data. Several communication channels can be used for this task: USB, serial

port, Ethernet. Other ones may be added in the future, like Bluetooth or WiFi.

We wanted to be able to drive several printers simultaneously within a single

Lua script, for example in a mass production test board. The problem is that

all input/output commands through the preceding channels are blocking: it is

therefore difficult to drive several printers in a single thread. This is why we

need some form of true multithreading for our application.

292

24 · Enhanced Coroutines in Lua

Multithreading models

In the book “Programming in Lua” (second edition), Roberto Ierusalimschy

explains in chapter 30 that Lua supports two models for multithreading. The

third model is the one we will describe.

Lua collaborative threads with memory sharing

In the first model, there is only one Lua state, but several coroutines running

sequentially inside it, by explicit yielding and resuming. As they run in the

same state, all these threads can access the global variables and the registry,

so it is easy to share data between them.

However, the operating system

has no knowledge about this mechanism. If one collaborative thread performs

a blocking I/O operation, all other coroutines are stopped during that time.

Consequently, this model does not fit our requirement. An option could be to use

asynchronous input/output, as they are available on most operating systems.

Unfortunately, this has several drawbacks: it is hard to implement, not portable

across platforms, and not all operations can be done that way.

Preemptive threads in separate Lua states, sharing no memory

In the second model, several Lua states are created, each running in a different

native preemptive thread. Because there is no connection between separate Lua

threads, C code must be added in order to share data between the interpreters.

This could be a solution for our application: we could imagine a C function,

taking as parameter a function to execute, that would create a preemptive

thread, which in turn would create a new Lua state and run the function

until it exits.

A blocking command only stops that specific thread.

While

possible, we don’t like this option, as it would be hard for the main script to

monitor the simultaneous processes without heavy intervention of C code and

OS semaphores.

Preemptive threads associated with coroutines

The third model is essentially an enhanced version of the first one. There is

only one Lua state, and several coroutines running inside it. Therefore the

threads can easily share data between them.

The key trick is to enable a

native preemptive thread to executes the input/output commands instead a Lua

coroutine. It works like this: the coroutine calls a C function when it has to

perform a blocking operation. That function checks if there is an idle native

thread from the pool, or instantiates a new one. It then sends a message to it,

and yields. Lua interpreter will switch to another coroutine, while the command

really starts executing in the background. To prevent memory corruption, the

native thread has no access to the Lua state: all its input arguments, output

results and error messages must be passed through the message structure.

293

Later on, the coroutine will be resumed, and if the command has finished

executing, output result is pushed onto Lua stack, or an error is thrown.

Implementation

The annexed file thread.c implements this third solution for multithreading.

It is partly based on the source code of LuaDura, but with all references to

printer communication removed. Instead, it exports some simple tasks, which

are widely available and used. The implementation is aimed to be both an

example program and a startup file in which you can add features for your

application.

The code supports both Windows and POSIX preemptive threads. To mini-

mize the number of compilation switches, the following POSIX functions are im-

plemented under Windows using native objects: sem init, sem post, sem wait,

sem destroy, pthread create, and pthread cancel. With a simple similar in-

terface, it should be possible to run the example on other operating systems as

well.

The implementation does not need any change in Lua 5.1 sources. Like

all standard Lua libraries, this code only uses the official API. And like Lua

itself, it is written in clean C, thus should compile unmodified in both C and

C++ languages.

A pool of C threads is maintained in the form of four lists, all initially empty.

When a coroutine ask for a blocking operation, we look at the idle list of threads

and take one from that list. If the list is empty, a new native thread is allocated

and initialized. A table placed in Lua registry is used to keep track of which

coroutine is using which C thread at any time.

Like with the coroutines, there is no need to explicitly close the threads.

They will be collected like any other objects. If you are low in system resources,

you can force a garbage collection by calling collectgarbage explicitly. For this

purpose, a second table is present in the registry. Each time a thread is taken

from the pool to execute a command, a userdatum is created and placed in the

table. When the thread becomes idle again, the table entry is set back to nil.

Because this userdatum is bound to a metatable of which the

gc method is

overridden, the unused operating system resources can be freed when a garbage

collection occurs.

The module exports one global thread table with a few functions. Four user

functions are exported into that thread table. They were chosen because they

are blocking functions, are simple and are available on most platforms. They

only pretend to be examples for user needs, although they may be useful as is.

• thread.sleep(milliseconds): a delay function, implemented with Sleep

on Windows and usleep on Unix.

• thread.system(command): calls the ANSI system function, which in turn

starts a command interpreter: sh on Unix, cmd.exe on Windows NT.

294

24 · Enhanced Coroutines in Lua

• thread.gets([maxbytes]): reads a line of text from the standard input. It

is implemented using the ANSI fgets function.

• thread.wget(hostname, filepath, [port]): a very minimal HTTP client,

like the Unix command line utility wget.

All functions follow the same scheme. They do not execute directly their

associated base function. Instead, each one fills a message structure with input

parameters from the Lua state. Necessary checks on the parameters are done at

that time. The message is then passed to one native thread for execution. And

the function should yield until the message finishes executing.

Here we face the major difficulty.

It is impossible in the standard Lua

implementation to yield inside a C function: only Lua functions can yield.

Although there are non-portable patches that allow this, we will stay with the

standard distribution and thus avoid yielding inside C code. Therefore, it is

necessary to use the following idiom, just after having sent the message to the

thread:

return lua_yield();

The lua yield function will register the yielded status into the Lua state,

and return –1. The net effect is that the calling Lua function will yield just after

the C function ends. When the coroutine is resumed, it would go ahead after the

call as if the command was successful. Because it is not the case, the C function

must be called again. The second time, the input parameters are useless, but

we will suppose that they are the same as before, to simplify the coding. If the

native thread has now finished executing the command, either the output data is

copied from the message back into the Lua state, or luaL error is called with the

error message. If the command is still executing, we again return lua yield().

So we have to call any of the four user functions inside a loop, necessarily

written in Lua language. As we wish to hide this complex mechanism from

the user scripts, the initializing function luaopen thread runs the following Lua

chunk just after having registered the user functions into the thread table:

for name,fct in pairs(thread) do

thread[name] = function (...)

local result

repeat

result = { fct(...) }

until result[1] ~= thread

return unpack(result)

end

end

With this mechanism, each C user function is overridden with a Lua closure

that calls the underlying C function inside a loop until it does not yield, and

returns all its result values.

The original user function is stored inside a

295

non-local variable, so it is inaccessible to user scripts. The loop test condition

may be quite surprising. Remember that when resuming a previously yielded

C function, its apparently returned values are in fact the parameters passed to

coroutine.resume. We just decided that the main scheduler function will pass

the global thread table as parameters for coroutine.resume. With this arbitrary

but unique value, the above loop is able to know if the C function has previously

yielded or not.

In order to avoid for the scheduler to do an active polling on working threads,

which would waste CPU time, a thread.wait function is exported in the library.

It will wait for an event and return the first coroutine from the finished list.

Coding

Now let’s take a closer look into the C implementation. Each C thread has a

state variable associated with it for the synchronization mechanism. The four

states used by the state machine are as follows:

• IDLE: Lua can send a command to the native thread.

• REQUEST: A command has been sent, execution can start. Lua coroutine

yields.

• WORKING: A command is in execution in native thread. Lua coroutine also

yields.

• FINISHED: A command has finished execution, Lua can read back results.

There are four double linked lists of threads, one for each state. Each time

a thread state changes, its item is removed from one list and placed into the

next one. A shared semaphore assures that operations on these list cannot be

interrupted by other threads. Also to avoid any real time problem, Lua is only

allowed to exchange data with the thread during IDLE state for the command,

and during FINISHED state for the output read back.

An important helper function is retrieve thread data. Given the Lua corou-

tine represented by the current lua State parameter, it returns a pointer to a

C thread structure. Four situations can occur:

• If called from the main Lua thread (lua pushthread returned 1), simply

returns nil.

• If a command is already started for the coroutine, uses the mapping table

to return the associated thread.

• If the list of idle threads is not empty, returns its first element.

• Otherwise, allocates a new structure and fills it with a C thread and two

semaphores. The mapping table is updated accordingly.

296

24 · Enhanced Coroutines in Lua

The central function in the code is exchange with thread.

After having

got the native thread structure from current coroutine, one of these scenarios

happens:

• If called from the main thread, the message is executed immediately; if an

error occurs, luaL error is directly called.

• If the thread state is IDLE, the message is passed to the thread. If an error

will occur during the execution by the native thread, the error string is

stored inside the message structure.

• When the thread state is FINISHED, the result is copied from the native

thread. If an error has been stored in the message structure, luaL error is

called with that text error message.

• Otherwise, this means that a command is still in execution; the function

yields again.

The synchronization between the native thread and the user functions uses

two semaphores for each thread. The first one controls the start of command.

It is signalled by the user function and waited by the thread. At the end of

execution, the opposite is done: the thread signals a semaphore awaited by

the user function. It also signals a shared semaphore that will be checked by

thread.wait.

An important function is thread.wait. It is responsible to stop the execution

of the main thread, without wasting CPU time, until one of the native threads

has finished execution of their command. If the list of finished threads is not

empty, it returns the first one. Otherwise, it waits on the shared semaphore

signals by the threads and tries again.

Wget user function

The only slightly complex user function in the library is called thread.wget and

is designed to perform an HTTP request to a web server. It can be used to build

a simple Web browser or a robot. This implementation does not need LuaSocket

library: it is completely written using simple C socket functions.

Typically, requesting several files in parallel takes less time to download than

if they are retrieved one after the other. This function is an alternative to the

example found in chapter 9.4 of “Programming in Lua”, which uses the select

function of LuaSocket library, with pure coroutine functions. If you have read

that chapter, you will be able to compare both approaches.

Function thread.wget takes three parameters: the server host name, the

file path to retrieve and the port number, defaulting to 80. Alternatively, the

second parameter can also be the complete HTTP request header, if you need

more control over the request.

The function just outputs the whole returned data into a single Lua string,

containing both the HTTP response header and the file retrieved. It is easy

297

to separate the header and the content parts in Lua language, using regular

expressions:

header, content = data:match("(.-\n)\r?\n(.*)")

Error handling

As already mentioned, if an error occurs during a command execution, we cannot

call lua error directly, because the native thread has no access to the Lua

interpreter object. Instead, it calls the local helper function error. This latter

begins by closing opened resources: freeing the output string and closing the

current socket, if available. Depending on whether or not it is called from a

coroutine, it then calls luaL error directly or stores the message for the caller.

The return value is only a syntactic trick to enable the use of the idiom like this

one inside the user functions:

return error(msg, "Some error occur");

This is also the reason for the existence of an unnecessary return value in

user execute * functions, although the return status could be used by the main

thread function to determine whether or not the command succeeded (in order

to perform logging, for example).

Example of Lua code

Scheduler

Besides the C implementation of this thread library, let’s have a look of some

Lua code using it. First, we need some form of scheduler function, which is a

sort of operating system replacement. The simplest form of the scheduler is this

one:

function thread.sched(threads)

while next(threads) do

local thr = thread.wait()

if not thr then thr = next(threads) end

if not coroutine.resume(thr, thread) then

threads[thr] = nil

end

end

end

The argument is a table of coroutine objects. Repeatedly, while there are still

active threads, it waits for the first thread that has finished a blocking com-

mand. It tries to resume it, passing the global table thread as argument to

coroutine.resume, so that the loop of previous listing knows that it had yielded.

298

24 · Enhanced Coroutines in Lua

If the resume fails, this either means that the user script has normally finished

its execution, or there was an error. On both cases, the thread object is removed

from the table, and will be garbage collected some time later (provided there is

no other reference to it). In the case of an error, it would be preferable to display

the error message or log it to some file, depending on the environment, but this

was just a simple example. Another limitation to this function is that it does not

resume coroutines which have yielded for other reasons than inside the thread

library (by calling coroutine.yield). When a coroutine is created, it is in the

suspend state, so we have to resume it once before calling the scheduler function

(alternatively, thread.sched may resume all threads before entering the while

loop).

Interpreter

In a multithreaded Lua script, the main function is typically the scheduler

function, which is called after the coroutine objects have been initialized. When

called from the standalone Lua interpreter, this means that until the last thread

has finished its job, no prompt is issued to the user. This can be annoying, since

you might want to keep the control of what is going on. But it is easy to write

a new interpreter in Lua, using the exported thread.gets function. Here is an

implementation which mimics the behavior of the regular Lua interpreter:

function thread.shell()

local line = ’’

local prompt = ’--> ’

while true do

io.write(prompt)

prompt = ’--> ’

line = line .. thread.gets(1000)

if line == ’quit\n’ then return end

line = line:gsub(’^=’, ’return ’)

local fct, err = loadstring(line, ’@stdin’)

if fct then

local res = { pcall(fct) }

if not res[1] or #res > 1 then

print(unpack(res, 2, #res))

end

line = ’’

elseif err:sub(-7,-1) == "’<eof>’" then

prompt = ’-->>

’

else

print(err)

line = ’’

end

end

end

299

The function is continuously issuing a prompt, slightly different from the de-

fault one of lua.c, so that you know you are in a multithreaded script, and reads

a line with the blocking function thread.gets. Until you press the RETURN key,

this script is suspended and other threads can execute other tasks. Like the

regular interpreter, an equal sign at the beginning of the line is a shortcut for

the return statement. The line is then compiled. If a compilation error occurs

at the end of the block, this means a multiple line instruction is typed, and so a

different prompt is issued, and the next read line is concatenated to the previous

one. You can exit the function by issuing the command “quit” on its own.

Complete example

The web site contains a more complete example for running simple user scripts.

All four user functions previously discussed are run in parallel. You may notice

that they do not to worry about the underlying multithreading; the scheduler

function with the help of the thread.wait function takes care about all low-level

synchronization management.

25

Using Lua in Pascal

Jeremy Darling

Why Lua?

In the world of Pascal development there are plenty of native solutions for

scripting an application.

Most of these are built with the Pascal language

itself and this only leads to limiting the user base for the scripting language in

question. Utilizing a well known and commonly used scripting language within

our applications only helps to expand our user base in the end.

There are other common scripting languages out there (JavaScript, Monkey,

and VBA to name a few), but as of the time of this writing, Lua is the most

robust and supported scripting language available. Due to the nature of how it

is built, Lua allows for great flexibility within an application and it incorporates

into Pascal applications seamlessly.

The basic needs

Before we start talking about how to use Lua within an application, we need to

discuss what is needed to get Lua on your system and use it within your projects.

First, you will need a copy of the Lua interpreter, you can download that

from lua.org easily enough. You will also need a copy of the Pascal headers for

Lua: I personally recommend the version on my website called pLua. You can

get it at http://www.eonclash.com/ just look under “Projects/Pascal Wrappers”.

Make sure you download the latest version of pLua and not the Generic Pascal

302

25 · Using Lua in Pascal

Wrappers as many bugs have been fixed and new features introduced in pLua

that haven’t been applied to the Generic Wrappers.

Instead of focusing on the “common” implementation of Pascal (Delphi), I will

be focusing on generic ANSI-ISO Pascal in its Object Oriented form. This will

allow the same code to be executed within FreePascal, Lazarus, Delphi, Kylix,

and most other implementations of Pascal.

We will also need a design problem to solve.

The problem

In order to provide the best walkthrough of integrating Lua into a Pascal appli-

cation as possible, I’ll present an application that would benefit well from Lua

integration. The application is a game of sorts where the user is up against the

HAL 9000 and must convince it to release control of the ship.

To keep with the ubiquitous first project (“Hello World”), our first implemen-

tation will load whatever script is passed in and execute it.

Project foundations (setting up the project)

The same process will be true for all projects here on out, so it will only be

covered here once. Setting up a new project to use Lua within Lazarus requires

that the compiler know where the libraries are and where to output the final

application. (The same is true for Delphi, Kylix, or FPC.)

Open Lazarus and click on Project/New Project, click Program, then Create.

You should be presented with a blank project and the basics of a console appli-

cation. Now, click on Project/Compiler Options, select the Paths tab, and enter

in the appropriate location (where you extracted the Lua wrappers to) into the

“Other Unit Files (-Fu)” and “Other Sources (.pp/.pas, used only by IDE not com-

piler)” edit boxes:

[pic]

Click Ok to close the dialog. Then do a Save All.

This sets up the Lazarus environment so that it knows where our Lua source

files are located (Other Sources edit box). The Other Unit Files edit box tells it

where to find the Lua libraries when it compiles our source.

Next change the uses section to include the proper units that you will need

access to (if you want to make sure that all units are available and compliable

then include lua, LuaWrapper, LuaObject, Cutils, and LuaUtils).

Perform a build to test everything (Ctrl+F9) and you should get a message

stating “Project name successfully built”.

Defining HAL 9000

Before we can actually build anything we need to define exactly what we are

going to build. As it’s outside the scope of this document to explain project design

and documentation, here is a basic idea showing what HAL 9000 will do:

303

Start Application

Check to see if game.lua file exists

If so then load it

Else

Throw an error

Setup the game and execute the script to prepare the environment

Player input and processing

Building HAL 9000

Our first iteration of HAL 9000 (H9K) will use lua.pas and the methods that it

surfaces. While using LuaWrapper would make our lives easier, it would also

hide knowledge of how Lua works under the hood. I’m a strong believer of show

and tell, but to keep the article on track and not cluttered with code, please look

at the different source files as you work though. For this first project notice the

comments and how we are setting up Lua.

The source shows a very basic implementation of ReadLn being brought into

Lua so that we can allow for some user interaction within our scripts.

To start with, we initialize a new instance of Lua by calling lua_open and

storing the return value in the global L (L is a common name for the active

instance of Lua; this comes more from historical background then anything and

you could name your instance anything you wish).

Next we start a try–finally block to make sure that lua_close(L) gets called,

as we must close any instances of Lua that we open (as with object creation, or

memory allocation, we don’t want to strand any memory on exit).

Now we test to see if the user passed in a default script to execute, mak-

ing sure that the file exists. If the file does exist, we load it with a call to

luaL_loadfile; in the case that it doesn’t exist, we load our default script with

luaL_loadbuffer.1

The call to luaL_openlibs loads the standard Lua libraries (base, table, io, os,

string, math, and package) instead of having to make a call to load each library

separately. If you need support for the debug library you will still need to call

luaopen_debug.

We then register a custom method (all methods are functions in Lua and have

a default return value assigned to them, whereas in Pascal we have procedures

and functions) called readln.

This is done via the call to lua_register by

passing the Lua state to register to, the name of the method, and the address of

the method. It is important to notice that the prototypes for methods that are to

be exported for use in Lua must use the cdecl directive and must conform to the

standard Lua Method Header: receive one argument (a pointer to a Lua State)

and returning an integer (containing the number of items placed on the stack as

return values).

1This is a good point to mention that the luaL* methods are not a standard part of the base Lua API, but instead are part of lauxlib (or Lua Auxiliary Library). Since they are built into the standard Lua binary release, they have been included in the lua.pas wrapper.

304

25 · Using Lua in Pascal

Finally, we call lua_pcall to execute the script that we loaded previously.

This complies and executes the script, thus allowing us to use surfaced methods

within the script at a later time.

The order that is presented above must be followed every time. If you try and

open libraries before you load your script, you will receive exceptions stating

that a particular method cannot be found due to the fact that the Lua instance

has not been initialized. You can’t register your private methods before you load

the libraries, and you must execute the loaded source code before you can do

anything with it (more on this later).

Read and writing variables

Having a scripting language integrated into your application is more than just

loading a script and executing it. It is having the ability to read and write

variables, objects, records, and other information to and from the script itself,

thus allowing you to change the way that your application reacts or interacts

with the user.

The next logical step in getting Lua integrated into your application is to

setup global variables that can be modified and read inside the Lua script and

inside your application. With this little bit of knowledge you could easily use

Lua as a configuration storage device that will give you more flexibility than say

.ini or registry files.

If you have read up on Lua outside of this article (and I hope you have), then

you know that Lua works on a stack and uses different global spaces to store

information. One of these spaces is LUA_GLOBALSINDEX, which is a table that is

used to store global name/value pairs to be accessed by the current script.

Writing

To modify or create a new global value we need to do three things: first, we have

to push the name or index associated with our value; second. we have to push

the value itself onto the stack; finally, we have to tell Lua to set the value in the

global table:

Lua_pushliteral(L, ’MyIdentifier’);

Lua_pushstring(L, ’Some Value’);

Lua_settable(L, LUA_GLOBALSINDEX);

As you can see in the code above (lua_pushstring), there are methods to read

and write each variable type that we use (at least until we start using the

variant type wrappers presented within the pLua unit). This should be nothing

new if you have stored or retrieved information from a standard .ini or registry

object.

We place our identifier onto the stack first with a call to lua_pushliteral;

this could also be done with a call to any of the lua_push* methods. Lua uses

a hash table lookup, so any type (except nil) can be used as a key within a Lua

305

table. We then place the value that we want to associate with the name onto

the stack using the proper lua_push* method. The final call is lua_settable

with our Lua State Pointer as the first argument and the table identifier as the

second (LUA_GLOBALSINDEX is a constant for the globals table). We can simplify

this code by using methods from the pLua and variants libraries within Pascal.

The variant library provides us a VarIsType function that we can use to see if

the variant type is a string (varString). pLua provides us with a method called

plua_RegisterValue that takes a Lua instance, value name, a value (quoted if

the value is a string), and a table index (defaulted to LUA_GLOBALSINDEX) that

performs the above actions for us. You will notice that this method is used quite

a bit though the examples that accompany this article.

Reading

Retrieving the value back out of the table is almost as easy:

Lua_pushliteral(L, ’MyIdentifier’);

Lua_rawget(L, LUA_GLOBALSINDEX);

If lua_isnil(L, 1) then

Else

MyString := Lua_tostring(L, ’MyIdentifier’);

We push the name of the variable that we want to retrieve onto the stack. We

then call lua_rawget with our Lua state pointer and table index. Then we test

for nil (lua_isnil). If the value on the top of the stack isn’t nil, we get the value

using the proper lua_to* method.

Just like writing a variable, there are some support methods provided by the

pLua library that allow us to minimize our source code. The main one that we

are interested in is plua_tovariant. This function takes the Lua state and the

stack index that we wish to work with and returns a variant type that contains

the value.

More on methods

Once we can read and write global variables, the next logical step is to move up

to surfacing methods (procedures and functions) from our Pascal environment

to our Lua environment. We will also need a way to call Lua methods (functions)

from within our Pascal source code. We touched lightly on this before, but this

time we will be looking at the specific needs of methods inside and out.

Differences between Lua and Pascal

One of the first things to keep note of is the difference between return values

and arguments. In Pascal we can mark a method argument as a var arg, or

input/output variable, out arg, and we can pass back a single value from any

306

25 · Using Lua in Pascal

function. Procedures don’t allow you to pass back any value, but can still contain

var args.

This isn’t true in Lua. All methods return a value (default is nil), and there

is only one method type (a function). Parameters to a method are also static, and

their values are cannot be marked as output of any kind. A method in Lua may

return as many values as it wishes and the return types are not pre-defined.

Surfacing a method

The most basic of methods that you will need to call from a Lua script is the

print function. While the standard Lua libraries surface print for us, there are

some aspects of its implementation that may not be desirable. For instance, by

default, print will only print to the standard output interface (console window).

If we want to have our script print to another output (say a memo), then we will

need to override the default print hander.

Remember that, since Lua is developed in C, we must always use the cdecl

compiler flag to let the compiler know that we expect the method to follow C

rules and not standard Pascal rules.

Our first call is to lua_gettop. This call will tell us how many arguments are

being passed from Lua to our method. In some methods you can use this as a

quick and dirty test to see if you are receiving the proper number of arguments.

In most, you will also need to provide a type check along with the argument

count check.

For our print implementation we will only be using this as a high for our

counter. Next we create an empty string that we can place the passed in values

into. We then iterate through all of the values passed in (notice that the Lua

stack starts at a positive index of 1 instead of 0 like C and Pascal), placing each

argument string representation into our container. We then add the combined

string value to our memo.

The last step is to tell Lua that we didn’t put anything back onto the stack (a

return value) and thus that it should set the return value itself. Remember that

Lua methods always have a return value.

Now that we have our method defined and ready within our application we

need to surface it to the Lua virtual machine. We do this after we register

the libraries that we want to make use of (in case we are overriding default

behavior).

Our first call is to lua_pushliteral with the Lua instance and the name that

we want scripters to use to call our method. This is followed by placing the

pointer to the method onto the stack using lua_pushcfunction. We then make a

call to lua_settable with the LUA_GLOBALSINDEX (that should now be becoming

very familiar).

Getting something back

When we want to return a value from a method call we have to push it onto

the Lua stack; we also have to tell the Lua virtual machine just exactly how

307

many items we have placed on the stack. To demonstrate this, we will surface a

method called Size, which will return the width and height of our applications

main form.

Unlike print, we don’t care how many arguments Lua is passing in; instead,

we only care about returning values. In cases like this, it’s a good idea not to

waste time checking the number of input arguments and instead just place our

values on the stack.

The two calls to lua_pushinteger place the width and then the height onto

the stack and are followed by us setting the return value of the function to 2.

This tells the Lua virtual machine that we returned two values. It’s important

to note the order that we pushed the return values, as when we use this method

from within our script we will need to know what to expect first and last. I’ve

stayed with the typical X-then-Y structure, but there is no good reason it couldn’t

be Y-then-X.

We register this method the same as any other method; in fact, this is a good

place to introduce a procedure for registering methods. pLua has a wrapper to

achieve this as well, and when you look at the source code you will find it using

plua_RegisterMethod instead of all of the hand code.

Calling Lua methods

It’s time to call Lua methods from Pascal. The code that is in place within the

example of this section is more complex than what I am going to describe, but

follows the same principles.

We have to tell Lua what method we would like to call, and then tell it what

argument values we want the method called with. If appropriate, we then need

to check for return values.

The first thing that we do is place the name of the method to be called onto

the Lua stack. We then use lua_rawget to retrieve the method address (if it

exists) and replace our name with it. It would be a good idea to perform a nil

check at this point, but it’s not necessary.

We need to loop through the values that we want to pass as arguments to the

method and place them onto the stack. This is followed by a call to lua_pcall

that makes the performs the actual call of the method within Lua.

After we have called the method, we then iterate all of the return values

placing them into a return array. It’s important to note that the return values

are in reverse order, thus they should be placed as: item n into slot 0 with item 0

going into slot n − 1. The reason for the ordering is more apparent if you think

about our Size example above: We placed the width and then the height onto

the Lua stack. So width is at position 2 with height at position 1.

Using records and objects

Within Lua there is no object type defined. Yet, it would be very difficult to

think of integrating a scripting language into an application without the ability

308

25 · Using Lua in Pascal

to surface objects and records. No worries though, Lua has mechanisms in place

to take care of object handling.

There are no hard and fast rules as to how you implement or handle objects

within Lua. In fact, a quick search of the web will result in many different ways

to handle objects. I’m going to present the one that I use and that I’ve found to

work very well in the end this isn’t the end all be all answer to objects. [CHECK!]

If you think that I forgot about records, you’re mistaken. A record is nothing

more than an object without methods. More accurately objects are just records

with method pointers. Thus, if we can handle objects then records will fall in

and work automatically.

Metatables

From the Lua manual (section 2.8): “Every value in Lua may have a metatable.

This metatable is an ordinary Lua table that defines the behavior of the original

value under certain special operations. You can change several aspects of the

behavior of operations over a value by setting specific fields in its metatable. For

instance, when a non-numeric value is the operand of an addition, Lua checks

for a function in the field __add in its metatable. If it finds one, Lua calls this

function to perform the addition.”

Metatables make object types possible within Lua. It is outside of the scope

of this article to cover all aspects of metatables, but we will need to at least

understand them at their very basics in order to implement objects and records.

If you understand how exactly OO works, then the concept of a virtual

method table (VMT) should not be alien at all. If you don’t, then think of a

VMT as an array of pointers that says what method address should be called

when a particular method is called in the code. This way an object can contain

only its private information (variables) and use a global method table (the VMT)

along with its private address (object pointer) to save space and make its calls.

This is the basis of polymorphic programming in general (OO). A metatable

allows us to implement a VMT of our own. The primary difference is that the

metatable doesn’t only have to surface methods, it can surface values, alternate

implementations (we won’t cover this), and methods.

Garbage collection and the Lua registry

Lua has garbage collection; basically this means that Lua will collect and delete

all types when they are no longer referenced. When it comes to objects this could

be a very bad thing, if for instance we created a display manager that knows to

draw all of the objects it contains every frame, we may not reference it again.

According to Lua it would get garbage collected right after we finished using it.

The Lua registry is a table much like the globals table. It doesn’t actually

exist within the Lua stack or environment; instead, it’s a virtual table that can

be used much like the Windows Registry. As long as something is referenced in

the Registry it won’t be garbage collected. The Lua registry gives us another

feature that we need: as long as our objects are referenced within it, they won’t

309

be optimized or moved within the Lua environment. This is another feature

within Lua that lets it run faster, cleaner, better, but it could also destroy our

objects.

We place items into the registry just like you would with any other table.

Simply push the name of the value that you want to store then push the value

itself. Call lua_settable or lua_rawset with LUA_REGISTERINDEX as the table to

being place the value into the registry.

To retrieve a value you push the value name and then call lua_gettable or

lua_rawget depending on your situation.

Finally introducing objects

It makes sense to look at how we can implement objects in Lua. Rather than

completely re-invent the wheel, we will be using a piece of code that has already

been developed within Lua as a reference.

This code can be found in the

Lua wiki at http://lua-users.org/wiki/InheritanceTutorial and I strongly

suggest that you read it before continuing.

Back to the registry

In order for Lua to make use of Pascal objects we will need to make sure that

Lua does not garbage collect its part of the object. The object will also need to

retain a pointer back into Lua so that it can retrieve method pointers, variable

values, and anything else that we choose to allow Lua to control.

This is where the Lua registry comes in. When we create a new object (either

from a Lua script or from our application), we will need a way to let Lua know

about it and let Lua know that we are managing it. The registry is the best place

(note that’s best and not only place) to do this.

Upon our objects instantiation we will need to request a registry entry. To do

this, we will use the pLuaObjects unit and a few calls to luaL_* methods.

We also need a way to remove our reference when the object is destroyed so

that Lua can garbage collect its part of the equation.

Hopefully the comments within the source will give good presence as to what

is going on, but if they don’t then please use your favorite search engine to learn

more about the Lua registry and garbage collection.

Object properties

Lua tables have built-in meta-methods to allow developers to modify their de-

fault behavior. We will use this to our advantage to implement variables (read

and write for the case of this document; read only or write only should be easy

to figure out) and methods. For variables we will need to override the default

__index and __newindex table entries to call our own getters and setters.

310

25 · Using Lua in Pascal

First we have to receive back a copy of the object that Lua is referencing.

This is achieved with a call to a custom function that returns a Pascal Object

from a Lua stack reference.

Within our TLuaObject descendant we will surface helper methods to read

and write variable values. These methods are: GetPropValue, SetPropValue,

GetPropObject, SetPropObject. They do exactly what they say: get a value, set

a value, get a sub-object and set a sub-object.

If you choose to use the pLuaObjects unit then you will find that you can

quickly and easily wrap existing objects. Take a look at the pLua demos folder

for the pLuaObjects and pLuaObjects2 demos for more explanation.

Object methods

Methods are a bit trickier. We will need to tell Lua that the method exists and

what class type the methods are tied to. We will then need to write a method

handler that prepares the proper arguments and then passes them to the object

instances actual method. We will need to check for results and out parameters

and put them back on the Lua stack. The latter part is the same as when we

covered methods above, the former is explained in the source code.

And the object

From everything above we can finally implement objects into our application

and have them extended by Lua scripts with events, methods, and properties.

H9K shows exactly how all of this comes together and allows you to create and

extend Pascal objects from within Lua itself.

Records

While I know that I said if we supported objects that records would fall in

automatically, but there is one caveat to records. You must use record pointers

instead of actual record types. This is due to the way that records are passed and

assigned within Pascal. If you assign an object instance to an object variable,

the variable contains a pointer to the object instance. On the other hand, if

you assign a record to a record variable then the entire contents of the record

are copied over to the variable. If you can keep this in mind and use record

pointers in all locations, then you will actually gain twofold. First, you will

guarantee that your records will work properly, and second, you will notice that

your application uses less memory.

Calling within a loop

There is a special case when it comes to Lua. That is calling a Lua script or

method from within a loop. If you have jumped ahead and attempted to do this

311

you might have noticed that you received access violations after the first call.

This is because simply loading the Lua code does not compile the source and

prepare the virtual machine (first part of the article).

Instead we need to load the source file, execute it once, and only then we can

call our loop routine.

Final words

Everything that has been covered in this article has been wrapped up into a nice

little package for you to work with. The files LuaWrapper.pas, pLuaObjects.pas,

and LuaObject.pas have helper classes, functions, and base classes for you to

extend and use. They have been built from experience and have quite a bit of

testing going on with them all of the time. As with all libraries though, you

should read and understand them as the author(s) are not responsible for any

damage to your system.

26

Porting Lua to a Microcontroller

Ralph Hempel

The Lua language was designed from the beginning to be small in its memory

footprint for both the developer and the target machine. The basic philosophy

is to provide a concise and unambiguous syntax that the developer can use and

depend upon.

The purpose of this gem is to outline some of the issues that come up when

porting Lua to an extremely memory constrained target. I’ll go over a basic in-

troduction to the target, which is the LEGO MINDSTORMS NXT brick, talk a

bit about how the run-time library is designed, and then introduce my compro-

mise between 32-bit long integers and single-precision floating-point numbers,

which I call “flongs”.

The target processor – why use Lua to program LEGO?

I have been fascinated by little languages ever since I started programming

embedded systems over 20 years ago. At the time, Forth was one of the few

languages that could be ported to a microcontroller and provide a mechanism

for compiling and interpreting right on the target hardware.

In late 1997, the original LEGO MINDSTORMS RCX system was released.

It had an H8/300 microcontroller with 32K of OTP memory containing the basic

firmware and about 32K of external RAM for applications. Again, at the time

314

26 · Porting Lua to a Microcontroller

Forth was about the only language I could imagine running right on the brick,

and the result was pbForth.

In 2005, the new LEGO MINDSTORMS NXT was released. It has an ARM7

based micro with 256K of rewritable on-chip FLASH and 64K of RAM. I could

have easily ported Forth to that device as well, but the syntax of Forth can make

marginally sane programmers cross the line.

In searching for other small interpreted languages, I evaluated two addi-

tional options: Lisp and Lua. While very powerful, Lisp suffers from the same

problems as Forth in terms of syntax. Since most programmers are very comfort-

able with infix notation, Lua provides a familiar syntax compared with Forth’s

postfix and Lisp’s prefix notation.

I really wanted to try to port Lua to this device just to see if it could be done,

and then realized that I would have a very powerful way of programming robots

interactively. This might be useful for academic purposes when the limitations

of the original GUI programming environment provided by LEGO are reached.

There are some special challenges to getting even a minimal Lua system

working on a deeply embedded system. Besides the obvious one of putting

together a toolchain that generates the code image, the more subtle problem

is to figure out what can or should be removed in order to make a useful system.

Think before you get started

The porting project did not have any immediate urgency, which gave me lots of

time to think about it before I ever got started. It also gave me time to read

about Lua, poke around in the Lua mailing list archive, and write a few non-

trivial programs in Lua. I was able to take the time to browse the Lua source

at leisure, noting how the different layers were isolated into separate files, and

generally getting the lay of the land.

Basically, I’m lazy. This forces me to work smart to get as much done as

somebody that works hard.

The weeks I spent getting the feel of the Lua

code convinced me that any time I spent fiddling with the source code would

be completely wasted. Eventually I had absorbed enough to put down these

guidelines to get me started:

1. Use as much of the stock Lua source distribution as possible.

2. Only load the base, table, and string libraries initially.

3. Add target specific routines in their own library.

Being lazy has also made me dependent on make, which I use in all kinds of

projects to make sure that I have to think and type as little as possible once I’ve

figured out how to get a job done.

I have a standard framework that I set up for any embedded systems project,

which I won’t describe in detail here but is available in the pbLua distribution.

In general terms, it starts with a directory that has the processor specific startup

315

files and a few I/O routines that blink LEDs, make sounds, or read and write

characters thought a serial port.

Once I have that framework set up and some minimal code compiled that will

flash LEDs, make sounds, or read and write characters through a serial port I’m

ready to move on. One other thing being lazy has taught me is that you will do

a lot more work later if you don’t start simple, gain confidence in your tools, and

only then add complexity.

Add the Lua kernel code to the build system

The next step was to take Lua source code and put it in a subdirectory in my

development tree. I then built a makefile that is compatible with my framework

and built the Lua object files from source.

The Lua code subdirectory had the full Lua version number in the name,

which was 5.1 when I first started. Since that time, I’ve upgraded the source

to 5.1.2 (also in its own directory). The upgrade did not change any of the

fundamental Lua operation, it is only a bug-fix release.

The upgrade was almost trivial because all I had to do was copy the old

luaconf.h and the custom makefile to the new directory. All I had to change

at the higher level was the name of the directory where the Lua source could be

located.

But surely some things had to be changed to port even the basic Lua code

to the ARM7 target? Yes, and thankfully all the changes were isolated to the

luaconf.h file. When changing this file, it’s generally not a good idea to actually

make the changes all through the file.

The designers of Lua have thoughtfully provided a space at the end of the

file where you can make all the changes you want and keep them in one place

for easy reference by you and other programmers trying to figure out what you

have done.

One thing most C compilers will complain about is redefining something that

has already been #defined. For example, I had to change the LUA_MAXINPUT from

512 to 256 bytes to conserve as much RAM as possible. To prevent the compiler

from complaining, make sure you #undef the value first, then #define it, like

this:

#undef

LUA_MAXINPUT

#define LUA_MAXINPUT 256

The other major change in luaconf.h involved making the default number

system of the first pass of the port long instead of double. Listing 1 shows those

changes.

If you’re reading closely, you’ll see that luai_numpow() just returns 1 instead

of the correct result. That’s a side effect of my lazy programmer brain, which

decided that I would deal with that problem later.

Getting the core Lua code to compile cleanly under my custom built gcc

toolchain was very easy — it basically worked the first time. Linking was an-

other story, and is the subject of the next section.

316

26 · Porting Lua to a Microcontroller

#undef LUA_NUMBER_DOUBLE

#undef LUA_NUMBER

#define LUA_NUMBER_LONG

#define LUA_NUMBER long

/*

@@ LUAI_UACNUMBER is the result of an ’usual argument conversion’

@* over a number.

*/

#undef

LUAI_UACNUMBER

#define LUAI_UACNUMBER long

/*

@@ LUA_NUMBER_FMT is the format for writing numbers.

@@ lua_number2str converts a number to a string.

@@ LUAI_MAXNUMBER2STR is maximum size of previous conversion.

@@ lua_str2number converts a string to a number.

*/

#undef LUA_NUMBER_FMT

#undef lua_number2str

#undef LUAI_MAXNUMBER2STR

#undef lua_str2number

#define LUA_NUMBER_FMT

"%li"

#define lua_number2str(s,n) sprintf((s), LUA_NUMBER_FMT, (n))

#define LUAI_MAXNUMBER2STR 32 /* 16 digits, sign, point, and \0 */

#define lua_str2number(s,p) strtol((s), (p), 10)

/*

@@ The luai_num* macros define the primitive operations over numbers.

*/

#if defined(LUA_CORE)

#include <math.h>

#undef luai_nummod

#undef luai_numpow

// Note, numpow() is just a placeholder until we get the real number

// system working...

#define luai_nummod(a,b) ((a)%(b))

#define luai_numpow(a,b) (1)

#endif

Listing 1.

317

Building a better run-time library

After building and linking a minimal Lua system, you’re left with a lot of

undefined symbols which are generally part of the run-time library (RTL). This

is something that systems developers don’t think too much about because the

libraries are often part of the operating system distribution.

Embedded systems developers have a couple of choices in the matter. The

first is to use the RTL that ships with your development system. In most cases

this is fine, but sometimes it’s a good idea to start from first principles and build

your own RTL.

In the case of GNU gcc, the standard embedded RTL is newlib. It’s a fine

library that gets a lot of use, but for this project I felt it was a bit bloated

especially in the IO section.

One reason to build your own RTL is that it can follow you around to other

projects, and you’ll have a better appreciation of some of the tradeoffs that

different implementations will give you. This is especially true in the standard

IO library.

Building your own RTL is not as hard as you might think if you break the

job down into smaller pieces and are careful in choosing the base code for each

part. Being lazy comes in handy again if you take the time to scour the web for

libraries that are already written and tested. Look for code that’s written to be

portable and runs on a variety of hardware. If you’re concerned about licensing,

look for code with an MIT style license (like Lua) or maybe a BSD license (like

FreeBSD). These give you the most flexibility in using the code but they do not

require you to publish the changes or improvements you might make.

Thread-safe considerations

One of the buzzwords you’ll hear in a discussion on embedded libraries is “thread-

safe” or “re-entrant”. In multi-tasking systems it is quite common for a routine

called by a task to be interrupted at any time for a task switch. The new task

may call the original routine as well. If the routine is thread safe, it won’t get

confused and return incorrect results to either task.

One of the first steps towards becoming thread safe is to not use global

variables. As long as a routine allocates all of its variables on the calling task’s

stack, then chances are the routine is thread safe.

The ANSI C library

The ANSI C library is a set of basic routines that are used by almost every

program you are likely to write, so it’s a good idea to get the most complete one

you can find.

The first and most obvious set of routines to get into the library are the string

routines. These are often written in assembler, but modern compilers do such a

good job of optimizing them that it’s often more trouble than it’s worth to do it

318

26 · Porting Lua to a Microcontroller

in assembler. If you do find that the routines are too slow, then go ahead and

rewrite them later. In the meantime you’ll have code that works.

The basic string library that I settled on is based on the Minix project from

Vrije Universiteit in Amsterdam. It’s a well known project that has been in use

for years, so I am confident that the library has had most bugs eliminated.

Besides the string routines, the Minix project source also yielded routines to

handle character classification, memory operations, and basic error and locale

handling:

memory

string

characters

locale

memchr.c

strcat.c

strncpy.c

isalnum.c

errlist.c

memcmp.c

strchr.c

strnlen.c

isalpha.c

locale.c

memcpy.c

strcmp.c

strpbrk.c

isascii.c

setlocale.c

memmove.c

strcoll.c

strrchr.c

iscntrl.c

strerror.c

memset.c

strcpy.c

strstr.c

isdigit.c

strcspn.c

strtoflong.c

isgraph.c

strerror.c

strtol.c

islower.c

strlen.c

tolower.c

isprint.c

strncat.c

toupper.c

ispunct.c

strncmp.c

isspace.c

isupper.c

isxdigit.c

chartab.c

There are a few things I changed in order to save space, and the main one is

in errlist.c where I set all errors to unknown except for the ones that are set

by Lua and its libraries. Other than that I did not touch the code because it was

unlikely that I’d be able to improve on it without breaking something.

The other place where I’ve taken some liberties is with the number conver-

sion part of the string library. The routine that the Lua interpreter uses to

convert strings to numbers is luaO_str2d. This routine operates as follows:

1. First, it tries to use the lua_str2number macro to read the string as a

number. If it makes no forward progress in reading the string, then the

conversion fails completely and luaO_str2d exits returning 0.

2. Next, it checks for a leading upper or lower case ’x’, in which case it

tries to convert the string as an unsigned hex number using the strtoul

function.

3. Next, remove any trailing spaces and check if we’ve reached the end of the

string we’re trying to convert, and if so, luaO_str2d exits returning 1.

4. If we get here, we have illegal trailing characters after the end of the string

we are trying to convert, so luaO_str2d exits returning 0.

I needed to make sure strtoul was available, and it’s part of the strtol.c

file. As you’ll see later in the section on the math library, the trick was figuring

out a way to get the interpreter to differentiate between floating point numbers

and integers in a way that was not too complicated. The custom routine I came

319

up with is called strtoflong and it’s a fairly straightforward solution to the

problem which is described in detail below, in section on “flongs”.

That rounds out the basic RTL function needed for the project. I’ll go over

the I/O routines and math functions later.

Memory allocation library

Most programs written for desktop computers allocate memory from the heap,

which is an area of memory set aside for objects that cannot be allocated at

compile time. Embedded systems that I’ve worked on in the past never had a

heap because it’s generally bad practice to count on never running out of memory

over the lifetime of your product. It’s better to allocate the memory at compile

time and then you know how much you can use now and in the future.

The Lua scripting language makes frequent use of the heap though, so I

had to go looking for a tried and true memory allocation scheme that would be

portable enough to run on the ARM7 microcontroller. The basic calls required

are malloc, realloc, and free, just like on a big operating system.

Fortunately, Joerg Wunsch, the author of avr-libc has done a fantastic job

of porting these key routines to the AVR series of microcontroller and in the

process has pared the C code down to a bare minimum.

After reviewing a number of other implementations, including the one in

newlib, I decided to go with the routines in avr-libc.

One thing to note is

that these routines are not thread-safe as they don’t disable interrupts around

critical sections and they use global variables.

I decided that this was an

acceptable trade-off in this case as Lua was the only user of the heap and there

was only one instance of Lua running on the NXT.

The only change I had to implement was to ensure that the memory block

was a multiple of a long-word (4 bytes) in size, so that the data is always aligned

on long word boundaries. Without that change, the pointers to the next block of

memory ended up on non-aligned addresses and caused the processor to throw

a hardware exception.

Lua garbage collection and the heap

The other significant difference is that there is no extra space available when

the system runs out of memory so the concept of sbrk is meaningless.

This leads to some interesting choices when deciding how to do garbage

collection. The standard Lua interpreter runs garbage collection whenever the

memory usage of the system doubles. In other words, suppose that the default

memory usage of a Lua program is 8K. When the Lua machine determines that

16K is in use, a garbage collection cycle is started, and then the limit for the

next cycle is set to twice the current limit, or 32K.

This is not too bad for desktop systems with lots of memory, but it’s absolutely

fatal for an embedded system with the kinds of resources that we have available.

There are two places in luaconf.h that control the operation of the garbage

320

26 · Porting Lua to a Microcontroller

collection system. The one that needs to be changed is LUAI_GCPAUSE, which

defines the multiple of memory use that triggers a garbage collection cycle. The

default value of 200 (interpreted as a percentage) gives the operation described

above. By reducing the default value to 110 in the luaconf.h file, the pbLua

system forces a garbage collection cycle when the memory usage increases by

only 10%. Assuming that there is 32K available for dynamic memory allocation,

this means that the last garbage collection limit increase occurs when about 29K

of memory is in use.

File IO routines for systems without files

The other main difference between embedded and desktop libraries lies in the

file I/O implementation. On a desktop system, we routinely do operations on

files without thinking about how they are implemented underneath.

The pbLua system still needs to read input to get it to the parser and write

the output so that it can be viewed by a human or another computer. So it’s clear

that we’ll need some sort of I/O subsystem, but do we really need the complexity

of a full filesystem? It turns out that initially we do not, but it’s still a good

idea to use the standard I/O system calls to implement the subset of operations

that we need. In this way we can ensure the maximum compatibility with the

Lua source code and at the same time leave room for expansion if we ever do

implement a more sophisticated filesystem.

The first step is to look for all the standard I/O calls made by Lua, and then

decide on which ones we actually need to implement. Once again, the work of

Joerg Wunsch makes my life easy by providing a minimal abstraction of the

standard I/O library that is suitable for microcontroller. I have seen similar

work before, but Joerg has documented his work very well, and once again I did

not have to change a single line of code to get it to compile or work.

There is no concept of actual files on the NXT. Instead the I/O library as-

sociates streams with devices at boot time. The first implementation used the

USB device port built into the NXT as a virtual serial port. All I had to do was

provide routines to send and receive individual characters from the serial port.

The addresses of these routines were placed in device description blocks, and

these blocks were in turn associated with the standard input, output and error

stream descriptors.

Once that was set up, the Lua interpreter was ready to receive characters

from stdin, send them to stdout, and if I did something stupid, an error message

was printed on stderr (which was the serial port too).

A second pass at the problem resulted in an implementation that allows the

user to choose either Bluetooth or USB as the standard console interface.

I still need to finish implementing a basic flat file storage system that uses

the internal FLASH and does not place undue stress on individual FLASH

blocks. My current thinking is a block of file descriptors that has a special GUID

or signature that can be distinguished at boot time will be simple to use and

allows the directory block to move around in FLASH.

321

A new kind of math using “flongs”

The standard Lua programming language is compiled to use double-precision

floating-point math, which makes sense on a standard desktop or server class

computer. Double-precision math has the advantage of being able to represent

the full range of 32-bit integers exactly in addition to a very large dynamic range

with about 14 digits of accuracy.

For small embedded systems, and certainly for the LEGO MINDSTORM

NXT, double-precision math comes with a penalty. The ARM processor inside

the NXT handles 32-bit integers very efficiently, but must do a lot of extra work

to do floating-point calculations.

Fortunately, the designers of Lua thought about the speed-versus-precision

tradeoff at a very fundamental level and came up with a clever macro based

scheme to decouple the implementation of the arithmetic operations from the

language. What this means for some applications is that we can specify the

underlying numerical type for all of Lua’s arithmetic operations. (Note that

there is a distinction between arithmetic operations that are intrinsic operations

like + and * in the Lua language, and the math operations such as sqrt and sin

that are part of the math library.) The actual implementation is a section in

luaconf.h that tells the compiler how you’d like to handle the numbers and was

discussed in the section on adding the Lua kernel to the build system.

One part that I did not mention was that the Lua arithmetic operations were

based on standard C operators and are defined as macros in luaconf.h In other

words, the basic operators — addition, multiplication, modulo, etc. — all result

in standard C expressions, and depending on the underlying type of the macro

parameters, the compiler chooses either fixed or floating-point math.1

Trading speed for accuracy

In the process of thinking about how best to take advantage of the processor in

the NXT, I decided to make long the default arithmetic type for Lua. Most of the

routines in the NXT firmware source use long parameters. This includes timers,

rotation counters, and many other fundamental structures. If we had to convert

between float to long all the time we’d be wasting all kinds of CPU cycles in the

process.

If doubles are too onerous, then what not use a single-precision float? The

standard single-precision float has one disadvantage, and that’s a significantly

reduced range of precision when it comes to fixed-point numbers, and for that

1As an aside, one of the things I like to do when I’m evaluating a new programming language is to read and try to understand how it’s implemented underneath. The first language I learned (besides assembler and C) was Forth. It had a wonderful structure that made it obvious how it worked. When I saw the Lua source code, I knew I was looking at something that had evolved over time and had some deep thought behind it. It looks obvious when you read the code that this is the “right way” but based on my experience, there were probably a few failed attempts before this. . . From the high level parser to the virtual machine, and the API that handles the interaction between C or assembler libraries and Lua itself, the Lua core is beautifully organized and a model of good code. I have much greater confidence that code is correct if it looks like it was carefully crafted.

322

26 · Porting Lua to a Microcontroller

reason it’s not a good choice to be the single numerical representation of this

implementation of Lua.

In addition to that, most of the time we’re interested in doing fairly simple

math when we’re designing robots. While the overhead of converting all num-

bers to float is less than for doubles, it’s still significant. Even simple operations

like addition and subtraction become much more complicated with floats than

with longs.

While these two tradeoffs competed against one another, I started to think

about ways in which a compromise could be struck — and the result is a hybrid

number type that uses standard representations for long and float while making

maximum use of the benefits of both. I call it “flong”.

The breakthrough came when I realized that the practical application of each

of the number types is in several distinct domains — in other words you use one

numeric type for the task at hand and avoid mixing them unless absolutely

necessary.

I often get asked why I don’t use standard or even light userdata for the

new numerical type. The answer is quite simply speed. Light userdata values

don’t have metatables that we can use to override their operators, and standard

userdata exacts a toll on the C API side.

Using flongs with the C math library

The compromise flong type gives us the speed to do the math operations that

we do most often in long integer math, while giving us the ability to use an

alternative representation for floats when we need them.

The disadvantage is that all floating-point operations, even basic arithmetic

on floating-point values, is done through a series of API calls. One additional

feature I implemented was the ability to automatically convert numbers in the

input stream to long if they contain a decimal point. To use the new routine, we

just change one line in luaconf.h like so:

#define lua_str2number(s,p) strtoflong((s), (p))

Once that decision was made, I just had to find a suitable single-precision

floating-point library written in ANSI C. This was much harder to do than I

thought because most implementations of this kind were done in assembler

language, and were otherwise incomplete or incompatible with my hardware.

Essentially, the flong is a union of an unsigned long and float that looks like

this:

union flong

{

float f;

long l;

};

Since I’ve compiled pbLua to use long as the default numerical type, the

parameters to API functions are passed as longs too. The flong union provides

323

a way to translate between the forms as needed without resorting to typecasts.

Here’s an example of how this is used in the sqrt function:

static int _sqrt(lua_State *L) {

union flong fl;

fl.l = luaL_optint(L, 1, 0);

/* 1st argument is sin */

fl.f = sqrtf(fl.f);

lua_pushinteger(L, fl.l );

return(1);

}

Choosing a single-precision float library

Fortunately, the work of Jesus Calvino-Fraga for the SDCC (Small Device C

compiler) project fit my needs almost exactly. His code is designed to be used

with the SDCC system and has a very small number of minor code changes

required to work with my build system. But the end result is wonderful. I

now have a complete single-precision library upon which I can base the math

routines I need.

One deviation from the standard C trig functions is that all input and output

is in degrees, not radians. This is because I wanted the users of pbLua to be able

to pick up the concepts quickly, and using degrees instead of radians makes that

barrier much lower.

One additional function that is required is a way to convert between longs

and floats, and these routines do just that:

f = tofloat( 45, 1 )

-- f = 45.000001

i,d = toint(f)

-- i = 45, d = 1

I set the precision of the floating-point portion of a flong to one part in

1,000,000 which is good enough for the kinds of math we’re doing and the limit

for single-precision floats anyways.

Conclusion

Building a complete Lua interpreter under Linux, BSD, OSX, or even Windows

is relatively simple when you use the makefiles provided with the source code.

Building Lua for use on a microcontroller with no underlying OS support re-

quires more careful consideration of tradeoffs between speed, size, and accuracy.

Embedding Lua on a constrained micro forces the programmer to think hard

about many things that are taken for granted on a desktop system.

From

math to memory, strings to stdio, almost everything is under the control of

the designer, and knowing what the tradeoffs are can help you to make better

decisions.

In the future, this project may take advantage of other work being done in the

Lua world, including keeping some table information in non-volatile memory,

324

26 · Porting Lua to a Microcontroller

and allowing pre-compiled code chunks to be stored and used later. I also plan

on implementing a simple, flat, filesystem that can be used to log data and store

raw or precompiled code.

In an ideal world, every university and high school would have a LEGO NXT

lab where students could learn about programming by designing and building

simple robots. I may have to settle for a bit less.

27

Writing C/C++ Modules for Lua

Ralph Steggink and Wim Couwenberg

We use Lua mostly as an embedded scripting environment in our software. Over

the years we have developed many C and C++ libraries that integrate tightly

with Lua code. Since one of us started to use Lua 4.0, a lot has changed in

the newer releases of Lua. The changes that have impacted our way of binding

C/C++ code the most are the introduction of metatables, environments, and a

standard module system.

Even with the now default module system there are still lots of ways that

modules can be written, both in Lua and in C. The module system is carefully

designed to support different approaches to modules on different target plat-

forms. It uses loaders to extend or modify the way in which modules are found

and activated [1]. There really is no preferred way, by design, to write modules.

This gem presents our approach to writing C/C++ modules for Lua. It was

adapted and tweaked over the last couple of years based on experience in several

of our projects. We illustrate our method by wrapping an essential portion

of the well known “libevent” library written by Niels Provos [2].

From its

documentation: “Libevent is an event notification library [. . . ]. The libevent

API provides a mechanism to execute a callback function when a specific event

occurs on a file descriptor or after a timeout has been reached.”

Generic module layout

Compared to our approach in the Lua 4.0 days, one change that stands out in

particular is the shift in focus from C code to Lua code in the implementation

326

27 · Writing C/C++ Modules for Lua

of C/C++ modules. Where for Lua 4.0 we tended to write a complete module

in C, we now only do minimal coding in C to expose a library’s API pretty much

verbatim to Lua and then shape it into a scripting friendly module in Lua code.

A C module for Lua hence consists of two parts: a public Lua script contain-

ing the module’s interface and a private C library that exposes the raw C API

as literally as possible to the Lua script. Here “public” means that only the Lua

script is ever required directly by client code.

The low-level API functions perform little to no argument checking since

they are called only from the accompanying Lua script in a controlled and safe

manner. Of course we must ensure that no low-level API function can “slip” out

of our module and into the open since calling such a function in any other way

than intended might lead to instant disaster.

Hence to make a C module for libevent we create Event.lua and a C-Event.so

(or C-Event.dll) file. Requiring the “Event” module will in fact find, load, and

run the Event.lua script. The Event.lua script will in turn require “C-Event” to

load the private C part and properly setup the entire event module.

The distribution of responsibilities between the Lua and C part of a module

is always roughly the same. The tables below list the typical content for both.

C part:

• Exposes C API to Lua part “as is”.

• Provides garbage collection methods for the module’s object types.

• Can define public constants.

Lua part:

• Defines structured Lua object methods around the low-level API calls.

• Sets up object meta tables.

• Exports object constructors in the module’s namespace.

• Offers structured error handling.

• Checks preconditions on method parameters (correct types and values).

Note that in general the C part does not add any function definitions directly

to the module’s namespace. The public interface is taken care of by the Lua

script.

It is convenient to add public (numerical or string) constant to the

module’s namespace directly from the C part. This saves us from copy pasting

their symbolic definition from a header file to the Lua script. The Event module

defines its public constants in the C part.

A private communication channel between the Lua and C parts is setup by

means of two local tables in the Lua script: A prv table that will hold the low

level C API calls and an aux table that holds anything from the Lua script

that must be available from within the C part. Being local, these tables are

not accessible outside of the module. In particular, the prv table that holds

dangerously low-level C calls is safely tucked away. (Similar to Lua Technical

Note #7 [3].)

327

When the private C part is required from the Lua script it just returns an

initialization function without any other side effects. This function is called

with aux, prv, and the module table _M as parameters. The aux table is set as

the environment of all low-level C API functions and this API is put in the prv

table. Public constants are defined directly in the module table. For the libevent

library, the script Event.lua would typically begin as follows (the exact script is

presented at the end of this gem):

-- save global functions

local require = require

-- setup the module namespace

module "Event"

-- prepare auxiliary and prv tables

local aux, prv = {}, {}

-- load the C initialization function

local initialize = require "C-Event"

-- obtain the private functions and provide access to auxiliary data

-- and to the module namespace

initialize(aux, prv, _M)

The initialization function does three things: It defines public constants from

event.h in the module’s namespace. Then it replaces its own environment with

the provided aux table such that it will be easily accessible to all API functions.

Finally, it places all the low-level API functions in the prv table. Note that by

pushing a C closure on the stack before storing it in ”prv” it will automatically

inherit aux as its environment table. The C part for libevent in Event.c would

roughly look as follows (the exact source file is at the end of this gem):

/* module initialization with "aux", "prv" and module tables */

static int initialize(lua_State *L)

{

< ... setup constants in module table (code omitted) ... >

/* set "aux" table as environment. */

lua_pushvalue(L, 1);

lua_replace(L, LUA_ENVIRONINDEX);

/* put low-level API functions in "prv" table. */

lua_pushvalue(L, 2);

luaL_register(L, NULL, prv_functions);

return 0;

}

/* main entry point for C part. the "C-" prefix is stripped. */

EVENT_API int luaopen_Event(lua_State *L)

{

/* just return the initialize function */

lua_pushcfunction(L, initialize);

return 1;

}

328

27 · Writing C/C++ Modules for Lua

Notice that this C code is really minimalistic.

Not even metatables are

created and manipulated in C: All such interesting stuff will be left to the Lua

script. Having established the generic setup of a module as a pair of a Lua script

and a private C library we will now discuss how a module can organize an API

in more scripting friendly objects.

Objects

Mostly a library’s API is structured in an object-oriented way. This is obvious

for C++ interfaces but is equally true for many C interfaces. For example,

libevent really introduces an “event” object of which the event_set function is a

constructor and the event_add function (among others) is a method. This object-

oriented approach is really convenient from a scripting point of view, so we will

want to structure the event module’s interface in terms of objects.

An object is modeled as a combination of a Lua table and a full userdata.

The userdata part represents an object from the C library while the Lua table

is used to store additional information with this object. For libevent we reserve

space for an event structure in a userdata. A Lua table is used to store the

callback function for an event. The Lua table can be set as the environment of

the userdata to make it easily obtainable from the userdata. Also, the userdata

can be put in the Lua table (by assigning it to some “private” field __udata say),

so that the userdata can be accessed from Lua.

The possibility to set an environment table for a userdata was introduced in

Lua 5.1 and is a great help to associate Lua data with userdata. In Lua 5.0 we

could only do this by maintaining a userdata-to-table mapping. Such a mapping

can still be necessary for some other purposes even in Lua 5.1, as we will see in

the libevent example, where we use it to retrieve a Lua object from a pointer-to-

void callback argument.

When writing a module primarily in Lua we are confronted with the follow-

ing restrictions of environments and metatables:

1. A userdata’s metatable cannot be set in a Lua script.

2. A userdata’s environment cannot be set or obtained in a lua script.

Restriction 1 means that the module’s C code must have access to the metat-

able to construct a userdata object since it cannot be set by the Lua script later

on. In our libevent example we put all metatables in the aux table (the environ-

ment of all C API functions) exactly for this reason.

Restriction 2 hinders the implementation of object methods in the following

sense. We could simply implement an object as a userdata with a metatable and

an environment to store associated data. In this case, each method receives a

userdata instance as their first self parameter. If object methods are written

in Lua (which we aim for), then a method cannot get at the environment ta-

ble of self — an inconvenient situation. Even though we could work our way

around this inconvenience (via a getenv function in the module’s prv table), we

implement objects differently.

329

An object is a plain Lua table that contains the userdata part as its __udata

field. This field is considered “private” although we do not take extra measures

to make it inaccessible. (It is private only by convention.) With this setup

for objects we need two metatables for each object type: one to specify public

methods and properties for an object and another one to specify a garbage

collection function for its userdata part. (Remember that the __gc metamethod

is never called for Lua tables.) The methods and properties metatable can be set

on an object (a Lua table) in the Lua script itself. The userdata metatable must

be set in the C part of the library when the userdata is created. Such metatables

are placed in aux for easy access from C.

Boxing and packing

Objects from C/C++ can be put into a userdata in several ways.

Common

practices are to “box” or “pack” objects into userdata.

Boxing an object in

userdata means that only a pointer to the object is stored in the userdata. A

NULL pointer can then be used to indicate that the object is no longer valid (e.g.,

it is explicitly discarded or got garbage collected). Lua’s io library boxes a file

pointer (FILE*) into a userdata and uses the NULL pointer to indicate that the file

is closed.

Packing an object in userdata means that the storage space for the object is

directly allocated in the userdata itself. Depending on the object type that is

packed, you might need to reserve some extra space for a field that indicates

the object’s validity, much like the NULL pointer does for boxed objects. In our

libevent example we chose to pack the event object instead of boxing it. Note

that even a C++ class instance with a non-trivial constructor and destructor can

be packed by using the placement operator new and an explicit destructor call.

The example in Listing 1 shows how to properly pack an instance of class type T.

(This example omits a “validity” flag.)

Note that Lua’s garbage collector will free the actual space that was reserved

for a userdata. Do not delete a packed object since the object is not allocated by

operator new (the runtime library will most likely assert if you try).

Packing modules into a single library

During development of C modules for Lua it is practical to put each module in

its own library. When deploying final code we find it convenient to distribute

only a single library containing all needed C modules. It took us some time to

realize that Lua 5.1 offers such a facility out of the box via what is called the “all-

in-one loader”. Requiring ”distro.C-Event” will eventually try to locate a library

called distro.so (or distro.dll) containing a function named luaopen_Event.

Multiple modules can be contained in the single library distro.so in this way.

Moreover, providing a separate library distro/C-Event.so in a distro sub-

directory will override module “C-Event” in distro.so since require will locate

330

27 · Writing C/C++ Modules for Lua

// for placement operator new

#include <new>

// create a userdata with a new packed instance of T

static int create_instance(lua_State *L)

{

// reserve space

void *space = lua_newuserdata(L, sizeof(T));

// construct instance in userdata space

// (this does NOT allocate memory)

::new(space) T;

// set object metatable (stored in the environment)

lua_getfield(L, LUA_ENVIRONINDEX, "meta_of_T");

lua_setmetatable(L, -2);

return 1;

}

// destroy a userdata with a packed instance of T

static int destroy_instance(lua_State *L)

{

// assume userdata is first parameter and packed object is valid

T *obj = static_cast<T*>(lua_touserdata(L, 1));

// explicitly destruct object (this does NOT free memory)

obj->~T();

return 0;

}

Listing 1.

specific libraries before resorting to the all-in-one loader. This allows to “patch”

selected modules from a distribution.

A libevent module

Finally, we present some of the techniques that we discussed to make a Lua

module for the libevent library. The code below is fully functional but still

intended as an example. Only a very small part of libevent’s API is included,

but enough to see that it is actually working. Of course, there is room for

lots of improvements and variations. Function and method parameters could

be checked for preconditions and error return codes translated into something

more sensible. However, what this example aims to show is that such things are

really easy to add simply in the module’s Lua script.

First, a small example that uses the Event module to echo stdin with a

timeout of 5 seconds:

331

Example.lua

require "Event"

local function callback(event, fd, event_type)

if event_type == Event.EV_TIMEOUT then

print("timeout")

elseif event_type == Event.EV_READ then

local s = io.read("*l")

if s then

print("you typed: " .. s)

event:add(5)

end

end

end

event = Event.create(1, Event.EV_READ, callback)

event:add(5)

Event.dispatch()

Event.lua

-- save the used globals

local math, require, setmetatable, pcall =

math, require, setmetatable, pcall

-- create the Event module

module "Event"

-- create the auxiliary and private C functions table

local aux, prv = {}, {}

-- open the C library, get the initialization function, and

-- call it with the aux, the prv and the module tables.

local initialize = require "C-Event"

initialize(aux, prv, _M)

-- create a weak table to map callback data to event objects

local events = setmetatable({}, {__mode = "v"})

-- metatable and garbage collector for Event object userdata

local umeta = {}

umeta.__gc = prv.del

-- metatable and method definitions for Event object

local Event = {}

332

27 · Writing C/C++ Modules for Lua

Event.__index = Event

function Event:add(timeout)

local sec, usec

if timeout then

sec = math.floor(timeout)

usec = (timeout % 1)*1e9

end

return prv.add(self.__udata, sec, usec)

end

function Event:del()

return prv.del(self.__udata)

end

-- global functions

function create(fd, event_type, handler)

local udata, ptr = prv.create(fd, event_type)

local event = setmetatable({}, Event)

event.__udata = udata

event.handler = handler

events[ptr] = event

return event

end

function dispatch()

prv.dispatch()

end

-- dispatch event (called from C function)

function aux.handle_event(ptr, fd, event_type)

local event = events[ptr]

if event then

pcall(event.handler, event, fd, event_type)

end

end

-- Event userdata metatable (used by "create" C function)

aux.metatable = umeta

Event.c (compiled into C-Event.so)

#include <sys/types.h>

#include <sys/time.h>

#include "event.h"

333

#include "lua.h"

#include "lauxlib.h"

/* event wrapper to pack into userdata */

struct levent {

struct event event;

lua_State *L;

};

/* The libevent callback handler */

static void handler(int fd, short event_type, void *arg) {

struct levent *ev = (struct levent *)arg;

lua_getfield(ev->L, LUA_ENVIRONINDEX, "handle_event");

lua_pushlightuserdata(ev->L, ev);

lua_pushnumber(ev->L, fd);

lua_pushnumber(ev->L, event_type);

lua_call(ev->L, 3, 0);

}

/* Module-global dispatch */

static int dispatch(lua_State *L) {

event_dispatch();

return 0;

}

/* create an Event userdata */

static int create(lua_State *L) {

/* Lua stack:

fd, event_mask */

int fd = lua_tonumber(L, 1);

int event_type = lua_tonumber(L, 2);

/* pack levent structure in userdata */

struct levent *ev = (struct levent *)lua_newuserdata(L, sizeof(*ev));

ev->L = L;

event_set(&ev->event, fd, event_type, handler, ev);

/* set userdata metatable (from aux table) */

lua_getfield(L, LUA_ENVIRONINDEX, "metatable");

lua_setmetatable(L, -2);

/* return userdata and address of levent (to index event objects) */

lua_pushlightuserdata(L, ev);

return 2;

}

334

27 · Writing C/C++ Modules for Lua

/* bind the lua add function to the libevent add function */

static int add(lua_State *L) {

/* Lua stack:

userdata [, seconds, useconds] */

int res;

struct timeval t, *pt = NULL;

struct levent *ev = (struct levent *)lua_touserdata(L, 1);

/* timeout specified? */

if (lua_type(L, 2) == LUA_TNUMBER) {

t.tv_sec = lua_tointeger(L, 2);

t.tv_usec = lua_tointeger(L, 3);

pt = &t;

}

res = event_add(&ev->event, pt);

lua_pushinteger(L, res);

return 1;

}

/* delete event.

doubles as garbage collection function */

static int del(lua_State *L) {

/* Lua stack:

userdata */

struct levent *ev = (struct levent*)lua_touserdata(L, 1);

int res = event_del(&ev->event);

lua_pushinteger(L, res);

return 1;

}

/* definition of a constant name/value pair */

struct constant {

const char * name;

int value;

};

/* constants */

static const struct constant constants[] = {

{"EV_TIMEOUT", EV_TIMEOUT},

{"EV_READ", EV_READ},

{"EV_WRITE", EV_WRITE},

{"EV_SIGNAL", EV_SIGNAL},

{"EV_PERSIST", EV_PERSIST},

{NULL, 0}

};

335

/* private functions */

static const luaL_Reg prv[] = {

{"dispatch", dispatch},

{"create", create},

{"add", add},

{"del", del},

{NULL, NULL}

};

/* initialize the Event module */

static int initialize(lua_State *L) {

/* Lua stack:

aux table, prv table, module table */

/* define constants in module namespace */

int index;

for (index = 0; constants[index].name != NULL; ++index) {

lua_pushinteger(L, constants[index].value);

lua_setfield(L, 3, constants[index].name);

}

/* set the "aux" table as environment */

lua_pushvalue(L, 1);

lua_replace(L, LUA_ENVIRONINDEX);

/* register prv functions */

lua_pushvalue(L, 2);

luaL_register(L, NULL, prv);

/* initialize event library */

event_init();

return 0;

}

/* entry point for C-Event module */

EVENT_API int luaopen_Event(lua_State *L) {

lua_pushcfunction(L, initialize);

return 1;

}

336

27 · Writing C/C++ Modules for Lua

References

[1] Roberto Ierusalimschy, Luiz Henrique de Figueiredo, Waldemar Celes,

“Lua 5.1 Reference Manual”, Lua.org, 2006.

[2] Niels Provos, “libevent – an event notification library”.

http://www.monkey.org/∼provos/libevent

[3] Roberto Ierusalimschy, “Technical Note 7 – Modules & Packages”.

http://www.lua.org/notes/ltn007.html

28

Interpreted C Modules

J ér ôme Vuarand

Lua 5.1 provides a flexible and powerful module mechanism. It can load two

types of modules: Lua modules, which are written in Lua, and binary modules,

which are written in any compilable language that can produce shared libraries.

Through this mechanism it is possible extend Lua in many ways, making it a

truly extensible language, for use as a general scripting tool.

However in certain circumstances these two extension mechanisms may not

be enough. Hopefully, the Lua module mechanism has been carefully written

and is itself extensible thanks to a searcher concept. The stock Lua interpreter

comes with a few searchers that implement the Lua 5.1 module system. But

it is possible to create additional searchers that will inject code inside the Lua

interpreter state in whatever way may be needed.

In this article I will introduce a method to extend the Lua module mechanism

to other kinds of modules. As an example, I will show how to support modules

written in uncompiled C, with the help of a tiny embeddable C compiler, TCC (for

Tiny C Compiler). This mechanism will allow us to load uncompiled C modules,

and to compile them on the fly. I will first explain how the searcher mechanism of

Lua works in detail, and how to hook into it. I will present TCC, and especially

libtcc, which is a library that is able to compile C code and relocate it in the

current executable for immediate use. We will finally see how to create a small

binding library for TCC that injects its C compilation ability in the Lua module

framework.

338

28 · Interpreted C Modules

The Lua module system

The Lua module system is based on two global functions, require and module.

Both use the package table to store information concerning already loaded mod-

ules and how to load additional modules. The basic usage of the module system

consists in loading Lua files or shared libraries to expose a table containing func-

tions to the script being executed. But these functions are much more powerful

than that. require has been designed with flexibility in mind, and its behaviour

can be totally configured. While module sticks closely to the module system and

in fact defines it, require is just a wrapper around a searcher concept, and what

exactly a searcher does is not limited. require existed before the Lua 5.1 mod-

ule system was introduced, and it has been carefully redesigned to keep its old

behaviour while allowing module to do its job.

require, searchers, and loaders

require defines a protocol to locate and execute external code. require is called

with a single parameter, which usually represents the module name ( modname).

For the moment try to consider a module as an abstract concept; it may be any

form of external code that can be executed. Actually, loading a module means

executing the module loader. For Lua modules the loader is the module main

chunk. For binary modules the loader is a function named luaopen_ modname.

For other types of modules it’s up to the searcher designer to determine what

the loader is.

require will first look in package.loaded to see if the module is already

loaded. This allows modules to control whether they can be loaded twice or

not. If a module loader sets package.loaded[ modname] to a value resolving to

true, the module won’t be re-loadable. If it sets that entry to false or nil, the

module can reloaded. Manually resetting that value allows to force a module

reloading next time it is require’d.

If require determines that it must load the module, it will look for the module

loader and execute it. To find a loader, require uses searchers. Searchers are

located in the package.loaders array: they are functions. require will call

each searcher, starting at index 1, with the module name as only parameter.

A searcher can have three possible behaviours. If it doesn’t find the module,

it returns a string that will be displayed by require if no searcher finds the

module. If it finds the module without any problem, it returns the module loader

(a function). Finally, if the searcher finds the module but is not able to extract

its loader, it can throw an error. This is the case for example when a Lua module

contains syntax errors. This error will propagate outside require.

If require has found a loader, it executes it with the module name as sin-

gle parameter, in an unprotected call. That way, if the loaders throws an er-

ror it is propagated outside of require, for example when a Lua module con-

tains a runtime error. The first return value of the module loader is stored in

package.loaded[ modname]. If the module loader returns no value and has not

339

set package.loaded[ modname], true is used instead (that means that by de-

fault a module is considered non-re-loadable). Finally, whether the module was

loaded or not, the content of package.loaded[ modname] is returned by require.

In stock Lua there are four predefined searchers. The first one, preload,

looks for loaders in the package.preload table, with the module name as a

key. The second one, Lua, looks for Lua source files with the package.path

string. If it finds a file it compiles it as a Lua chunk and that chunk is the

loader. The third one, C, looks for shared libraries with the package.cpath

string. If it finds one corresponding to the module name, it looks for a function

called luaopen_ modname in it, and that function is the loader. Finally, the last

searcher, called Croot in Lua source code or the all-in-one searcher in the Lua

manual, will also look for a shared library with the package.cpath string. But

instead of using the module name to locate the shared library, it will only use

the first component of it (everything before the first dot). To locate the loader

inside the library, it uses the full module name. These searchers and their

configuration strings (package.path and package.cpath) are documented in the

Lua manual.

TCC

Tiny C Compiler

TCC is a C compiler targeted for x86 platforms written by Fabrice Bellard (of

ffmpeg and QEMU fame). It is very small and very fast. It is so fast that it is

used as a JIT compiler to interpret C programs, and that’s precisely the feature

we’ll be using in this gem. Originally TCC was derived from OTCC, which is

aimed to be the smallest self-compiling C compiler. OTCC was not capable of

compiling full C99, only a subset of it, but that subset was C99 compliant and

enough to build itself. TCC has kept that minimalistic approach while being

much more usable in a production environment.

TCC is heading toward full ISO C99 compliance. It does have some exten-

sions, but not as numerous as GCC’s. It has almost no support for older versions

of C not covered by C99. Also it has no support for C++, so C++ programmers

used to GCC extensions must take care to program strictly in C. But past these

little restrictions, TCC can compile most C code without any problem and very

quickly (for example it can boot a typical Linux 2.4 from sources in less than 15

seconds on an average PC).

libtcc

TCC can be used as a standalone compiler, but it is internally built around a

compiling library, libtcc. That library can be used from external projects, to

compile and link C code. But libtcc is also able to relocate dynamically generated

code into memory, and to return pointers to functions and other symbols in that

340

28 · Interpreted C Modules

relocated code. That’s the way TCC is used as a C interpreter, and that’s the

feature we’re going to use to create Lua modules at runtime from C source code.

Libtcc follows classic C compilation steps. First you can add include paths,

library paths, and predefined symbols. Then each source file is compiled. All

compiled files are finally linked between each other and with external libraries

to produce the output binary. TCC can link to dynamic libraries too, but on some

platforms it may require a specific import library (read the TCC manual for more

information). The output binary can be saved into an executable file (ELF on

Linux, PE on Windows), or it can stay in memory and be directly executed. In

the latter approach, TCC can retrieve pointers to symbols in the binary and give

them to the calling code. If the symbol is a function, it can be directly executed

as a function pointer.

Special care must be taken when accessing symbols in the relocated binary.

Libtcc has no way to ensure that the retrieved symbol is of a specified type, so

you have to carefully handle the pointers returned and cast them to the proper

type. This is especially important for functions pointers, since calling a function

with an incorrect number of parameters or with the wrong calling convention

can invalidate the stack pointer and lead to unpredictable results, among which

the program crash is the least problematic. Here the fact that Lua uses a single

function prototype almost everywhere will be very handy and will avoid many

complications.

Another problem must be handled: data execution prevention hardware

features. On some platforms, which includes modern x86 derivatives, there

are safety mechanisms built in the memory hardware to separate code from

data in memory. Libtcc output is considered as data (it is written by current

process) and code (we want to execute it), so attempting to call functions in TCC

relocated code is considered by the underlying hardware as a violation of the

data/code separation and may interrupt the whole process. In your application

you must make sure that this behaviour will be allowed (with its potential

security implications). This is handled neither in libtcc nor in the libtcc binding

presented here, since it’s a very hardware and OS specific issue.

TCC binding

The TCC binding I’m going to expose here was originally based on the one

written by Javier Guerra. It has been widely rewritten and extended to be

used as a Lua searcher. This binding is split in two parts. The first part, the

luatcc module presented in this section, is a simple binding to libtcc and allows

to compile and execute C code. The second part, the luatcc.loader module, is a

module searcher that locates C source files and compiles them as Lua modules.

The TCC binding is articulated around a context or state concept. A context

is like an instance of the compiler. It has its own paths, you can add several

source files, declare several libraries, and it can produce a single output binary.

However, you can access several symbols in that binary. To create a context, just

call the luatcc.new function. The module source code is not of much interest:

341

it’s just a simple C library binding, so I won’t explain it here; source files are

self-explanatory. Here is a basic example that extracts a function called hello

from a C source string:

local luatcc = require ’luatcc’

local context = luatcc.new()

context:compile([[

#include <lua.h>

int hello(lua_State* L)

{

lua_pushstring(L, "Hello World!");

return 1;

}

]])

context:relocate()

local hello = context:get_symbol("hello")

print(hello())

As you can see, you must call the methods compile, relocate, and get_symbol

of the TCC context object. compile accepts as a second parameter the chunk

name, which can be useful when you have several source files and an error oc-

curs. Here we don’t add include paths; TCC will use its predefined ones to locate

lua.h. These predefined paths are defined at TCC compilation time.

TCC searcher

The TCC searcher is very simple. It mimics Lua and C searchers. We will

examine its source code and comment each section.

The module header

module(..., package.seeall)

local luatcc = require("luatcc")

local function new_context()

local context = luatcc.new()

--context:add_include_path("some/path/to/header/files")

--context:add_library_path("some/path/to/library/files")

return context

end

local DIRSEP = ’/’

We first declare the searcher module itself, with the name provided by require.

We then require luatcc as a dependency, since we will use it to compile the inter-

preted C module. The new_context function is used to create a luatcc context. It

exists primarily to ease addition of searcher-wide compilation parameters, such

as standard include paths or library paths. DIRSEP is the directory separator.

You may have to modify it to fit your OS.

342

28 · Interpreted C Modules

The searcher function

The searcher itself is a function, which takes as its first and only parameter the

module name.

local function search(modulename)

-- Read source

local filename

local file

local errmsg = ""

for path in package.tccpath:gmatch"[^;]+" do

filename = path:gsub("%?", (modulename:gsub("%.", DIRSEP)))

file = io.open(filename)

if file then

break

end

errmsg = errmsg.."\n\tno file ’"..filename.."’"

end

if not file then

return errmsg

end

The first action of the searcher will be to locate the C source file for the module

we are trying to load. To do that we use the content of the package.tccpath

variable in the same way that the Lua and C searchers use package.path and

package.cpath respectively. Each tested path which doesn’t match is added to

an error message. If the module is not found, that filename list is returned to

require in a string. The format is the same as the one used by the Lua and C

searchers: each path is prefixed with a new line and a tab character.

local source = assert(file:read"*a")

The content of the module source file is read entirely. That way we will be able

to locate pragma directives inside the file (see below). It is not the most efficient

way to load the file, especially if the source file is very big, but that is left as an

optimization for future versions.

-- Get luatcc pragma commands

local commands = {}

for command,argstr in source:gmatch"luatcc[%s]*([a-z_]*)%(([^)]*)%)"

do

commands[command] = commands[command] or {}

local args = {}

for arg in argstr:gmatch"[^,]+" do

table.insert(args, arg)

end

table.insert(commands[command], args)

end

343

Since we have no way to add compilation options specific to a module, the TCC

searcher will read some pragma directives inside the module source file. An

alternative method would have been to load a second file containing those pa-

rameters. With pragma directives we can keep the module configuration atomic.

Also it does not prevent us from adding a second optional file containing other

configuration parameters (for example if we need platform-specific parameters).

Each directive has the form:

#pragma luatcc commandname(commandparameters)

The searcher will load all such found commands in the commands table. Each

entry has the command name as key and a array as value. That array contains

arrays, each containing the parameters of command. For example if you want

to access the second parameter of the first foo command, you would access it

through commands["foo"][1][2]. This system gives us an extensible way to add

commands. All unused commands will be simply ignored. For the moment, the

only command used is use_library, but we could add commands for each luatcc

API function.

-- Interpret pragma commands

--- use_library

local libdeps = {}

if commands.use_library then

for _,args in ipairs(commands.use_library) do

local libdep = args[1]

table.insert(libdeps, libdep)

end

end

This section of the searcher is the interpretation of the pragma directives. As

mentioned above, the only directive used at the moment is use_library. Here

we simply build a list of libraries using the first parameter of each use_library

command.

Next we create a luatcc context and allocate three local variables. Most luatcc

functions return a boolean, success, and eventually an error message, errmsg,

if the boolean is false. Also, for more safety, we will call these functions through

pcall and the first return value of pcall, indicating the call success, will be

stored in result.

local context = new_context()

-- Compile file

local result,success,errmsg =

pcall(context.compile, context, source, filename)

if not result then

error("error loading module ’"..modulename.."’ from file ’"..

filename.."’:\n\t"..success, 0)

end

assert(success, errmsg)

344

28 · Interpreted C Modules

The first step is the compilation of the file. We use filename as the chunk

name passed to TCC since the source has been directly read from the file without

modification. On error we will throw a Lua error. As mentioned before, when

describing the searcher behaviours, a searcher has to return a string if it doesn’t

find a module, but it can throw an error if it finds the module but cannot load

it. This is the behaviour used by the Lua and C searchers, and we simply mimic

it here. Our first error case is when the pcall fails. In that case we throw

a simple error message containing the pcall error message, stored in pcall

second return value, here success. If the pcall went right but the compilation

failed, the assert will throw the appropriate error message.

-- Add libraries

for _,libdep in ipairs(libdeps) do

result,success,errmsg = pcall(context.add_library, context, libdep)

if not result then

error("error loading module ’"..modulename.."’ from file ’"..

filename.."’:\n\t"..success, 0)

end

assert(success, errmsg)

end

This step is similar to compilation. We simply call add_library for each library

declared in the module pragma directives. The error mechanism is the same as

before.

-- Relocate binary code

result,success,errmsg = pcall(context.relocate, context)

if not result then

error("error loading module ’"..modulename.."’ from file ’"..

filename.."’:\n\t"..success, 0)

end

assert(success, errmsg)

Here again we simply call the relocate method of the luatcc context, with the

same error handling as before.

-- Extract symbol

local chunk

result,chunk,errmsg = pcall(context.get_symbol, context,

"luaopen_"..string.gsub(modulename, "%.", "_"))

if not result then

error("error loading module ’"..modulename.."’ from file ’"..

filename.."’:\n\t"..chunk, 0)

end

assert(chunk, errmsg)

return chunk

end

345

The last step of the loading mechanism is a bit different: instead of a success

boolean value, get_symbol returns a function, which is the module loader. If

all went well, the searcher simply returns the loader to require, which is

responsible to execute it.

Installing the searcher

Finally, the searcher module contains two last parts.

local priority

if type(package.tccpriority)==’number’ and package.tccpriority>=1 then

priority = math.min(#package.loaders+1, package.tccpriority)

end

table.insert(package.loaders, search, priority)

The first one will add the interpreted C module searcher to the searcher list

used by require: package.loaders. We simply use table.insert to add the

searcher to the searcher list. You can specify the searcher priority through the

global variable package.tccpriority. If you don’t specify it, it will default to

nil, letting table.insert add the searcher at the end of the list. This will give

interpreted C modules the lowest priority when several modules of different

type have the same name. To change that priority you can simply assign a

positive integer value to package.tccpriority before loading the tcc.loader

module (the integer 1 is the highest priority).

package.tccpath = "./?.c"

The last line is simply the initialization of the path parameter used by the

searcher to locate the C source files. Here we look for modules in the current

directory, but that path could be extended to include standard system-wide

paths, just like package.path or package.cpath.

Conclusion

The main purpose of this gem was to show you how easy it is to add a completely

new kind of module to Lua. With only a few tens of lines, you can convert an

existing binding to some other form of programming into a module searcher. My

example, which is able to load uncompiled C modules, is just an example. With

the same principle, you could load Java classes or their .NET equivalents. You

could access some web services, just by loading some interface definition file.

You could load modules from Python, TCL, or Ruby.

There is also another field of development for more Lua searchers. With

new module searchers, you could simply change the way the searcher locates

the code.

Instead of loading the modules from some directory according to

package.path, you could look for the modules online, either in some company-

specific intranet or in the wild internet. You could load the modules from a

346

28 · Interpreted C Modules

compressed form, just unzipping it before actual loading. You could locate the

module in some big archive which contains all the data of your game. You could

add a versioning scheme just like the smart one present in Ruby Gems. You

could decrypt the module on the fly, or simply check it against a hash key before

loading it.

The possibilities are endless. The Lua 5.1 module system is a mechanism

that makes Lua module distribution and management much simpler and clearer,

providing a standard. But that standard has been cleverly designed and it em-

powers programmers in such a way that it makes Lua interoperability with

other computing systems much easier than it was before.

Document Outline

- Contents

- --- Programming Techniques

- Per-Thread Library Context

- Performance Tips

- Vardump

- Serialization with Pluto

- Abstractions for LuaSQL

- Boostrapping a Forth in 40 Lines Code

- Large-Scale Change using Metatables

- --- Design Techniques

- MVC Web Development with Kepler

- Filters, Sources, Sinks & Pumps

- Lua as Protocol Language

- Lua Script Packaging

- Objects Lua-Style

- Exceptions

- --- Algorithms & Data Structures

- Word Ladders

- Building Data Structures & Iterators

- Primer of Scientific Computing

- Complex Structured Data Input

- Common Data Structures

- Tic-Tac-Toe & Minimax Decision

- --- Game Programming

- Game & Tool Creation

- Event System for Script-driven Games

- Game Programming

- Efficient Game Scripting Engine

- --- Embedding & Extending

- Enhanced Coroutines

- Lua in Pascal

- Porting Lua to Microcontroller

- C/C++ Modules for Lua

- Interpreted C Modules

Table of Contents

Contents

--- Programming Techniques

Per-Thread Library Context

Performance Tips

Vardump

Serialization with Pluto

Abstractions for LuaSQL

Boostrapping a Forth in 40 Lines Code

Large-Scale Change using Metatables

--- Design Techniques

MVC Web Development with Kepler

Filters, Sources, Sinks & Pumps

Lua as Protocol Language

Lua Script Packaging

Objects Lua-Style

Exceptions

--- Algorithms & Data Structures

Word Ladders

Building Data Structures & Iterators

Primer of Scientiﬁc Computing

Complex Structured Data Input

Common Data Structures

Tic-Tac-Toe & Minimax Decision

--- Game Programming

Game & Tool Creation

Event System for Script-driven Games

Game Programming

Efﬁcient Game Scripting Engine

--- Embedding & Extending

Enhanced Coroutines

Lua in Pascal

Porting Lua to Microcontroller

C/C++ Modules for Lua

Interpreted C Modules