Tony Finch – blog

I spent Thursday and Friday last week at FiBL, an organic agriculture research centre, in Frick, a small town between Basel and Zurich. I was there for the Lua Workshop 2011. There were a bit more than 40 attendees, of which two were women, with a good spread of ages from about 20-65. Some themes came up repeatedly: live code upgrades, distributed systems, Lua internals, LaTeX Beamer slides, and really good demos. Here's my summary of the talks. The slides should appear on the workshop website in due course.

The first day started with Alexander Gladysh, who talked about his efforts to find a good strategy for implementing declarative embedded domain specific languages in Lua. Lua was originally designed to be good as a data description language, so it has quite nice lightweight syntax for EDSLs. Alexander found that implementing the infrastructure for EDSLs often got bogged down in spaghetti code and boilerplate. In his talk he explained how he improved the situation using common data structuring conventions and traversal algorithms. It was a very high bandwidth talk with over 60 slides (most containing code) in less than 40 minutes. I need to go back to read the slides at leisure.

Next was Fabien Fleutot from Sierra Wireless. They make embedded system monitoring equipment, for checking the performance and status of things like wind turbines and street lights. The communication between these in-the-field assets and the back-end monitoring systems is usually via GPRS, so it's important for them to minimise bandwidth usage, and they mostly have to rely on the assets to initiate outgoing connections since their connections are often down or behind NAT. Their monitoring devices run embedded Lua and make heavy use of coroutines, with a nice IPC framework and scheduler. Fabien described how this makes it easier for them to do in-field upgrades and quickly adapt their products for new customers. They're planning to release their framework as open source.

After coffee, Gaspard Bucher talked about and showed videos of his exploits using computers to support live dance and music performance, including turning his body into a MIDI device so he could use music to make his muscles contract involuntarily, and attaching motion sensors to dancers so their movements can be translated into sounds. The latter required a lot of custom hardware and research into using support vector machines to distinguish different kinds of movement. He explained how the pressures of performance seriously damaged the maintainability of his code. He's now working on Lubyk which is a Lua system with many bundled libraries, intended to provide a basis for his computer-assisted performances. It is a distributed development environmnet, using Bonjour to tie the components together, with a graphical editor to link the parts and allow you to live-edit the code on remote systems. Very swish, but still a bit raw.

He was followed by Roberto Ierusalimschy, the principal author of Lua. He talked about some of the design considerations the Lua team use when evolving the language, and discussed some of the areas that are particularly troublesome - table length, varargs, bitwise operations on floating point numbers.

After lunch Wim Couwenberg of Océ talked about using Lua as a diagnostic tool for the controller software for their large printers. The controller software itself is a set of C++ processes running on a Windows PC attached to the printer. They have an IPC framework with an XML-based IDL, which includes extensive logging of the activity of the system as a whole. Wim replaced their log processing tools with a much smaller Lua script which translates an IDL specification to Lua and runs it to process a log. He extended the XML IDL with Lua snippets that can verify the interface requirements have been followed. This improved their ability to debug the system so much that they are using more IPC to get better visibility into the system. A nice success using just base Lua without add-ons.

Next was Valerio Schiavoni of Neuchatel University, talking about the "Splay" system for making distributed application research easier. It runs on PlanetLab and provides some higher level facilities for distributed IPC, and makes it easy to push Lua code out to a selection of machines, run it in a sanbox, collect the data, and produce visualizations of the results. He did a very smart live demo of a virus propagation simulation on 200 machines around the world.

After coffee was Peter Cawley, an undergraduate at Oxford University. He talked about his work finding and exploiting holes in Lua's type system using cleverly crafted bytecode. By default Lua is compiled without the assertions that do thorough type checking of values passed by C embedding/extending code to the Lua API. This means that (for instance) you can make the low-level table index function try to interpret a string as a table. The bytecode interpreter is mostly typesafe but there are a few holes which can be exploited, which Peter described in some detail. Lua 5.1 has a bytecode verifier which is supposed to make it safe to load untrusted bytecode, but because of Peter's exploits, Lua 5.2 has no verifier and instead makes it easy to restrict loaded code to source only, which is safe provided the library is suitably restricted. Peter has gone on to write a couple of replacement bytecode verifiers. The first is based on a type inference algorithm, but this turned out to be too slow to be practical. The second is based on a simpler analysis of which stack slots are variables or temporaries or unused, and verifying that they are used consistently.

The last talk on Thursday was Erik Hougaard talking about the series of robots called "Crazy Ivan" which he built with his brother-in-law. They compete in the Danish Technical University RoboCup, which is an annual obstacle course for autonomous robots. Erik compared the structure of the competition to a Formula 1 race, in that the competitors have time to set up their robots to suit the course, and there is a qualification stage before the final competition. Crazy Ivan's software is customized during the competition to follow the specific course and win points by performing tricky tasks. They use Lua to make it faster to adapt the software, in particular the high level plan of the course and tasks. As well as videos of previous competition performances, Erik did a live demo of the current version of Crazy Ivan, whose control computer is a complete PC running Windows XP with on-board copies of Visual Studio and PIC programming tools for the robot's peripherals. He ran a remote desktop session from his laptop to the robot to show the computer vision algorithms it uses to follow the track and spot golf balls etc. Lots of fun!

On Thursday evening we had an organic wine tasting at FiBL's winery, with a talk on the diffrences between organic and traditional viticulture. After that we went for a meal at a local restaurant.

The first speaker the next morning was Gaetan Morice who also works at Sierra Wireless. He was speaking about the Lua development tools they are putting together. They are keen to open up their devices for their customers to program, for which an attractive development environment is necessary. They have put together a lot of open source components, starting with Eclipse and adding JnLua, MetaLua, LuaDoc, and a lot of their own code to produce a pretty swish IDE. They are planning to release it as open source under the umbrella of the Eclipse Foundation, since they want to avoid the appearance of trying to lock their cusomers into a proprietary system. His live demo included lots of nice features like code completion, highlighting instances of the same variable, and their debugger.

The second talk of the morning was Roberto again, filling in for another speaker who was unfortunately ill. This talk was about new features in Lua 5.2, including more flexible coroutine yields, emergency garbage collection when memory is low, ephemeron tables to avoid some memory leaks, lexically scoped global tables, light C functions, the bitwise operation library, and the goto statement. The talk covered the rationale for the features and the implementation challenges for the trickier ones. He's always pleased when an improvement allows him to delete code.

After coffee, Henning Diedrich talked about d'Arc, which is a performance oriented extension to the Lua API. He bypassed the standard API in order to make his JSON serializer faster, since it was a bottleneck in his game engine. The original PUC Rio implementation of Lua and LuaJIT are similar enough that Henning could make d'Arc work with both of them. The main feature of d'Arc is a faster table traversal function. In the standard Lua API tables are traversed using the lua_next() function which has to re-check its arguments and re-find its iteration point on every call. darc_traverse() inverts control so it keeps state and calls down to a fold function for each table element. There was some discussion between several people about the wisdom of bypassing the API and whether Lua's tables were keeping their performance promises.

The next talk was by Ashwin Hirschi, talking about the Reflexis Flow framework for web applications. This is a declarative EDSL for describing a workflow between dynamic web pages, intended to be accessible to non-programmers. Ashwin did an impressive live coding demo putting together a little web app with a form taking a name, height, and weight, and calculating the person's BMI. He then added some bells and whistles including a Google Gauge widget to make the BMI number look more pretty, and a database search for looking up names from a list of workshop attendees. It can also draw a graph to visualise the traversals between pages. He then showed off the traversal graph for a larger application, which is a meta-recursive web-based editor for Reflexis Flow applications.

After lunch, Patrick Rapin of Olivetti described LuaDura, which they use for diagnostics and servicing of their printers. Their firmware is written in C++, and they have a nice reflection system that allows you to make arbitrary virtual method calls via RPC over the service port. LuaDura ldownloads a description of all the printer's functionality from the service port into Lua on a controlling computer. It includes some really nice readline support so you can tab-complete over the printer firmware API. It's also multi-threaded, and can control multiple printers at once, or run in the printer firmware itself. LuaDura is used a lot for development of printers, and in production for things like flashing the printer's serial number. Patrick finished his talk by abusing a printer, making its scanner LEDs shine with random colours, and making the paper and head motors play the William Tell Overture in two tone polyphony.

Then Francesco Santorini from the University of Basel Hospital talked about using Lua for processing images from MRI scanners. The development environment for their Siemens MRI scanner does not encourage experimentation or ad-hoc image processing pipelines. Their practice before Francesco developed IceLuva was to transport images by sneakernet to a system running MATLAB. After plugging Lua into the Siemens software as a module, they were able to easily translate their MATLAB scripts into Lua and fit them neatly into the existing system, displaying the output of the pipeline on the scanner's console. Francesco was sadly unable to bring the scanner with him to do a live demo, but did have a recording of the scanner playing Smoke on the Water.

After coffee, Peter Odding spoke about Lua/APR. He started work on binding the Apache Portable Runtime library to Lua in 2007, at which time he did not know how to program in C, but approached the project with the attitude, "how hard can it be?". After learning about manaual memory management, segmentation faults, and Windows calling conventions, he released Lua/APR in 2010. One of the problems he had to overcome is the mismatch between APR's pool-based allocation strategy and Lua's garbage collector; his solution is to allocate a pool for each APR object, which is simple if not all that efficient. Lua/APR includes some shaky prototype multithreading support, which is still under development. Peter wants to get it to the stage of being able to implement a performant web server in Lua.

Finally, the last talk was given by one of the organizers of the workshop, Marc Balmer. He talked about his project to run Lua inside the NetBSD kernel. His goals are to use it for rapid prototyping of drivers, including reverse engineering undocumented hardware. Other possibilities include scripted autoconfiguration. At the moment he has the infrastructure for creating Lua states in the kernel, autoloading Lua modules, and running scripts using this context. His next goal is to write a working driver for some simple watchdog hardware. One other significant area of difficulty is multi-processor safety. He has promised to present a talk at FOSDEM in February on his experiences writing drivers in Lua.