If you write Lua C modules, there is a corner case surrounding finalizers (ie. __gc metamethods) that you should be aware of. I haven’t run into a public and prominent warning about this, so I decided to write it up in this entry. I only discovered it myself through trial and error.

Lua provides the __gc metamethod for releasing resources associated with a userdata. It is commonly used in C modules to free any C-level objects or resources that were allocated by a type.

#include "lauxlib.h"
#include "mytype.h"

#if LUA_VERSION_NUM == 501
#define setfuncs(L, l) luaL_register(L, NULL, l)
#define luaL_setmetatable(L, name) \
  luaL_getmetatable(L, name); \
  lua_setmetatable(L, -2)
#elif LUA_VERSION_NUM == 502
#define setfuncs(L, l) luaL_setfuncs(L, l, 0)
#endif

static const char *MYTYPE = "mytype";

typedef struct {
  mytype_t *val;
} MyTypeWrapper;

static int newobj(lua_State *L) {
  MyTypeWrapper *obj = lua_newuserdata(L, sizeof(MyTypeWrapper));
  obj->val = mytype_new();
  luaL_setmetatable(L, MYTYPE);
  return 1;
}

// BROKEN -- DON'T DO THIS!
static int call(lua_State *L) {
  MyTypeWrapper *obj = lual_checkudata(L, 1, MYTYPE);
  mytype_call(obj->val);
  return 0;
}

// BROKEN -- DON'T DO THIS!
static int gc(lua_State *L) {
  MyTypeWrapper *obj = lual_checkudata(L, 1, MYTYPE);
  mytype_free(obj->val);
  return 0;
}

static const struct luaL_Reg mm[] = {
  {"__gc", gc},
  {"__call", call},
  {NULL, NULL}
};

int luaopen_ext(lua_State *L) {
  luaL_newmetatable(L, mytype);
  setfuncs(L, mm);
  lua_pushcfunction(L, &newobj);
  return 1;
}

It turns out this will work nearly all of the time. But there is one very unusual corner case that can reliably cause gc() to run before call()! This program can trigger this behavior on both Lua 5.1 and Lua 5.2 (and LuaJIT):

local ext = require "ext"

if _VERSION >= 'Lua 5.2' then
  function defer(fn)
    setmetatable({}, { __gc = fn })
  end
else
  function defer(fn)
    getmetatable(newproxy(true)).__gc = fn
  end
end

local y = {}
defer(function() y[1]() end)
y[1] = ext()

Basically, any Lua code that runs inside a __gc metamethod can get access to a userdata that has already been finalized! This can crash your C extension if you don’t handle this case.

There are two main solutions to this problem:

  1. Clear the userdata’s metatable inside the finalizer. This will ensure that the userdata fails any subsequent luaL_checkudata() check later. The downside is that the error message for trying to call a method on a finalized value will be very unhelpful (something like “attempt to index field ‘?’ (a user data value)”)
  2. Set a “finalized” flag inside the finalizer, and check this flag right after calling luaL_checkudata(). For example, you could set the pointer to NULL and check for this. This gives you the benefit of being able to return a custom error message like “attempted to call into dead object.”

Here is an example of what the second solution might look like:

mytype *mytype_check(lua_State *L, int narg) {
  mytype *obj = luaL_checkudata(L, narg, MYTYPE);
  if (!obj->val) luaL_error(L, "called into dead object");
  return obj;
}

static int call(lua_State *L) {
  MyTypeWrapper *obj = mytype_check(L, 1);
  mytype_call(obj->val);
  return 0;
}

static int gc(lua_State *L) {
  mytype *obj = mytype_check(L, 1);
  mytype_free(obj->val);
  // The critical step that will prevent us from allowing
  // a call into a dead object.
  obj->val = NULL;
  return 0;
}

For a bit more discussion about this, see this lua-l thread where I raised the issue.

The moral of the story is: anytime you are using __gc from a C module, you need to handle the case where the finalizer gets called before other methods. Otherwise a user could SEGV your module.