I myself, always prefer to use some form of chaining instead of open addressing. That’s because with open-addressing, it’s harder to tell when an item was removed or does not exist at all, and as a result the hash becomes much more flimsy. Maybe there’s some magic way to achieve it, but I’m still ignorant of it.

For example, I want pbstream to be usable in embedded system. Here’s an example of a guy who’s in a situation I want to accommodate: GPB on non-Linux, non-Windows OS?.

I hear what you’re saying wrt. just using Python’s dict. Although accommodating languages like Python, Ruby, Lua, etc. is one of my goals, I also want this library to perform well standalone, in cases where there is no other hashtable implementation available. Also, of my two hashtable use cases, the int->void* lookup that happens in the critical path of parsing is the more important one to me by far, and this happens at a layer below where you’d interface something like Python or Ruby.

It makes me really happy that you’re independently interested in Gazelle! pbstream and Gazelle are actually soon to be intertwined: Gazelle needs a bytecode format and an AST serialization format (I mean to switch to protobufs from LLVM bitcode), and pbstream needs a way to parse .proto files. pbstream->Gazelle won’t be a hard dependency though — you’ll be able to load .proto definitions in binary form if you want.

I’m afraid I’m drawing a complete blank on your friend Matthew O’Connor though, and I’ve never been associated with Pivotal. Is there something I’m forgetting?

But if you’re calling this hash lookup from Python anyway, it’s silly to worry about any of the micro-efficiencies you’re talking about. Just use Python’s dict; you might be able to do epsilon better, but that will be swallowed up by the bytecode dispatch overhead and function call and return (on the order of 30-300 instructions per bytecode).

BTW, you left out a slight variation of linear probing: you make each hash bucket big enough to contain more than one item. Of course this doesn’t save you from having to deal with the case for where the entire hash bucket fills up, when you either need to use one of the other strategies to figure out where the next item goes, or expand the hash table.

@Kragen: I should have said “lookup speed trumps all within the constraint that I don’t know the table at compile time. One of the things I’m trying to deliver is a Python API that doesn’t require you to build a C extension for each different proto type you want to parse (ick!).

I’m considering adding a JIT using LLVM or similar sometime in the future — if I do that then I could do what you say.

A really straightforward way to implement this strategy (for integer keys) is to generate a C file with a switch statement in it, use GCC to compile it to a shared library, and then dlopen the shared library in order to call into it. (Alternatively you could use LLVM or GNU Lightning.)

The nearest hashing equivalent is “perfect hashing”, where you search for a hash function that hashes each key into a distinct bucket in a minimal-size hash table. gperf is a widely-available perfect-hash generator. I haven’t measured and so I don’t know which one of these is faster.