My reasons aren’t ideological, they are highly practical. Basically I am realizing that while object-oriented C is OK for a while, it’s very weak at inheritance. Inheritance in C involves a lot of casting, duplicated code and/or macros, and careful discipline. The main problems with this are:

• the code gets longer and less readable
• the code involves more possibly-unsafe operations like casts

Both of these problems make the code ultimately more difficult to audit for security. And getting upb audited for security is something I plan to do very soon.

I am coming to believe that porting to C++ would make upb smaller (in lines of code) and easier for verify for security. However, there are a few major disadvantages that are giving me pause:

• there are still some contexts in which C++ is a no-go, like the Linux kernel, embedded systems that only have a C compiler (but no C++), or projects that want to stay C-only. Doing this port would make upb unsuitable for these contexts.
• projects that are currently C-only would need to create C++ source files to call upb APIs, and will have to link in the C++ runtime
• (possible) C++ could result in a larger binary.

When I look at the downsides though, they don’t seem to pertain to my initial goals of making upb useful for Python, Lua, Ruby, etc. extensions, and for use inside Google. Being useful for really restricted embedded systems is a far-off use case. So it’s sounding like porting to C++ is the right thing to do.

I hope it significantly reduces the line count, as I expect it will. That will make me feel better about giving up the minimalism of C. I will definitely be compiling with -fno-exceptions -fno-rtti -fvisibility-inlines-hidden on gcc. I also won’t be using any of the C++ standard library (not even <string>).

A lot of people mentioned that my last theme didn’t handle preview correctly, which I agree was quite annoying — sorry about that! Unfortunately my new theme doesn’t have preview at all. Finding the perfect WordPress theme is harder than it seems.

So as I mentioned I expect the first release of upb to come within the next week or two. It will only have parsing (not serializing), but most of the core functionality besides that is done. I expect that the core will stay relatively static, and all the enhancements/innovations will come from modules that are layered on top of that. For example, some of the things I have in mind are:

• Extensions for every language known to man. The first ones will be Python, Lua, and Ruby (in that order). These will take some thought to get just right, because I want the memory management to be tightly integrated. In particular, I want protobuf objects in these languages to be able to reference string data from the original protobufs, but all be memory-managed appropriately.
• Support for the Protocol Buffer text format.
• An easy way to parse only selected fields. For example, I want to be able to say (in Python):

  time, url = logrecord.parse_fields(proto_data, "time", "url")

  This can be optimized out the wazoo, because you can skip all fields and submessages that you don’t care about, and you can stop parsing once you have the data you need.

And of course, performance, performance, performance. I can’t wait to get my hands on some real profiles and see what my next optimization target will be.

Tonight I was working on the upb compiler (and upb’s first release is impending, by the way), and ran it under Valgrind as I frequently do to catch memory leaks. There weren’t any leaks, but I did notice that the program had allocated 80kb of memory over the course of its run.

People who are less OCD than I would probably shrug off 80kb of memory. But intuitively 80kb sounded high to me given how much data this program was dealing with, and I wanted to know where all those allocations were coming from.

I didn’t know off the top of my head how I might profile where my memory allocations were happening, but I had a hunch that Valgrind would be there for me. And sure enough, one of the tools included in Valgrind is Massif: a heap profiler.

A few short shell commands later:

$ valgrind --tool=massif ./upbc
$ ms_print massif.out.17604 | less

…and I had this call graph sitting in front of me:

93.72% (74,243B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->58.49% (46,336B) 0x40194E: upb_table_init (upb_table.c:34)
| ->38.13% (30,208B) 0x4019E9: upb_strtable_init (upb_table.c:45)
| | ->29.09% (23,040B) 0x40343A: upb_msg_init (upb_msg.c:44)
| | | ->29.09% (23,040B) 0x402CEE: insert_message (upb_context.c:193)
| | |   ->25.85% (20,480B) 0x402E80: addfd (upb_context.c:223)
| | |   | ->12.93% (10,240B) 0x40263F: upb_context_init (upb_context.c:30)
| | |   | | ->12.93% (10,240B) 0x40167C: main (upbc.c:195)
| | |   | |
| | |   | ->12.93% (10,240B) 0x403135: upb_context_parsefds (upb_context.c:283)
| | |   |   ->12.93% (10,240B) 0x4016BA: main (upbc.c:198)
| | |   |
| | |   ->03.23% (2,560B) 0x402D56: insert_message (upb_context.c:203)
| | |     ->03.23% (2,560B) 0x402E80: addfd (upb_context.c:223)
| | |       ->01.62% (1,280B) 0x40263F: upb_context_init (upb_context.c:30)
| | |       | ->01.62% (1,280B) 0x40167C: main (upbc.c:195)
| | |       |
| | |       ->01.62% (1,280B) 0x403135: upb_context_parsefds (upb_context.c:283)
| | |         ->01.62% (1,280B) 0x4016BA: main (upbc.c:198)
| | |
| | ->05.82% (4,608B) 0x4046E0: upb_enum_init (upb_enum.c:14)
| | | ->05.82% (4,608B) 0x402C35: insert_enum (upb_context.c:167)
| | |   ->05.82% (4,608B) 0x402DBC: insert_message (upb_context.c:209)
| | |     ->05.82% (4,608B) 0x402E80: addfd (upb_context.c:223)
| | |       ->03.23% (2,560B) 0x403135: upb_context_parsefds (upb_context.c:283)
| | |       | ->03.23% (2,560B) 0x4016BA: main (upbc.c:198)
| | |       |
| | |       ->02.59% (2,048B) 0x40263F: upb_context_init (upb_context.c:30)
| | |         ->02.59% (2,048B) 0x40167C: main (upbc.c:195)
| | |
| | ->01.62% (1,280B) 0x402612: upb_context_init (upb_context.c:27)
| | | ->01.62% (1,280B) 0x40167C: main (upbc.c:195)
| | |
| | ->01.62% (1,280B) 0x402629: upb_context_init (upb_context.c:28)
| | | ->01.62% (1,280B) 0x40167C: main (upbc.c:195)
| | |
| | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->20.36% (16,128B) 0x4019C4: upb_inttable_init (upb_table.c:40)
|   ->17.45% (13,824B) 0x403417: upb_msg_init (upb_msg.c:42)
|   | ->17.45% (13,824B) 0x402CEE: insert_message (upb_context.c:193)
|   |   ->15.51% (12,288B) 0x402E80: addfd (upb_context.c:223)
|   |   | ->07.76% (6,144B) 0x40263F: upb_context_init (upb_context.c:30)
|   |   | | ->07.76% (6,144B) 0x40167C: main (upbc.c:195)
|   |   | |
|   |   | ->07.76% (6,144B) 0x403135: upb_context_parsefds (upb_context.c:283)
|   |   |   ->07.76% (6,144B) 0x4016BA: main (upbc.c:198)
|   |   |
|   |   ->01.94% (1,536B) 0x402D56: insert_message (upb_context.c:203)
|   |     ->01.94% (1,536B) 0x402E80: addfd (upb_context.c:223)
|   |       ->01.94% (1,536B) in 2 places, all below massif's threshold (01.00%)
|   |
|   ->02.91% (2,304B) 0x4046F5: upb_enum_init (upb_enum.c:15)
|     ->02.91% (2,304B) 0x402C35: insert_enum (upb_context.c:167)
|       ->02.91% (2,304B) 0x402DBC: insert_message (upb_context.c:209)
|         ->02.91% (2,304B) 0x402E80: addfd (upb_context.c:223)
|           ->01.62% (1,280B) 0x403135: upb_context_parsefds (upb_context.c:283)
|           | ->01.62% (1,280B) 0x4016BA: main (upbc.c:198)
|           |
|           ->01.29% (1,024B) 0x40263F: upb_context_init (upb_context.c:30)
|             ->01.29% (1,024B) 0x40167C: main (upbc.c:195)
|
->07.98% (6,324B) 0x403AF1: upb_msgdata_new (upb_msg.c:158)
| ->07.96% (6,308B) 0x403F05: upb_msg_reuse_submsg (upb_msg.c:241)
| | ->07.96% (6,308B) 0x404497: submsg_start_cb (upb_msg.c:325)
| |   ->07.96% (6,308B) 0x4056E8: push_stack_frame (upb_parse.c:288)
| |     ->07.96% (6,308B) 0x4057ED: parse_delimited (upb_parse.c:316)
| |       ->07.96% (6,308B) 0x405A91: upb_parse (upb_parse.c:369)
| |         ->07.96% (6,308B) 0x4045BA: upb_msg_parse (upb_msg.c:352)
| |           ->07.96% (6,308B) 0x404631: upb_alloc_and_parse (upb_msg.c:361)
| |             ->07.96% (6,308B) 0x4030CA: upb_context_parsefds (upb_context.c:274)
| |               ->07.96% (6,308B) 0x4016BA: main (upbc.c:198)

This might look somewhat daunting if you’re not as deeply familiar with upb as I am. But it immediately told me what I wanted to know: almost 60% of the memory is being used by upb’s int->record and string->record hash tables. That seems a little bit high. And it’s being allocated right when the tables are constructed (upb_table_init), not as a result of a resize.

Breaking open the code, I found a table minimum size that I had imposed as an attempt to limit the number of resizes. Resizes have a high overhead — in my hash table implementation, they result in everything being re-hashed and all the memory being re-allocated, so I had imposed a minimum size of 16 in my constructor:

void upb_table_init(struct upb_table *t, uint32_t size, uint16_t entry_size)
{
  t->count = 0;
  t->entry_size = entry_size;
  t->size_lg2 = 1;
  while(size >>= 1) t->size_lg2++;
  t->size_lg2 = max(t->size_lg2, 4);  /* Min size of 16. */

When I inserted some print statements to compare how often this minimum was taking effect, I saw that there were tons of tables that were trying to allocate just a few (0-10) entries. With my minimum, they were always being allocated at least 16. And what’s more, in all these cases I knew up front how many entries I planned to insert! So there was no danger of a resize anyway.

I removed this minimum size and the memory usage of my program dipped to about 55kb (from 80kb — a ~30% reduction!) That seems a bit more reasonable, though I’m sure it’s not the last of my efforts to make sure upb’s memory footprint stays small.

Anyway, the point of this entry is that now I know about a new tool (Massif) that is at my disposal whenever I need it. It’s easy to use and requires almost no set-up. I can run it on a whim whenever I want to collect memory usage data. I have just become a little bit more resourceful.

Valgrind has tons of spiffy tools of this sort that ship with it. I wonder how many people know about them.

Another tool I had a similar reaction to was WireShark. I was experiencing a redirect loop bug in my browser and wanted to submit a useful report to the developers. The useful information here is the contents of all the HTTP traffic that was occurring during the loop. I fired up WireShark (as a first-time user) and found out relatively quickly how to sniff the network interface, capture my HTTP session, dump it at the HTTP layer (as opposed to the TCP layer or something else), and dumped it to a text file. Massively spiffy.

Learn an amazing new tool today! And then tell me about it!

I’m extremely happy with how it’s shaping up. It’s getting more and more complete, and yet still staying quite “micro.” Notably, it just recently crossed 1500 lines of C (I don’t count auto-generated code in this), and it compiles to not quite 10kb of object code. Throw in the auto-generated code (reflection data that describes the types in descriptor.proto — this is what allows loading other proto definitions at runtime) and this jumps to about 16kb. Keep in mind that this effectively has all the major features of the main protobuf implementation (albeit packaged in different ways), but the main protobuf implementation is just over 1MB of object code!

I still think I can reach the main protobuf implementation’s performance, or maybe even exceed it. I will consider the project a failure if I can’t get within 15% of its performance.

Anyway, just wanted to be sure everyone knows I’m still alive and very much working on this. Now back to work.

struct foo {
  unsigned int a:1;
  unsigned int b:2;
}

Without the bit-field specifiers (the “:1″ and “:2″ above) this structure would be at least four bytes long, and both “a” and “b” would be capable of representing integers from 0 to 65535. With the bit-field specifiers the structure is likely only one byte long, and “a” and “b” can only represent 0-1 and 0-3, respectively. Bit fields are a way to store many distinct values compactly inside a struct, to save memory.

I am using bitfields in upb to store flags for whether each field is set or not. When I generate a structure definition for a specific proto type, code in the header file looks something like this:

struct google_protobuf_DescriptorProto {
  union {
    uint8_t bytes[1];
    struct {
      bool name:1;  /* = 1, optional. */
      bool field:1;  /* = 2, repeated. */
      bool nested_type:1;  /* = 3, repeated. */
      bool enum_type:1;  /* = 4, repeated. */
      bool extension_range:1;  /* = 5, repeated. */
      bool extension:1;  /* = 6, repeated. */
      bool options:1;  /* = 7, optional. */
    } has;
  } set_flags;
  struct upb_string* name;
  UPB_STRUCT_ARRAY(google_protobuf_FieldDescriptorProto)* field;
  UPB_STRUCT_ARRAY(google_protobuf_FieldDescriptorProto)* extension;
  UPB_STRUCT_ARRAY(google_protobuf_DescriptorProto)* nested_type;
  UPB_STRUCT_ARRAY(google_protobuf_EnumDescriptorProto)* enum_type;
  UPB_STRUCT_ARRAY(google_protobuf_DescriptorProto_ExtensionRange)* extension_range;
  google_protobuf_MessageOptions* options;
};

(that’s from descriptor.h, which is automatically generated from descriptor.proto from the official protobuf implementation).

I “union” the bitfield with bytes so I can set the bytes en-masse more efficiently. This seems like a really nice solution, because then client code that wants to test whether a field is set in a protobuf can write code like:

if(fd->set_flags.has.message_type) {
   // ...
}

So unfortunately the commenters on Reddit were pretty down on bit-fields and the aforementioned article. The points they raised filled me with despair that I would have to abandon bit-fields. This is bad because the only real alternative I would have for this kind of code generation is to generate explicit getters and setters for each bit of each structure! The code would then immediately become quite uglified:

if(google_protobuf_DescriptorProto_has_message_type(fd)) {
  // ...
}

Besides being longer and forcing you to re-type the whole name of the enclosed type, this approach forces you to generate lots of nearly-identical code and pollute the symbol namespace with tons of these useless functions that add no real value. I was bumming about this.

But as I investigated the supposed problems with bit-fields, I became convinced that the problems were tractable and that I could in good conscience press forward with my plans to use them as intended.

Here Be Dragons

“Almost everything about [bit] fields is implementation-dependent.” –K&R, Section 6.9

From my reading of the C99 standard, here are the guarantees you get (or don’t get) with bit-fields.

1. Bit-fields will be packed as tightly as possible, provided they don’t cross storage unit boundaries: YES. From the standard: (6.7.2.1 #10):

An implementation may allocate any addressable storage unit large enough to hold a bit-field. If enough space remains, a bit-field that immediately follows another bit-field in a structure shall be packed into adjacent bits of the same unit.

The first part just means that the compiler can allocate any number of bytes to a bit-field, as long as it has enough bits. So if you write:

struct foo {
  unsigned int a:8;
}

…a will be at least 1 byte long, but could be more. But the second part is where we get the nice guarantee that:

struct foo {
  unsigned int a:1;
  unsigned int b:2;
}

…will always be packed into a single byte. sizeof(struct foo) could be greater than 1, but you are guaranteed that a and b are in the same byte.

2. Bits are allocated low-to-high (or high-to-low): NO. This is not guaranteed by the spec, so an implementation could choose to allocate bits either starting at the high bit or the low bit of each byte. The following program will test an implementation to see which it is:

#include <stdio.h>
int main() {
  union {
    struct {
      unsigned int a:1;
      unsigned int b:2;
      unsigned int c:3;
    } bitfield;
    unsigned char ch;
  } u = {.ch = 0};
 
  u.bitfield.a = 1;
  u.bitfield.c = 7;
  switch(u.ch) {
    case 0x39: printf("Bits allocated LOW-TO-HIGH.\n"); break;
    case 0x9C: printf("Bits allocated HIGH-TO-LOW.\n"); break;
    default: printf("Oddball allocation: 0x%02hhx.\n", u.ch); break;
  }
  return 0;
}

My Goal

What I am trying to do is create a C struct like the above, but also be able to perform data-driven reflection on that C struct based on generic routines like:

INLINE bool upb_msg_is_set(void *s, int index)
{
  return ((char*)s)[index / 8] & (1 << (index % 8));
}

This means that my reflection routines like the one above need to be able to mimic the implementation-defined bit ordering as described above. The routine above does not — it assumes that the bits are assigned from low to high. But a few #ifdefs should make the above not too difficult.

Performance

In theory, the ideal compiler should love bit-fields because they give it maximum information with which to optimize loads and stores. I see a lot of people claim that some compilers generate bad code for bit-fields, but that is a possibility with any construct you use. From what I can see GCC generates very good code for bit-fields.

So my conclusion is that the hurdles of bit-fields are tractable, and unless some new information comes my way, I plan to move forward with my plan to use them.

I had no such luck with pbstream. The tagline is currently:

pbstream - a stream-oriented implementation of protocol buffers.

It’s true that offering streaming semantics is one distinguishing feature of my implementation, but the way it’s growing it’s no longer really “stream-oriented.” It will offer both stream-oriented and structure-oriented semantics. Both will be equally supported/encouraged — it will be a matter of what best suits your needs.

If anything, the number one distinguishing feature of my implementation is that it is minimal. It gives you a set of tools to use whatever paradigm is the right trade-off for you. It gives you building blocks to assemble as you see fit. There are never “riders” — things like memory management that you have to take along with the parts you really want.

So what’s in a name? Desired characteristics:

• Unique, reasonably Googleable.
• Communicates that it is a protobuf implementation, and if possible communicates the philosophy described above.
• The name (or a reasonable abbreviation thereof) makes for a nice prefix that can be appended to all my identifiers. Right now it’s pbstream_this and pbstream_that.

Originally I just called it “pb” but that was a tad bit too generic. I slightly like “pblab” (communicates that it gives you building blocks for forging your own strategies), but I dislike the connotation that it is unfinished, unpolished, not production quality.

Perhaps I could give it a name, but keep using “pb_” in my identifiers. It could also help me organize the names of the different parts. I could use names like:

• pb_stream_parser_* for the stream parser.
• pb_struct_* for the code that defines an in-memory structure.

I kind of like that. But I still need a name for the package itself that is better than “pb”. Ideas?

So what is the brick wall I hit with pbstream? I had conflicting goals that I couldn’t figure out how to reconcile. One goal was to include as little policy related to memory management as humanly possible. In other words, if you’re using pbstream to store fully-parsed protobufs in memory — I’ll use the SAX vs. DOM analogy again, where this is the DOM case — then I want pbstream to not define any semantics for how the memory for that tree is managed. Is it reference-counted, garbage-collected, or is it just allocated/deallocated in one big chunk? Every runtime already has its own answer to this question. I don’t want to include a memory management strategy, which adds to the code size and complexity, just to have it be an annoying thing that you have to integrate into whatever runtime you’re already using.

Or, as I said in a previous entry: Oh fantastic! Because definitely the one thing that my application doesn’t already have is a memory manager.

On the other hand, I realized that I really do want the ability to pass an in-memory protobuf representation between language implementations. For example, if there’s a C library that conjures up some complex protobuf, I want to be able to pass that parsed in-memory protobuf to Python without serializing/deserializing. I want different runtimes to be able to look at this memory and make sense of it.

So far so good. This alone doesn’t require memory management. But I also wanted two things that do pose a memory-management problem:

• I wanted to pass protobufs between language implementations without copying.
• I wanted one language implementation to be able to modify the protobuf that had been created by a different language implementation.

The first is a problem because if you take a protobuf you created in C and pass it to Python, once that Python returns C has no idea whether Python has taken references to any of the protobuf’s data. If C frees the protobuf, Python could try to later read from the freed memory. If C doesn’t free the protobuf, the memory will leak. The fundamental problem is that C and Python have no way of cooperating to be sure the memory is freed only when the last reference that either of them holds is dropped.

The second is a problem for a similar reason. Suppose you take a protobuf you created in C and pass it to Python, and Python wants to mutate that protobuf. Suppose the protobuf has a submessage, which is implemented as a reference to that other message. If in python you say:

message.submessage = other_submessage

…Python now has to decide what to do with the submessage that message already had. Should it free it or not? If you free it and C had a reference to it, C references freed memory. If you don’t free it and C doesn’t have a reference to it, it leaks.

I fretted over the question of what to do about this for weeks. I didn’t want to give up the ability to pass protobufs between languages without serializing/deserializing. But I was loath to include the slightest amount of memory management code because of the implied complexity, run-time overhead, and code size effects. So everything ground to a halt while I tried to reconcile these conflicting desires in my head.

At one point I convinced myself that the only way out was reference-counting. The in-memory protobufs themselves would contain one extra integer for the reference count, and the code would contain just these little extra reference count increments/decrements and checks. Then C and Python would both maintain reference counts to these shared data structures. For Python it would be a sort of two-stage reference counting, since the individual Python objects would be reference-counted, and when their reference count dropped to zero they would decrement the reference count of the shared protobuf structure one.

Eww, Eww, EWW!! Now maybe you can understand a bit better why I haven’t gotten anything done for a while. There’s no way I could be very happy implementing this scheme.

So quite recently I found a way to relax the sharing requirements just enough that I can get away with not implementing any memory management. My new strategy is:

• language A can reference language B’s protobufs, but only for as long as B guarantees that all the references will continue to be valid. In most cases this just means that when B calls A and passes a protobuf, A can only reference B’s protobufs before it returns control back to B.
• In general, languages cannot mutate each other’s protobufs unless they have some special arrangement with regards to memory management. I thought I really needed this capability in Gazelle, but I’m no longer convinced I do.

So with that resolved, work is resuming, and I hope to have some results soon. I always hate making promises though, because I’m unwilling to press on when I don’t have a satisfactory answer to a question (as the last month demonstrates), and this can delay my progress long beyond my original intentions.

So my latest OCD goal this quest is to make it truly 100% C99 compliant: compilable on the weirdest and most nontraditional target, and free of any undefined behavior. I like most people have spent several years lulled into the x86/GCC monoculture. It’s time for me to transcend that narrow-minded view.

So I’ve set out to obtain every C compiler and lint tool I can possibly find, hoping that each one can show me even the smallest spot of imperfection that I can proceed to purge from my crystalline statements and declarations. I was ashamed to realize that I couldn’t even think of a time I had used a non-GCC/MSVC compiler in almost five years, and I didn’t know which ones were available to me. I am happy to say that I have eradicated some of that ignorance, and already have found some bugs that GCC didn’t show me!

So what C compilers are out there? Here are some of the ones I came across:

• Sun Studio, Sun’s C and C++ compiler. To my surprise, it is available for free. Includes a lint tool.
• Intel C++ Compiler. You can obtain a non-commercial version for free if you absolutely, totally, cross-your-heart promise that you are not compensated in any way for your work. I have a vision of the Intel guys following around open-source developers at conferences and revoking them their non-commercial licenses at the first sign of any swag.
• Digital Mars C and C++ compiler, which is freely available for Windows and DOS.
• Visual C++ Express is a free download from Microsoft. Apparently has very poor C99 support though.
• The Portland Group has a C and C++ compiler, but there doesn’t seem to be any way to use it without paying several hundred dollars.
• Comeau has a C and C++ compiler that takes great pride in its standards-compliance. You just have to get over the curiously awful web page, which actually says “Bursting With So Much Language Support It Hurts!” No free use (though at $50, it’s not a big hurdle — I paid more for a parking ticket today. If only I hadn’t had six such parking tickets to pay. For parking on my own street. But that’s a story for another day). There is a web form (!) where you can paste your code and see error/warning messages though, which could be all you need to test standards-compliance.
• SDCC, the Small Device C Compiler is a GPL’d C compiler that targets tiny embedded processors. I was looking quite forward to this one as a target that is wildly different from other C compilers. Unfortunately, though it has partial C99 support, it does not support 64-bit integer types, and I’m not quite willing tilt at that particular windmill of making pbstream portable to compilers without 64-bit integer support.
• TCC, the Tiny C Compiler. Fabrice Bellard is an absolute bad-ass, but unfortunately TCC doesn’t seem to support much of C99 (it choked on a for loop that defined a variable).
• Clang, the LLVM compiler. My version of Ubuntu doesn’t seem to have this available, and I always find LLVM a bit of a pain to compile and install from source, but I would like to try this when I get the chance.

So this should be enough variety that I can feel confident making smug assertions about pbstream’s portability. Now the only question is whether I’ll shell out $50 for the Comeau compiler just for that added potential for portability smugness.

I’ve learned to anticipate that there are dark and unstudied corners in even the most fundamental computer science topics. And hashing appears to be no exception. I’m investigating hashing at the moment because I need hash tables in a few places for my rapidly-developing Protocol Buffers implementation pbstream. So I wanted to take a quick survey of the state-of-the-art in hash tables, and implement a minimal hash table that suits my needs.

What are my needs? If someone defines a protobuf like so:

message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
}

…I need to build two hash tables for looking up fields: one keyed by number (1, 2, 3) and one keyed by name (name, id, email).

You might say “why not just use an array for looking up fields by number?” Indeed that’s what I’ll do in most cases, but field numbers can be as large as 2**27 and needn’t be allocated densely. So gracefully degrading to a hash table is important (one practical reason why the field numbers might not be dense is if the client is using extensions). To get the best of both worlds I’ll follow Lua’s lead and have the field number table be a hybrid array/hashtable, where the size of the array part is such that it’s at least half full.

My usage pattern for these hashtables is that I’ll build them once (when I parse a .proto file), then do lookups in the critical path of parsing. So insert time is practically irrelevant, but lookup time is extremely important, because it’s in my critical path. I’ll gladly trade some memory for fast lookups.

So I figured I’d take a few hours to discover the state-of-the-art in hashtables, implement that, and be on my merry way. Unfortunately there doesn’t seem to be much agreement about what the state of the art even is!

Let’s start with even the simple and most basic question: how big should a hashtable be? The literature differs here: some of it claiming that hashtables should have sizes that are prime numbers and some saying they should be powers of two.

But once you’ve decided that, you’ve only just entered the jungle of collision-resolution strategies. There are a lot of them, and it’s not at all clear to me that the trade-offs are well understood. A list of ones I’ve come across, and my current understanding of what they mean:

• linear probing: if your hash function is h(k), then a collision in T[h(k)] means you should also try T[h(k)+i] for i=1 to some limit.
• quadratic probing: like linear probing, but also throw in a T[h(k) + i + j^2], to spread it out a little.
• double hashing: like linear probing, but you scale the jumps by another hash function: T[h(k) + g(k)*i].
• chaining: all entries that collide are in a linked list, so for lookups you search this list. chaining can be either internal (table entries have both values and links, links point to other locations in the table) or external (table is just pointers to the head of linked lists, inserts always allocate a new node).
• cuckoo hashing: Wikipedia makes this sound like hashing’s best-kept-secret. You use two hash functions: if you do an insertion and the spot you want is taken, you kick whatever was already there out and put it in its alternate spot.
• two-way chaining: use two hash functions (like cuckoo and double hashing), but you use the two hash values to find the two possible chains the value could be in.

Few papers seem to offer a comprehensive account of the trade-offs between these. Some authors claim that linear probing is best on modern CPUs because of excellent locality (cache-friendly), but this paper [PDF] takes exception to that claim, saying that the better overall performance of double-hashing negates the cache effects. And indeed, none of the real-world implementations I found use linear probing.

A quick survey of mainstream hashing implementations shows that they can’t agree on what’s best either.

• Lua uses internal chaining with powers-of-two table sizes and “Brent’s variation” (so-named because of a 1973 paper called Reducing the retrieval time of scatter storage techniques — so much for caching-conscious schemes).
• Ruby uses internal chaining with prime table sizes.
• Python uses a custom probing scheme. You have to read the comments in the source file itself — they do more to make hashing sound like black magic than I possible could.
• Perl appears to use external chaining, but its code is a bit hard to follow.

Even if you decide on a collision strategy, you still have to decide on a hash function, and there are surprises lurking for you there as well. Conventional wisdom about hashing is that you want a hash function that randomly distributes the inputs to the outputs. But Python and Lua both hash integers to themselves (so h(5) = 5)!

There are just a dizzying number of variations on hashing. Given my relatively simple use case (one-time construction, lookup speed trumps all) you’d think there would be a clear and obvious answer. But I don’t see one.

Given my deep respect for Lua’s implementation, I’m tempted to just start with that and worry about trying to optimize it later. I’m slightly disconcerted with its focus on getting good performance even when the table is quite full, because as I mentioned for my use case I’ll gladly trade modest amounts of RAM for faster lookup. At least it’s someplace to start.

But seriously, who knew that hash tables had so many variations, with no clear winner?