A Flame About 64-bit Pointers

It is absolutely idiotic to have 64-bit pointers when I compile a program that uses less than 4 gigabytes of RAM. When such pointer values appear inside a struct, they not only waste half the memory, they effectively throw away half of the cache.

The gcc manpage advertises an option “-mlong32″ that sounds like what I want. Namely, I think it would compile code for my x86-64 architecture, taking advantage of the extra registers etc., but it would also know that my program is going to live inside a 32-bit virtual address space.

Unfortunately, the -mlong32 option was introduced only for MIPS computers, years ago. Nobody has yet adopted such conventions for today’s most popular architecture. Probably that happens because programs compiled with this convention will need to be loaded with a special version of libc.

Please, somebody, make that possible.

I always thought this made a lot of sense. People have asked distro-makers for this before without a lot of success, but it looks like this is now being worked on by high-profile people in the Linux community. It is called The x32 ABI (see the LWN coverage for a more digestible description). It’s exciting because in some benchmarks this can outperform the x86-64 ABI by 10% or more. It’s a tradeoff — if you don’t need to address more than 4GB of memory, you can get faster programs because smaller pointers have better cache utilization. You’ll use less memory too.

This could have been done in a way that operated nearly the same as “compatibility mode” (ie. running 32-bit binaries on a 64-bit CPU/OS), which would have required only minimal changes to the kernel/toolchain. But it looks like their plans are more ambitious: they want to be able to use the optimized SYSCALL64 instruction (which is “much faster” than int 0x80 according to H. Peter Anvin), and they’re looking at fixing other problems like 32-bit time_t. So it’s a more substantial effort, but it looks like there’s significant interest and momentum behind this.

Thinking about how this would affect upb, my impression is that I could use my x86-64 JIT unmodified with x32, since it appears to have all of the same calling conventions. It has the same set of callee-save registers and the same set of registers for parameter transfer, and I think these are the main things upb’s JIT-ted code depends on.

Notably, I think upb’s deserialization is ready for casual and experimental use. Definitely don’t trust any data to it until it’s better tested, though. I won’t be releasing until things have converged a bit more (and are better tested).

I gave a talk about upb at Google yesterday that was well-received. One question that comes up is “how are you beating the generated code from the protobuf compiler?” For the record, my JIT appears to be about 25% faster than Google’s protobuf release on a completely apples-to-apples test. It is a bit surprising, since my code has basically the same structure as protobuf’s generated C++ — I don’t invent any new optimizations or anything like that. I think it really comes down to generating better assembly than gcc is.

We live in a day and age where common wisdom is that you can’t beat a good C++ compiler, or at least not by much, and I think this is probably true for 99% of use cases. But I was first inspired to think that I could beat the C++ compiler in this case by reading this mailing list post from Mike Pall where he explains why you can still beat the compiler for interpreter main loops. A protobuf parser is surprisingly similar to a byte-code interpreter main loop, so I thought I’d give it a shot.

Below is just the simplest example I could dig up of a side-by-side comparison of my code vs. the compiler’s. What follows is the code to parse a single fixed64 value:

  ; upb JIT assembly:
  mov    rdx,QWORD PTR [rbx+0x2]    ; load fixed64 val out of buffer
  add    rbx,0xa                    ; advance buffer by 10 (2 for tag)
  mov    QWORD PTR [r12+0x40],rdx   ; store fixed64 value in message
  or     BYTE PTR [r12+0x1],0x4     ; set hasbit
  cmp    rbx,QWORD PTR [r15+0xaf8]  ; check for end-of-buffer
  jae    
  mov    rcx,QWORD PTR [rbx]        ; load next tag
  cmp    cx,0x1b0                   ; next field+wt in order?
  je     

There’s not a lot left to cut away here. Compare with protobuf/gcc-generated code:

  ; protobuf/gcc code:
  mov    ecx,DWORD PTR [rbx+0x10]   ; load buffer end
  mov    rax,QWORD PTR [rbx+0x8]    ; load buffer
  sub    ecx,eax                    ; if (buffer_end_ - buffer_ <= 7)
  cmp    ecx,0x7
  jle    
  mov    rax,QWORD PTR [rax]        ; load fixed64 val
  mov    rdx,QWORD PTR [rbp-0x48]   ; load this
  mov    QWORD PTR [rdx],rax        ; store fixed64 val in this
  add    QWORD PTR [rbx+0x8],0x8    ; advance buffer
  or     DWORD PTR [r12+0x74],0x800 ; set hasbit
  mov    rdx,QWORD PTR [rbx+0x10]   ; load buffer end
  mov    rax,QWORD PTR [rbx+0x8]    ; load buffer
  mov    ecx,edx
  sub    ecx,eax                    ; if (buffer_end_ - buffer_ <= 1)
  cmp    ecx,0x1
  jle    
  cmp    BYTE PTR [rax],0xb0        ; check first byte of next tag
  jne    
  cmp    BYTE PTR [rax+0x1],0x1     ; check second byte of next tag
  jne    

There is some poor register allocation going on here — gcc is repeatedly loading buffer_ and buffer_end_ even though it has plenty of registers to play with. All in all, gcc is generating over twice the number of instructions, over twice the number of loads, and more stores too. Note that this is taken from the middle of a very large C++ function with a big switch statement and a bunch of gotos, and I think it’s just difficult for a compiler to do good register allocation under these constraints.

If the C++ compiler could know the difference between fast paths and slow paths and do the register allocation solely for the fast paths (spilling everything for the slow paths) it might have a better shot. But still I think it’s just a hard problem.

Why JIT?

Before I get to the numbers, I should explain what it even means for upb to have a JIT. If you’re not interested in the technical details, feel free to skip this section to go straight to the impacts and what this means for upb.

So why a JIT? After all, protobufs are not an imperative language. You can’t write an algorithm in a .proto file. You can’t apply compiler techniques like SSA, strength reduction, tracing, or really any of the things you’d expect from a JIT on a platform like the JVM. There is no byte code or intermediate representation to speak of. So why would upb have a JIT and what on earth would it do?

To review, a .proto file defines a schema for some messages. In this sense, it is comparable to JSON Schema or XML Schema.

message Person {
  required int32 id = 1;
  required string name = 2;
  optional string email = 3;
}

To support forwards and backwards compatibility, we don’t make any assumptions about which fields we’re going to see or in what order. Instead, on the wire each field is preceded by its tag number. Logically, the serialized version is something like:

- field number: 1, wire type: varint, value: 1
- field number 2, wire type: delimited, value: "Josh Haberman"
- field number 3, wire type: delimited, value: "jhaberman@gmail.com"

If you were going to write a parser for this, you might create a table of fields, keyed by field number, and a main parser loop that looks something like this:

while (!done) {
  int type = fields[get_field_number()];
  switch (type) {
    case WIRE_TYPE_VARINT:
      parse_varint();
      break;
    case WIRE_TYPE_DELIMITED:
      parse_delimited();
      break;
    // etc.
  }
}

We call this a “table-based parser.” We look up the type in a table of fields, and use that to branch to the correct value parsing function. There are minor variations on this, like using a function pointer instead of a switch statement, but it’s the same general idea. To see an actual program that implements this style of parser, check out my bare-bones 100-line protobuf parser that I wrote a few years ago: pb.c.

In a way this resembles a byte-code interpreter, and the serialized protobuf resembles byte-code. For example, see the main loop of the Lua interpreter. It has a similar pattern where it extracts an opcode and uses that to branch into a giant switch statement.

It turns out that if we generate code that is specific to one message type, and takes advantage of the fact that fields are usually encoded in order, we can beat this table-based parser significantly. Google’s main “protobuf” software has done this for a long time: their strategy is to generate C++ classes that are specific to a proto message type (like Person in the above example). This code has been highly tuned and optimized, and the generated parsing code looks something like this:

while((tag = input->ReadTag()) != 0) {
  field_number = GetFieldNumber(tag);
  wire_type = GetWireType(tag);
  switch(tag) {
    // optional float field1 = 1;
    case 1:
      if (wire_type != FLOAT_WIRE_TYPE) goto error;
      input->ParseFloat();
      // Saves the switch() and type check branch if fields are in order.
      if (input->ExpectTag(2, DOUBLE_WIRE_TYPE)) goto parse_field2;
      break;

    // optional double field2 = 2;
    case 2:
      if (wire_type != DOUBLE_WIRE_TYPE) goto error;
parse_field2:
      input->ParseDouble();
      // saves the switch() and type check branch if fields are in order.
      if (input->ExpectTag(3, INT32_WIRE_TYPE)) goto parse_field3
      break;

    // optional int32 field3 = 3;
    case 3:
    // [...]
  }
}

This code also has a big switch() statement at the top level, but the targets are field numbers instead of wire types. More importantly, each field’s case has a line like this:

if (input->ExpectTag(3, INT32_WIRE_TYPE)) goto parse_field3

This is key, because if the fields occur in order (which they usually do) then this branch will always be taken and it will be predicted correctly by the CPU. This can make a huge difference.

While this kind of C++ code generation has benefited Google’s protobuf software in speed, I always found it inconvenient from a dynamic languages perspective. What dynamic language user wants to have to generate C++ code and link that into their interpreter? We’re not talking about a single C++ extension that you compile once: every single message you define in a .proto file generates different C++ code. So you can’t just apt-get install libprotobuf-python and be done with it, you have to generate C++ for every message that your specific app wants to use.

This is where the JIT comes in. If we can generate machine code at runtime, we can get the speed benefits of the generated code without having to generate, compile, and link C++ into our interpreters or VMs. You could compile the library once, and after that you can dynamically load message definitions but still get the fastest possible parsing.

Preliminary results

I knew all of the theory I just described before I started writing a JIT. But theory is just that — theory. What would actually happen when I implemented a JIT?

The results exceeded my expectations. I still am being somewhat cautious, because there are so many dimensions to any performance equation, and because benchmarks are not guaranteed to correspond to real-world performance. That said, here are my preliminary results. You can reproduce these by obtaining upb from GitHub and running make benchmarks followed by make benchmark (you have to define -DUPB_USE_JIT_X64 to get the JIT).

Parsing an 80k protobuf into a data structure repeatedly,
calling Clear() between each parse.  (proto2 == Google protobuf)

proto2 table-based parser      38 MB/s
proto2 generated code parser   265 MB/s
upb table-based parser         340 MB/s
upb JIT parser                 741 MB/s

The results above are designed to be as apples-to-apples as possible. In other words, I disabled upb’s optimization that avoids memcpy() because Google’s protobuf doesn’t support it in its open-source release. I think the reason that even my table-based parser beats Google’s generated code here is because proto2 implements Clear() in a sub-optimal way that requires an extra pass over the message tree; see my comment in upb_msg.h for more info.

Things get even better if we allow ourselves to drop the constraint of being apples-to-apples with proto2. upb is capable of stream-based parsing like SAX, whereas proto2 only supports a DOM-based approach where you parse into a data structure. If we include the performance numbers for stream parsing, and for DOM-based parsing that avoids memcpy, we get:

upb table-based stream parser    420 MB/s
upb JIT no-memcpy() DOM parser   870 MB/s
upb JIT stream parser           1430 MB/s

If you’re like me when I first saw these numbers, your jaw is on the floor at seeing almost 1.5GB/s doing stream parsing of protocol buffers. At this point we are 5x proto2′s highly optimized generated code.

upb’s JIT isn’t complete and can’t handle all cases, but these performance numbers should still be valid because these benchmarks only used the parts of the format that are currently supported.

So when can I USE it? And can I help?

One reason I’ve avoided posting these extremely positive results so far is that I hate to get people excited about something that’s still not ready for users yet.

Adding support for the JIT required making extremely large and intrusive changes to the core interfaces, like this 3000 line refactoring that had to be completed before I could write even the first line of the JIT. I need to have the flexibility to change core interfaces like this still, because the design is still converging. So as much as I wish I could say it’s ready to go, I still need to hold off on a real release.

The good news is that the design is still making huge strides. In the last few weeks I’ve been refining my scheme for how upb will integrate into different VM’s and language runtimes, and I feel more confident than ever that the language bindings for Lua, Python, etc. will be some of the fastest extensions ever offered for this kind of functionality.

As a preview for where this is going, I think that upb will even be usable as a JSON library, offering speed that is as good or greater than any existing JSON libraries. JSON can be directly mapped onto a subset of protobufs (JSON only uses double-precision numbers) and the protobuf text format is already extremely similar to JSON. So all the work I’ve done to optimize memory layout and dynamic language object access should apply.

And while I’m really happy to get offers to help out, it’s still at a stage of design where I need to be doing most of the work. Working on upb generally involves pacing around my apartment deep in thought about all the requirements and use cases I want to satisfy and brainstorming a million different approaches until I converge on the the one that is the smallest, fastest, and most flexible. It’s hard work but I love it, and the more time that passes the more convinced I am that this is going to be big.

However, if I’ve learned one thing over the last several years of working on Gazelle and upb, it’s that I am extraordinarily bad at knowing how close I am to being ready to release. I don’t want to make any predictions or promises. I honestly thought I was almost ready to release upb a year and a half ago, but I’ve almost completely rewritten it twice since then. Each time it gets significantly better and closer to being what I want it to be. Once I have a release I’ll describe more about the stages of evolution it went through and how each iteration was objectively better than the one before it.

The core interfaces have gotten to a point where I’m really happy with them and feel no more need to rework them. I think they will continue to evolve incrementally, but not in a way that requires redoing them completely. Here are the most core interfaces — if you’re interested in upb, I recommend reading these headers. I’ve just added substantial comments to explain them, and more than anything these will give you a taste of what upb is all about:

• upb_stream.h, the streaming interfaces for doing SAX-like tree traversal of protobuf data, and abstractions of fread()/fwrite(). These are probably the most important interfaces in all of upb, since all of the encoders and decoders are based on them.
• upb_string.h, an immutable, length-delimited (instead of NULL-terminated), reference-counted string type.
• upb_def.h, the data structures for a protobuf schema (.proto file) and routines for loading them.

By the way, I also mean to write zero-overhead C++ wrappers around the above to give you C++ programmers a nicer interface at no cost.

With those set, I have been rapidly getting everything building/working again. It’s a bit annoying to rewrite upb_def.c for the third time (literally) but it feels good knowing the interfaces are right.

So I have renewed optimism that I’ll be releasing soon. And once I’m happy with upb, it’s back to Gazelle.

My reasons aren’t ideological, they are highly practical. Basically I am realizing that while object-oriented C is OK for a while, it’s very weak at inheritance. Inheritance in C involves a lot of casting, duplicated code and/or macros, and careful discipline. The main problems with this are:

• the code gets longer and less readable
• the code involves more possibly-unsafe operations like casts

Both of these problems make the code ultimately more difficult to audit for security. And getting upb audited for security is something I plan to do very soon.

I am coming to believe that porting to C++ would make upb smaller (in lines of code) and easier for verify for security. However, there are a few major disadvantages that are giving me pause:

• there are still some contexts in which C++ is a no-go, like the Linux kernel, embedded systems that only have a C compiler (but no C++), or projects that want to stay C-only. Doing this port would make upb unsuitable for these contexts.
• projects that are currently C-only would need to create C++ source files to call upb APIs, and will have to link in the C++ runtime
• (possible) C++ could result in a larger binary.

When I look at the downsides though, they don’t seem to pertain to my initial goals of making upb useful for Python, Lua, Ruby, etc. extensions, and for use inside Google. Being useful for really restricted embedded systems is a far-off use case. So it’s sounding like porting to C++ is the right thing to do.

I hope it significantly reduces the line count, as I expect it will. That will make me feel better about giving up the minimalism of C. I will definitely be compiling with -fno-exceptions -fno-rtti -fvisibility-inlines-hidden on gcc. I also won’t be using any of the C++ standard library (not even <string>).

A lot of people mentioned that my last theme didn’t handle preview correctly, which I agree was quite annoying — sorry about that! Unfortunately my new theme doesn’t have preview at all. Finding the perfect WordPress theme is harder than it seems.

So as I mentioned I expect the first release of upb to come within the next week or two. It will only have parsing (not serializing), but most of the core functionality besides that is done. I expect that the core will stay relatively static, and all the enhancements/innovations will come from modules that are layered on top of that. For example, some of the things I have in mind are:

• Extensions for every language known to man. The first ones will be Python, Lua, and Ruby (in that order). These will take some thought to get just right, because I want the memory management to be tightly integrated. In particular, I want protobuf objects in these languages to be able to reference string data from the original protobufs, but all be memory-managed appropriately.
• Support for the Protocol Buffer text format.
• An easy way to parse only selected fields. For example, I want to be able to say (in Python):

  time, url = logrecord.parse_fields(proto_data, "time", "url")

  This can be optimized out the wazoo, because you can skip all fields and submessages that you don’t care about, and you can stop parsing once you have the data you need.

And of course, performance, performance, performance. I can’t wait to get my hands on some real profiles and see what my next optimization target will be.

Tonight I was working on the upb compiler (and upb’s first release is impending, by the way), and ran it under Valgrind as I frequently do to catch memory leaks. There weren’t any leaks, but I did notice that the program had allocated 80kb of memory over the course of its run.

People who are less OCD than I would probably shrug off 80kb of memory. But intuitively 80kb sounded high to me given how much data this program was dealing with, and I wanted to know where all those allocations were coming from.

I didn’t know off the top of my head how I might profile where my memory allocations were happening, but I had a hunch that Valgrind would be there for me. And sure enough, one of the tools included in Valgrind is Massif: a heap profiler.

A few short shell commands later:

$ valgrind --tool=massif ./upbc
$ ms_print massif.out.17604 | less

…and I had this call graph sitting in front of me:

93.72% (74,243B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->58.49% (46,336B) 0x40194E: upb_table_init (upb_table.c:34)
| ->38.13% (30,208B) 0x4019E9: upb_strtable_init (upb_table.c:45)
| | ->29.09% (23,040B) 0x40343A: upb_msg_init (upb_msg.c:44)
| | | ->29.09% (23,040B) 0x402CEE: insert_message (upb_context.c:193)
| | |   ->25.85% (20,480B) 0x402E80: addfd (upb_context.c:223)
| | |   | ->12.93% (10,240B) 0x40263F: upb_context_init (upb_context.c:30)
| | |   | | ->12.93% (10,240B) 0x40167C: main (upbc.c:195)
| | |   | |
| | |   | ->12.93% (10,240B) 0x403135: upb_context_parsefds (upb_context.c:283)
| | |   |   ->12.93% (10,240B) 0x4016BA: main (upbc.c:198)
| | |   |
| | |   ->03.23% (2,560B) 0x402D56: insert_message (upb_context.c:203)
| | |     ->03.23% (2,560B) 0x402E80: addfd (upb_context.c:223)
| | |       ->01.62% (1,280B) 0x40263F: upb_context_init (upb_context.c:30)
| | |       | ->01.62% (1,280B) 0x40167C: main (upbc.c:195)
| | |       |
| | |       ->01.62% (1,280B) 0x403135: upb_context_parsefds (upb_context.c:283)
| | |         ->01.62% (1,280B) 0x4016BA: main (upbc.c:198)
| | |
| | ->05.82% (4,608B) 0x4046E0: upb_enum_init (upb_enum.c:14)
| | | ->05.82% (4,608B) 0x402C35: insert_enum (upb_context.c:167)
| | |   ->05.82% (4,608B) 0x402DBC: insert_message (upb_context.c:209)
| | |     ->05.82% (4,608B) 0x402E80: addfd (upb_context.c:223)
| | |       ->03.23% (2,560B) 0x403135: upb_context_parsefds (upb_context.c:283)
| | |       | ->03.23% (2,560B) 0x4016BA: main (upbc.c:198)
| | |       |
| | |       ->02.59% (2,048B) 0x40263F: upb_context_init (upb_context.c:30)
| | |         ->02.59% (2,048B) 0x40167C: main (upbc.c:195)
| | |
| | ->01.62% (1,280B) 0x402612: upb_context_init (upb_context.c:27)
| | | ->01.62% (1,280B) 0x40167C: main (upbc.c:195)
| | |
| | ->01.62% (1,280B) 0x402629: upb_context_init (upb_context.c:28)
| | | ->01.62% (1,280B) 0x40167C: main (upbc.c:195)
| | |
| | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->20.36% (16,128B) 0x4019C4: upb_inttable_init (upb_table.c:40)
|   ->17.45% (13,824B) 0x403417: upb_msg_init (upb_msg.c:42)
|   | ->17.45% (13,824B) 0x402CEE: insert_message (upb_context.c:193)
|   |   ->15.51% (12,288B) 0x402E80: addfd (upb_context.c:223)
|   |   | ->07.76% (6,144B) 0x40263F: upb_context_init (upb_context.c:30)
|   |   | | ->07.76% (6,144B) 0x40167C: main (upbc.c:195)
|   |   | |
|   |   | ->07.76% (6,144B) 0x403135: upb_context_parsefds (upb_context.c:283)
|   |   |   ->07.76% (6,144B) 0x4016BA: main (upbc.c:198)
|   |   |
|   |   ->01.94% (1,536B) 0x402D56: insert_message (upb_context.c:203)
|   |     ->01.94% (1,536B) 0x402E80: addfd (upb_context.c:223)
|   |       ->01.94% (1,536B) in 2 places, all below massif's threshold (01.00%)
|   |
|   ->02.91% (2,304B) 0x4046F5: upb_enum_init (upb_enum.c:15)
|     ->02.91% (2,304B) 0x402C35: insert_enum (upb_context.c:167)
|       ->02.91% (2,304B) 0x402DBC: insert_message (upb_context.c:209)
|         ->02.91% (2,304B) 0x402E80: addfd (upb_context.c:223)
|           ->01.62% (1,280B) 0x403135: upb_context_parsefds (upb_context.c:283)
|           | ->01.62% (1,280B) 0x4016BA: main (upbc.c:198)
|           |
|           ->01.29% (1,024B) 0x40263F: upb_context_init (upb_context.c:30)
|             ->01.29% (1,024B) 0x40167C: main (upbc.c:195)
|
->07.98% (6,324B) 0x403AF1: upb_msgdata_new (upb_msg.c:158)
| ->07.96% (6,308B) 0x403F05: upb_msg_reuse_submsg (upb_msg.c:241)
| | ->07.96% (6,308B) 0x404497: submsg_start_cb (upb_msg.c:325)
| |   ->07.96% (6,308B) 0x4056E8: push_stack_frame (upb_parse.c:288)
| |     ->07.96% (6,308B) 0x4057ED: parse_delimited (upb_parse.c:316)
| |       ->07.96% (6,308B) 0x405A91: upb_parse (upb_parse.c:369)
| |         ->07.96% (6,308B) 0x4045BA: upb_msg_parse (upb_msg.c:352)
| |           ->07.96% (6,308B) 0x404631: upb_alloc_and_parse (upb_msg.c:361)
| |             ->07.96% (6,308B) 0x4030CA: upb_context_parsefds (upb_context.c:274)
| |               ->07.96% (6,308B) 0x4016BA: main (upbc.c:198)

This might look somewhat daunting if you’re not as deeply familiar with upb as I am. But it immediately told me what I wanted to know: almost 60% of the memory is being used by upb’s int->record and string->record hash tables. That seems a little bit high. And it’s being allocated right when the tables are constructed (upb_table_init), not as a result of a resize.

Breaking open the code, I found a table minimum size that I had imposed as an attempt to limit the number of resizes. Resizes have a high overhead — in my hash table implementation, they result in everything being re-hashed and all the memory being re-allocated, so I had imposed a minimum size of 16 in my constructor:

void upb_table_init(struct upb_table *t, uint32_t size, uint16_t entry_size)
{
  t->count = 0;
  t->entry_size = entry_size;
  t->size_lg2 = 1;
  while(size >>= 1) t->size_lg2++;
  t->size_lg2 = max(t->size_lg2, 4);  /* Min size of 16. */

When I inserted some print statements to compare how often this minimum was taking effect, I saw that there were tons of tables that were trying to allocate just a few (0-10) entries. With my minimum, they were always being allocated at least 16. And what’s more, in all these cases I knew up front how many entries I planned to insert! So there was no danger of a resize anyway.

I removed this minimum size and the memory usage of my program dipped to about 55kb (from 80kb — a ~30% reduction!) That seems a bit more reasonable, though I’m sure it’s not the last of my efforts to make sure upb’s memory footprint stays small.

Anyway, the point of this entry is that now I know about a new tool (Massif) that is at my disposal whenever I need it. It’s easy to use and requires almost no set-up. I can run it on a whim whenever I want to collect memory usage data. I have just become a little bit more resourceful.

Valgrind has tons of spiffy tools of this sort that ship with it. I wonder how many people know about them.

Another tool I had a similar reaction to was WireShark. I was experiencing a redirect loop bug in my browser and wanted to submit a useful report to the developers. The useful information here is the contents of all the HTTP traffic that was occurring during the loop. I fired up WireShark (as a first-time user) and found out relatively quickly how to sniff the network interface, capture my HTTP session, dump it at the HTTP layer (as opposed to the TCP layer or something else), and dumped it to a text file. Massively spiffy.

Learn an amazing new tool today! And then tell me about it!

I’m extremely happy with how it’s shaping up. It’s getting more and more complete, and yet still staying quite “micro.” Notably, it just recently crossed 1500 lines of C (I don’t count auto-generated code in this), and it compiles to not quite 10kb of object code. Throw in the auto-generated code (reflection data that describes the types in descriptor.proto — this is what allows loading other proto definitions at runtime) and this jumps to about 16kb. Keep in mind that this effectively has all the major features of the main protobuf implementation (albeit packaged in different ways), but the main protobuf implementation is just over 1MB of object code!

I still think I can reach the main protobuf implementation’s performance, or maybe even exceed it. I will consider the project a failure if I can’t get within 15% of its performance.

Anyway, just wanted to be sure everyone knows I’m still alive and very much working on this. Now back to work.

struct foo {
  unsigned int a:1;
  unsigned int b:2;
}

Without the bit-field specifiers (the “:1″ and “:2″ above) this structure would be at least four bytes long, and both “a” and “b” would be capable of representing integers from 0 to 65535. With the bit-field specifiers the structure is likely only one byte long, and “a” and “b” can only represent 0-1 and 0-3, respectively. Bit fields are a way to store many distinct values compactly inside a struct, to save memory.

I am using bitfields in upb to store flags for whether each field is set or not. When I generate a structure definition for a specific proto type, code in the header file looks something like this:

struct google_protobuf_DescriptorProto {
  union {
    uint8_t bytes[1];
    struct {
      bool name:1;  /* = 1, optional. */
      bool field:1;  /* = 2, repeated. */
      bool nested_type:1;  /* = 3, repeated. */
      bool enum_type:1;  /* = 4, repeated. */
      bool extension_range:1;  /* = 5, repeated. */
      bool extension:1;  /* = 6, repeated. */
      bool options:1;  /* = 7, optional. */
    } has;
  } set_flags;
  struct upb_string* name;
  UPB_STRUCT_ARRAY(google_protobuf_FieldDescriptorProto)* field;
  UPB_STRUCT_ARRAY(google_protobuf_FieldDescriptorProto)* extension;
  UPB_STRUCT_ARRAY(google_protobuf_DescriptorProto)* nested_type;
  UPB_STRUCT_ARRAY(google_protobuf_EnumDescriptorProto)* enum_type;
  UPB_STRUCT_ARRAY(google_protobuf_DescriptorProto_ExtensionRange)* extension_range;
  google_protobuf_MessageOptions* options;
};

(that’s from descriptor.h, which is automatically generated from descriptor.proto from the official protobuf implementation).

I “union” the bitfield with bytes so I can set the bytes en-masse more efficiently. This seems like a really nice solution, because then client code that wants to test whether a field is set in a protobuf can write code like:

if(fd->set_flags.has.message_type) {
   // ...
}

So unfortunately the commenters on Reddit were pretty down on bit-fields and the aforementioned article. The points they raised filled me with despair that I would have to abandon bit-fields. This is bad because the only real alternative I would have for this kind of code generation is to generate explicit getters and setters for each bit of each structure! The code would then immediately become quite uglified:

if(google_protobuf_DescriptorProto_has_message_type(fd)) {
  // ...
}

Besides being longer and forcing you to re-type the whole name of the enclosed type, this approach forces you to generate lots of nearly-identical code and pollute the symbol namespace with tons of these useless functions that add no real value. I was bumming about this.

But as I investigated the supposed problems with bit-fields, I became convinced that the problems were tractable and that I could in good conscience press forward with my plans to use them as intended.

Here Be Dragons

“Almost everything about [bit] fields is implementation-dependent.” –K&R, Section 6.9

From my reading of the C99 standard, here are the guarantees you get (or don’t get) with bit-fields.

1. Bit-fields will be packed as tightly as possible, provided they don’t cross storage unit boundaries: YES. From the standard: (6.7.2.1 #10):

An implementation may allocate any addressable storage unit large enough to hold a bit-field. If enough space remains, a bit-field that immediately follows another bit-field in a structure shall be packed into adjacent bits of the same unit.

The first part just means that the compiler can allocate any number of bytes to a bit-field, as long as it has enough bits. So if you write:

struct foo {
  unsigned int a:8;
}

…a will be at least 1 byte long, but could be more. But the second part is where we get the nice guarantee that:

struct foo {
  unsigned int a:1;
  unsigned int b:2;
}

…will always be packed into a single byte. sizeof(struct foo) could be greater than 1, but you are guaranteed that a and b are in the same byte.

2. Bits are allocated low-to-high (or high-to-low): NO. This is not guaranteed by the spec, so an implementation could choose to allocate bits either starting at the high bit or the low bit of each byte. The following program will test an implementation to see which it is:

#include 
int main() {
  union {
    struct {
      unsigned int a:1;
      unsigned int b:2;
      unsigned int c:3;
    } bitfield;
    unsigned char ch;
  } u = {.ch = 0};

  u.bitfield.a = 1;
  u.bitfield.c = 7;
  switch(u.ch) {
    case 0x39: printf("Bits allocated LOW-TO-HIGH.\n"); break;
    case 0x9C: printf("Bits allocated HIGH-TO-LOW.\n"); break;
    default: printf("Oddball allocation: 0x%02hhx.\n", u.ch); break;
  }
  return 0;
}

My Goal

What I am trying to do is create a C struct like the above, but also be able to perform data-driven reflection on that C struct based on generic routines like:

INLINE bool upb_msg_is_set(void *s, int index)
{
  return ((char*)s)[index / 8] & (1 << (index % 8));
}

This means that my reflection routines like the one above need to be able to mimic the implementation-defined bit ordering as described above. The routine above does not -- it assumes that the bits are assigned from low to high. But a few #ifdefs should make the above not too difficult.

Performance

In theory, the ideal compiler should love bit-fields because they give it maximum information with which to optimize loads and stores. I see a lot of people claim that some compilers generate bad code for bit-fields, but that is a possibility with any construct you use. From what I can see GCC generates very good code for bit-fields.

So my conclusion is that the hurdles of bit-fields are tractable, and unless some new information comes my way, I plan to move forward with my plan to use them.

I had no such luck with pbstream. The tagline is currently:

pbstream - a stream-oriented implementation of protocol buffers.

It’s true that offering streaming semantics is one distinguishing feature of my implementation, but the way it’s growing it’s no longer really “stream-oriented.” It will offer both stream-oriented and structure-oriented semantics. Both will be equally supported/encouraged — it will be a matter of what best suits your needs.

If anything, the number one distinguishing feature of my implementation is that it is minimal. It gives you a set of tools to use whatever paradigm is the right trade-off for you. It gives you building blocks to assemble as you see fit. There are never “riders” — things like memory management that you have to take along with the parts you really want.

So what’s in a name? Desired characteristics:

• Unique, reasonably Googleable.
• Communicates that it is a protobuf implementation, and if possible communicates the philosophy described above.
• The name (or a reasonable abbreviation thereof) makes for a nice prefix that can be appended to all my identifiers. Right now it’s pbstream_this and pbstream_that.

Originally I just called it “pb” but that was a tad bit too generic. I slightly like “pblab” (communicates that it gives you building blocks for forging your own strategies), but I dislike the connotation that it is unfinished, unpolished, not production quality.

Perhaps I could give it a name, but keep using “pb_” in my identifiers. It could also help me organize the names of the different parts. I could use names like:

• pb_stream_parser_* for the stream parser.
• pb_struct_* for the code that defines an in-memory structure.

I kind of like that. But I still need a name for the package itself that is better than “pb”. Ideas?