Comments on: LLVM Bitcode vs. Protocol Buffers

By: Ismael Juma

Ismael Juma — Mon, 09 Mar 2009 00:52:55 +0000

Hi,

“Space-efficiency in protocol buffers is horrible for repeated-fields. Thus arrays are a problem and the only work around is store a binary blob.”

Note that protocol buffers now supports a packed representation for repeated fields of primitive types:

http://code.google.com/p/protobuf/source/detail?r=92

Ismael

By: josh

josh — Tue, 13 Jan 2009 07:17:44 +0000

@Chris K: yes, the fact that all values are tagged with a key in protocol buffers does give it nice forwards and backwards compatibility. It’s also true that if you’re using repeated fields as an array you’ll pay 1 byte per array element, which could be as bad as 100% overhead (if your array is bytes) or as low as 12% overhead (if your array is int64s).

@simon: I want to stay 100% C so that the binaries stay as small as possible (currently 20k compiled) and impose as few requirements on the host platform as humanly possible. You can call C from anything (like even Objective C or LLVM Assembly Language). To me staying 100% C is a no-holds-barred way to be as useful as possible in as many weird situations as possible.

@Dave: Cool, though my intention is to write it in such a way that there is no code generation, so that it can be used by bindings to higher-level languages like Perl, Python, Ruby, etc. So my intended design differs a little from the existing C implementation.

@Eric: thanks man. I had fun at Amazon and definitely remember coming across your name in association with good things!

By: Eric

Eric — Tue, 13 Jan 2009 01:29:57 +0000

So that’s what you’ve been up to? Amazon could use more folks like you, too bad we lost ya.

By: Dave Benson

Dave Benson — Mon, 12 Jan 2009 19:21:42 +0000

there is a c binding for protocol buffers now: btw: http://code.google.com/p/protobuf-c/

By: simon francis

simon francis — Mon, 12 Jan 2009 17:27:42 +0000

Why did you want to stay 100% C?

By: Chris K

Chris K — Mon, 12 Jan 2009 14:42:54 +0000

I wrote a Haskell implementation of Protocol Buffers and I know nothing about Bitcode.

The binary format for protocol buffers has one outstanding strength: it is backwards and forwards compatible for many kinds of changes in the specification. Usually one has to resort to a text format for this property or to a self-describing binary format.

Space-efficiency in protocol buffers is horrible for repeated-fields. Thus arrays are a problem and the only work around is store a binary blob.