This is very exciting for me, because the premise that a text editor could use a real parser as you type was one of my major motivations for creating Gazelle. Most text editors clever pattern matching for the syntax highlighting, but since they aren’t real parsers they can get confused. More importantly, they can’t give you true syntax-aware completion. The notable exception is Java editors like Eclipse, which do this quite well, and I hear that Microsoft’s IDEs are good at it too. But I always wanted an editor where you could add new languages easily and have more ability to tinker with how this syntax information is used. I think it could open up so many doors.

Given this, I was very excited when Rasmus Andersson from the Kod project wrote to the gazelle-users mailing list asking if Gazelle was a dead project, or if it was something he and his team should invest effort into integrating with Kod. I replied that it was definitely not a dead project, but that it is still subject to major change. That didn’t seem to dissuade him though, and I look forward to learning from their needs and folding their improvements back into Gazelle.

However, if I’ve learned one thing over the last several years of working on Gazelle and upb, it’s that I am extraordinarily bad at knowing how close I am to being ready to release. I don’t want to make any predictions or promises. I honestly thought I was almost ready to release upb a year and a half ago, but I’ve almost completely rewritten it twice since then. Each time it gets significantly better and closer to being what I want it to be. Once I have a release I’ll describe more about the stages of evolution it went through and how each iteration was objectively better than the one before it.

The core interfaces have gotten to a point where I’m really happy with them and feel no more need to rework them. I think they will continue to evolve incrementally, but not in a way that requires redoing them completely. Here are the most core interfaces — if you’re interested in upb, I recommend reading these headers. I’ve just added substantial comments to explain them, and more than anything these will give you a taste of what upb is all about:

• upb_stream.h, the streaming interfaces for doing SAX-like tree traversal of protobuf data, and abstractions of fread()/fwrite(). These are probably the most important interfaces in all of upb, since all of the encoders and decoders are based on them.
• upb_string.h, an immutable, length-delimited (instead of NULL-terminated), reference-counted string type.
• upb_def.h, the data structures for a protobuf schema (.proto file) and routines for loading them.

By the way, I also mean to write zero-overhead C++ wrappers around the above to give you C++ programmers a nicer interface at no cost.

With those set, I have been rapidly getting everything building/working again. It’s a bit annoying to rewrite upb_def.c for the third time (literally) but it feels good knowing the interfaces are right.

So I have renewed optimism that I’ll be releasing soon. And once I’m happy with upb, it’s back to Gazelle.

Intel has added a number of new instructions to Nehalem and it has sped up others. The 4.2 version of Intel’s SSE vector extensions takes the x86 ISA back to the future just a bit by adding new string manipulation instructions. I say “back to the future” because ISA-level support for string processing is a hallmark of CISC architectures that was actively deprecated in the post-RISC years; typically, when a writer wants to give an example of crufty old corners of the x86 ISA that have caused pain for chip architects, string manipulation instructions are what he or she reaches for. But the new SSE 4.2 string instructions are aimed at accelerating XML processing, which makes them Web-friendly and therefore modern (i.e., not crufty).

I chuckled a bit when I read this. I’m not very purist when it comes to hardware. If these instructions will make my parsers faster, then they sound great to me!

The four new instructions are:

• pcmpestri: packed compare of explicit length strings, returning index
• pcmpestrm: packed compare of explicit length strings, returning mask
• pcmpistri: packed compare of implicit length strings, returning index
• pcmpistrm: packed compare of implicit length strings, returning mask

The variants are as follows:

• implicit length strings are NULL-terminated, explicit strings have an explicit length (ie. the whole input register).
• they can return an index into the source string (if you were searching for something) or a mask (if you wanted to test each character of the input

Both let you scan a 128-bit SSE register (treating it as either 16 8-bit characters or 8 16-bit characters) and perform all kinds of searches/comparisons. The instructions are configurable; you supply a control word that specifies all of the different variations of the instructions. For example, are the input values signed or unsigned, are we comparing against ranges or specific values, etc.

The reciprocal throughput of these instructions is high (2 cycles) but the latency is annoyingly slow (9 cycles). This means that you have to wait nine cycles after issuing the instruction before you can use the result. It’s hard to think of too many useful things you can execute in parallel while you’re waiting for that answer. As a side note, these figures come from Intel’s Intel® 64 and IA-32 Architectures Optimization Reference Manual, which says that the latency number is a worst case estimate:

Actual performance of these instructions by the out-of-order core execution unit can range from somewhat faster to significantly faster than the latency data shown in these tables.

I’m not enough of a hardware geek to know what to actually expect.

Still, that’s nine cycles to wait before getting a lot of really useful information. In addition to returning the index or mask, the instructions set several of the flags in useful ways.

So what processors have SSE 4.2? Or in other words, how long will my impatient self have to wait to try them out? Apparently SSE 4.2 is available on Penryn, which is the second-gen Core 2, which debuted in 2007/2008. It uses a “45 nm process”, which I’m sure means something to hardware geeks but not to me. All I know is that it’s not the Core 2 that’s inside the MacBook Pro sitting on my lap. And of course SSE 4.2 is in the new Nehalem.

Next up for Gazelle 0.5: operator-precedence parsing and the ability to build parse trees. Following up on my earlier post about Bitcode and Protocol Buffers, I’ve wholeheartedly decided to go the Protocol Buffer route, not only for the bytecode itself, but also for parse trees and abstract syntax trees. More about this later.

I first had this idea when I visited the excellent regular expression reanimator. It’s this beautiful little visualization that runs in your browser and shows you how a DFA transitions in response to text. And all you have to do to use it is click that little link!

My dream is that you can develop your grammars interactively. You type rules and as you type you see little graphs build themselves up, state by state. You add a kleene star and you see the extra transitions add themselves to the existing graph. You put some sample text in another textarea and watch it start syntax-highlighting itself as you add the rules to properly recognize its structure. Or you can develop your own syntax highlighting color schemes by typing syntax highlighting rules and again seeing them incrementally take effect on a block of sample text.

None of these ideas are brand-new. ANTLR of course has ANTLRWorks, which is an IDE for developing ANTLR grammars. But I feel that having this on the web would really lower the barrier to entry and make it accessible to more people. And despite how people poo-poo JavaScript and HTML, the web is not such a bad platform these days — especially now that SVG support is getting really good. SVG is the perfect tool for drawing graphs, styling the nodes in various ways, and having them do useful things when you hover/click on them.

So anyway, I really want this. But the hurdles to making this happen are great!

• Gazelle’s compiler is written in Lua. No real way to run Lua in a browser right now. Possible options for doing this in the future: a Lua -> JavaScript compiler, a Lua VM written in JavaScript, porting the Gazelle compiler to JavaScript, running the Lua interpreter inside NativeClent, and running the Lua interpreter inside Alchemy.
• Gazelle’s runtime is written in C. Possible options: port the runtime to JavaScript, run it inside Alchemy, or run it inside NativeClient.
• The Graphviz graph layout package is also written in C. Possible options are the same as the Gazelle runtime. I was seriously considering porting just the layout algorithms to JavaScript and had even started the port, but have recently been realizing how CPU-intensive these algorithms are even in C. In JavaScript they could only be slower. With Alchemy they would be slower than pure C, though it’s hard to say how much. With NativeClient they would run at native speed.

So it pretty much looks like I’m stuck waiting for either Alchemy or NativeClient to be prime-time. While I could always run all this software on the server and AJAX the results in constantly, I don’t think that would provide a smooth enough user experience (not to mention the load it would put on my server — these algorithms are kind of expensive).

As far as Alchemy vs. NativeClient: Alchemy will be slower (2-10x according to them) since it compiles your C/C++ to the Flash VM instead of to machine code, and it will require the Flash plugin. NativeClient will run at roughly native speeds, but will require a plugin download (most people will have flash already) and one that has potentially scary security implications if the NativeClient guys made any mistakes.

Time to play with Alchemy and see if I can get it to work! And hope that NativeClient makes a lot of progress in no time!

I’m not proud of this. I’d like to take the opportunity to make a commitment to putting those days behind me. I am extremely proud of Gazelle and what I think it brings to the table, and I’ve tried to consistently acknowledge my debts to prior work, especially ANTLR. Indeed, in the Gazelle manual I say:

Gazelle’s algorithm takes a great deal of inspiration from ANTLR, the LL(*) parser generator written by Terence Parr. While Gazelle seeks to augment the theory that Terence developed, the foundations of Gazelle are strongly based in ANTLR’s achievements. Like ANTLR, Gazelle is a top-down parser that models its lookahead as possibly-cyclic state machines. Gazelle’s algorithm for generating this lookahead is based on ANTLR’s algorithm.

Still, I clearly need to be much more careful not to jump to conclusions, and be less susceptible to bravado. If Gazelle is as good as I think it is, its capabilities should speak for themselves.

1. when Gazelle encounters a parse error, it should report it to the API in a sane way instead of doing assert(false) or just printing a message to stderr and aborting
2. Gazelle should provide line and column information to the API (this is more complicated than it might sound — read the Wikipedia article on “newline” and the Unicode recommendations on Newline for some of the complications)
3. Gazelle’s C API should be namespaced (with gzl_ or somesuch on all the identifiers) instead of polluting the global namespace with functions called things like “parse”
4. a “make install” target, so that you can just type “gzlc” for the compiler instead of “. lua_path; ./compiler/gzlc”

These sound like a good set of goals for a 0.4 release. I may drop a few or think of a few others.

Whitespace handling was for a long time the biggest thing preventing Gazelle from being actually useful. I had a whitespace-handling mechanism back in 0.1, but had to rip it out in 0.2 because it was naive and premature. It’s back now, and while it looks mostly the same from the user’s perspective, it’s totally different under the hood.

Here’s the current picture of a JSON grammar, using the new whitespace facility:

// Grammar for JSON, as defined at json.org.

@start object;

object   -> "{" (string ":" value) *(,) "}";
array    -> "[" value *(,)              "]";

str_frag -> .chars=/[^\\"]+/ |
            .unicode_char=/\\u ([0-9A-Fa-f]{4})/ |
            .backslash_char=/\\[ntbr"\/\\]/;
string   -> '"' str_frag* '"';

number   -> .sign="-"?
            .integer=/ 0 | ([1-9][0-9]*) /
            .decimal=/ \. [0-9]+ /?
            .exponent=/ [eE] [+-]? [0-9]+ /? ;

value    -> string | number | "true" | "false" | "null" | object | array;

whitespace -> .whitespace=/[\r\n\s\t]+/;

@allow whitespace object ... number, string;

It’s the @allow statement at the end that triggers the whitespace facility (which are more generally called “subparsers”). That statement means that any time the parser is inside an object (eg. all the time), but not inside a number or a string, whitespace can appear anywhere. More specifically, for all states in such rules, the @allow statement defines a “whitespace” transition back to the same state.

As a result, the rule graphs start looking like this:

All the whitespace transitions are a little bit annoying to look at, and I’ll definitely have to tweak the visualization to show them in a nicer way. But this gives you an idea for what’s going on.

What’s nice about this scheme is that you get to specify the whitespace rules in a simple way (“whitespace is allowed everywhere except in these certain rules”), but then the whitespace becomes a part of the same grammar data structure and is analyzed by the same algorithms as everything else.

It’s worth mentioning that this design is significantly different than the traditional design, which is to have a separate lexer simply discard whitespace and comments before they ever hits the parser. The lovely thing about doing it my way is that the whitespace becomes a part of the parse tree (if you want it to — you’ll be able to turn this off). This means:

1. round-trips from text -> parse tree -> text are lossless
2. therefore programs can operate on the parse tree to do whitespace-preserving transformations
3. programs can also analyze the parse tree with regard to how it follows style rules
4. programs like doxygen that consider comments and whitespace significant could operate on a Gazelle parse tree

See the Release Notes for what has improved since the last release. The gist of is it that the compiler is a lot more robust: it catches more error cases and will never hang due to errors in the input. There is also explicit ambiguity resolution, which makes it possible to resolve the ambiguous “if then else” case explicitly. And this is all accompanied by a large test suite, since this grammar analysis is difficult and subtle.

The biggest remaining problem that keeps Gazelle from being practical is still the lack of whitespace handling. And this is exactly what the next release will fix.

Here’s the problem. I’m at a point where I want to support prioritized choice in Gazelle grammars. Prioritized choice is a way of letting grammar writers explicitly resolve ambiguities that have crept into their grammar. Take the following grammar rule:

s -> "X" | "Y";

The two alternatives in this rule (X and Y) have equal priority. That doesn’t actually matter in this case, since no text can match them both, but now consider changing the rule a bit:

s -> "X" | "X";

Now there is a problem, because an X matches both alternatives. There are two legal parse trees that are both correct according to the grammar. As a result this grammar is ambiguous. A parser doesn’t have any reason to choose one over the other, because they are of equal priority.

The solution to the problem is to let the user specify which alternative should be taken when both match. If we allow users to do this by introducing prioritized choice, our solution looks very much like how Parsing Expression Grammars (also known as PEG’s) work. And the syntax that has become very standard in the PEG literature is to use “/” as the operator for prioritized choice. So our ambiguous grammar would be made unambiguous by using prioritized choice:

s -> "X" / "X";

Now the parser will always choose the first alternative, and the grammar is no longer ambiguous. You might wonder “what’s the point?”, since the second alternative will never be taken, but in many other cases of ambiguity, the choice of lesser priority will be taken. The classic example of ambiguity (which can be resolved in this way) is if-then-else:

stmt -> "if" expr "then" stmt |
        "if" expr "then" stmt "else" stmt |
        expr;
expr -> "true" | "false" | /[0-9]+/;

This grammar is ambiguous, because the fragment:

if true then if true then 1 else 2

…can be parsed in two different ways. The “else” could be assigned to either “if” — both are valid according to the grammar.

So I definitely know I will be introducing prioritized choice. The question is what syntax to use. Like I mentioned before, I would really really like to use “/” since there is a strong convention from the PEG world for it. But unfortunately this introduces a major ambiguity in Gazelle’s own grammar, because it conflicts with the syntax for regex literals. What does this mean?

s -> a / b;
a -> c / d;

Is that two rules that both use prioritized choice, or is it one rule that has a giant regular expression in it?

s -> a / b;
a -> c / d;

I really don’t want to give up using slashes to introduce regular expression literals, because that is a really strong convention that a lot of programmers are familiar with. But I also really don’t want to give up using the really strongly established convention of using slash for prioritized choice.

Anyone have an idea for a compromise I might like? The only one I can think of is adding a prefix to the regex literals sort of like Perl’s m{} form for regular expressions. It could be something like m/regex/. But I hate to add that uglification to the grammar, especially since regular expressions are going to be SO much more common than prioritized choice.

Any ideas, anyone?

Bonus: the other form of ambiguity resolution

Choice/alternation isn’t the only possible source of grammar ambiguity. Gazelle’s constructs for optional and/or repeating grammar fragments can also be ambiguous. The if-the-else example is perhaps more idiomatically written in Gazelle like so:

s -> "if" expr "then" stmt ("else" stmt)? | expr;

It’s written differently, but it’s just as ambiguous as before. So we need the same sort of tool to resolve this ambiguity — a way to say “prefer to match the optional part” or “prefer to NOT match the optional part.” This is equivalent to defining a “greedyness” to these operators like Perl lets you do in regexes. The syntax I am planning for these is:

s -> a b?; // no preference to match the b or not
s -> a b+; // no preference to keep matching b or not
s -> a b*; // no preference to keep matching b or not

non-greedy variants
s -> a b?-; // prefer to NOT match b (non-greedy)
s -> a b+-; // prefer to STOP matching b (non-greedy)
s -> a b*-; // prefer to STOP matching b (non-greedy)

greedy variants
s -> a b?+; // prefer to KEEP matching b (greedy)
s -> a b++; // prefer to KEEP matching b (greedy)
s -> a b*+; // prefer to KEEP matching b (greedy)

I’m not following the Perl convention here (Perl uses “?” instead of “-” to make the match non-greedy). But on the other hand:

1. Perl doesn’t have a way to explicitly make the match greedy — matches are greedy by default. So it has no convention for a “make greedier” operator.
2. not very many people know/use Perl’s greediness operator anyway

So to write the if-then-else example with explicit ambiguity resolution, you would write:

"if" expr "then" stmt ("else" stmt)?+ | expr;

…because when there is an ambiguity, we want to bind the else to the most recent if.

An aside: “great, so I can just start slapping greedy and non-greedy modifiers on everything, just to be safe, right?”

Not so fast there. Ambiguity resolution isn’t something you should take lightly, because a grammar with an ambiguity in it presents a user-facing irregularity in your language. For example, the if-then-else example is a user-facing issue: either “else” pairing makes logical sense, and it’s totally arbitrary how you decide to resolve the ambiguity. You have to tell your users about it. You should strive to absolutely minimize the numbers of such ambiguities in your language whenever you can!

The danger of using ambiguity resolution operators willy-nilly is that you don’t necessarily know if they’re actually resolving ambiguities. For example, if you wrote:

s -> "X" / "Y";

…the prioritized choice here is completely extraneous, since this grammar is unambiguous already. Because ambiguities are so important to keep track of and educate your users about, you don’t want to have to wonder whether a prioritized choice is resolving an ambiguity or not. You want to know that it is.

To this end, Gazelle will keep track of all ambiguity-resolution operators and error if any of them are never actually used to resolve an ambiguity. So you can’t just sprinkle them around “just to be safe,” you must only use them where they are effective. And this will make Gazelle grammars much more useful, because you will know exactly where all your ambiguities are.

By the way, Gazelle can’t detect any ambiguity in your grammar; calculating whether an arbitrary context-free grammar is ambiguous or not is undecidable. Basically, all grammars fall into one of several categories:

1. Gazelle can handle the grammar (ie. the grammar is in the green region of the diagram in my previous entry).
2. Gazelle can handle the grammar except for the fact that it’s ambiguous.
3. Gazelle cannot handle the grammar, and would still not be able to even if you resolved the ambiguities.

If your grammar falls into category 3, Gazelle doesn’t know whether the grammar is ambiguous or not. But for grammars in categories 1 or 2, Gazelle knows exactly where the ambiguities are.

Bonus #2: a word about Parsing Expression Grammars

Something you might say (especially if you’re a PEG fan) is “gosh Josh, if you’re going to all this work to introduce ambiguity resolution, why not just use PEG’s flat-out and abandon all this LL(*) nonsense?” Because essentially what I’m doing is supporting both PEG-based constructs (like ordered choice) and context-free grammar constructs like non-ordered choice. And PEG’s support a larger set of grammars than Gazelle ever will (I believe they support any non-left-recursive PEG, though I don’t have a reference for this).

There are two really major reasons why I don’t think PEG’s are the way to go. The first one is related to the point I was making before. With PEG’s all choice is prioritized choice. However, with PEG-based tools you have no idea if there are real ambiguities in your grammar or not. PEG’s are defined such that ambiguity does not exist, but this does not make if-then-else style issues go away, it just sweeps them under the rug. Even if you don’t call it “ambiguity,” it is still a user-facing issue that you as a language designer definitely want to know about! If you have the following in a PEG:

s -> a / b;

…you have no way of knowing if this represents an ambiguity, which means that you don’t know if changing the grammar to:

s -> b / a;

…will make a substantive difference or not! You’re flying blind.

The second reason is that Gazelle’s parsing algorithm will be far, far faster than packrat parsing (which is the algorithm used to parse PEG’s). Both algorithms will have linear asymptotic complexity (though as I briefly mentioned before, I believe that LL(*) grammars can degrade to n^2 in degenerate cases; though I will argue that this will never happen in any sane language). But though both are linear, packrat parsing’s constant factor is far higher, and packrat parsing uses O(n) space where n is the size of input.

This might all sound like hot air, but check out these real, hard numbers. On the homepage of the Aurochs parser generator, which uses PEG’s and packrat parsing to parse JavaScript, they say:

How much memory does it use?

An awful lot.

Grammar	Input size	Memory consumption	CPU time	Memory overhead
Javascript	140kb	380Mb	1s	2700
Javascript	71kb	180Mb	0.45s	2500
Javascript	1717	5.6Mb	0.01s	3200

I can’t vouch for whether this is a great implementation of packrat parsing, but the authors sure seem to know what they’re doing. I hope you’ll agree that 380Mb of memory just to parse 140kb of text is horrendous, and 1s is quite slow too. Gazelle can’t parse JavaScript yet, but here is a rough ballpark estimate of what the performance will be like to parse 140kb of text once it can:

• memory: Gazelle uses 24 bytes per entry on its stack, plus 12 bytes for every token of lookahead. These are the only non-fixed costs. If we assume a parse tree that goes 100 deep, and an input that requires 100 tokens of lookahead (both ridiculously large assumptions), Gazelle’s memory footprint is 3.6kb. This is 0.0009% of the packrat parser’s memory requirement.
• CPU: Gazelle currently parses JSON at 18.8Mb/s. JavaScript will be more complicated than JSON to parse, but Gazelle will also undergo a lot of optimization that hasn’t been done yet, so I’ll go with the 18.8Mb/s figure. That’s 134x faster than the 140kb/s Aurochs is quoting.

Like I said, the Aurochs authors seem to be very smart — I’m not saying they’ve done a bad job. I’m just saying that if Aurochs is any indication of what we can expect from packrat parsing, it is orders of magnitude slower, and takes enormous amounts of memory even for modest amounts of input text. If anyone has better numbers to show for packrat parsing, I’m all ears.

So even though it’s more work, I’m 110% convinced that using this combination of LL(*) with explicit ambiguity resolution is the way to go, and is the best way to achieve the goals that I have set out for this project.