• The download link is on the Gazelle webpage.
• Read about what’s changed in the Release Notes.
• Read the latest version of the manual online, which has a “tour” section walking you through some simple examples.
• Post any questions or comments about the software to the gazelle-users Google Group.

To me, Gazelle 0.2 represents a significant shift. With 0.2, Gazelle is finally in a place where I think it’s ready for people to tinker with. In 0.2 there is enough documentation to figure out what’s going on, and the command-line programs like gzlparse have reasonable –help messages and can do useful things. Starting with Gazelle 0.2, your problems are my problems: things that don’t work right should either be fixed or be written down as TODO items.

Gazelle 0.1 to 0.2 was a major overhaul — implementing LL(k) lookahead took major surgery. Half the time the code didn’t actually work, because major rewrites were only partially done. I expect all that to change with Gazelle 0.2 — I want future releases to be far more incremental, and for every commit to leave the repository in a working state. I want a 0.2.1 and 0.2.2 that fix a lot of the edge cases that still aren’t right in 0.2 (you can read more about these shortcomings in the “Tour” section of the manual or in the TODO).

There are still many major features to add to Gazelle in the future — you can see a list in the TODO. But again, I think these can be added without breaking the tree in the meantime.

There’s one major bummer about 0.2. I had to completely remove the “@ignore” feature, which was Gazelle’s answer to letting you ignore whitespace/comments without having a separate lexer. I removed it because I realized that the abstraction I had invented for expressing this concept was not quite right, and that a more general-purpose abstraction was the right answer — the abstraction I have in mind will also handle things like languages embedded in other languages (like Ruby inside HTML: RHTML). But the bummer is that for the moment, Gazelle has no answer for how to ignore whitespace/comments. So it’s clearly not useful for real work yet.

So try it out and send any feedback you have to gazelle-users. Thanks!

His blog entry basically makes fun of Cisco for inventing/releasing another RPC system. It’s not clear exactly what he thinks they should have done instead. What is strange about this criticism is that tons of technology companies have developed their own RPC system — Facebook and Cisco publicly, and other technology companies I am familiar with in a not-so-public way. Guess what: large commercial distributed systems are built largely on RPC. Is he arguing that all of the engineers at these companies simultaneously got the bad idea of investing in something they don’t need? If RPC is such a bad idea, then why is everybody doing it?

“Everybody’s doing it” obviously isn’t a justification alone, but it definitely puts the onus on the person making the critique to show why it’s a bad idea. I got a better idea where he was coming from when I read the mailing list post. Here’s the heart of his argument:

the fundamental problem is that RPC tries to make a distributed invocation look like a local one.This can’t work because the failure modes in distributed systems are quite different from those in local systems, so you find yourself having to introduce more and more infrastructure that tries to hide all the hard details and problems that lurk beneath. That’s how we got Apollo NCS and Sun RPC and DCE and CORBA and DSOM and DCOM and EJB and SOAP and JAX-RPC, to name a few off the top of my head, each better than what came before in some ways but worse in other ways, especially footprint and complexity. But it’s all for naught because no amount of infrastructure can ever hide those problems of distribution. Network partitions are real, timeouts are real, remote host and service crashes are real, the need for piecemeal system upgrade and handling version differences between systems is real, etc. The distributed systems programmer *must* deal with these and other issues because they affect different applications very differently; no amount of hiding or abstraction can make these problems disappear.

Finally something we can agree on! Yes, on a network shit happens, and no sane RPC system will try to hide this from you.

But then again, I don’t know of any RPC system that tries to hide this from you except possibly CORBA. Maybe there’s a horrible history here I don’t know about, but no RPC system I have ever encountered tries to hide from you the fact that on a network, shit happens.

So what are his other criticisms?

RPC systems in C++, Java, etc. also tend to introduce higher degrees of coupling than one would like in a distributed system. Typically you have some sort of IDL that’s used to generate stubs/proxies/skeletons — code that turns the local calls into remote ones, which nobody wants to write or maintain by hand. The IDL is often simple, but the generated code is usually not. That code is normally compiled into each app in the system. Change the IDL and you have to regenerate the code, recompile it, and then retest and redeploy your apps, and you typically have to do that atomically, either all apps or none, because versioning is not accounted for.

Yay, we can agree again. RPC systems that make you do an “all at once” upgrade are a bad idea. But again, no RPC system I have encountered makes you do this. Does this mean that the RPC system guarantees for you that the old and new protocols are compatible? Of course not — you don’t want your framework to be some big “I know what’s best for you” mommy that does really expensive things to solve this problem, like loading both versions of your code at the same time. But any RPC framework worth its salt makes it possible to have different interface versions interoperate. Adding a new parameter? No problem, old servers simply won’t see it. Completely changing the semantics of your call? No problem — just give the new call a new name.

Steve’s criticism amounts to “sucky RPC systems suck.” Yes Steve, yes they do. But a lot of the technology world is running on non-sucky RPC systems, and from time to time you get a glimpse of that when a company like Facebook or Cisco releases their internal RPC system to the outside world. Did Steve check to see if Cisco’s new RPC system is subject to any of his critiques? I haven’t, but I would suspect it isn’t.

• complete Strong-LL(k) lookahead support. (it’s not 100% complete yet, but it’s definitely solid enough for a 0.2 release)
• a command-line compiler program (gzlc) that takes reasonable options and is simple enough to use by reading its –help
• a “tour” section for the manual
• a command-line program (gzlparse) that can output the parse tree in a useful format, so you can see how Gazelle parses your input text.
• a test suite, so that when people report bugs I can add the bugs to the test suite and not regress.
• (stretch): make Gazelle self-hosting, so that the parser is more robust and easier to understand than the hand-written recursive descent parser I’m currently using. I don’t want people to have to deal with corner-case parser bugs.
• a way to visualize grammars, to spot-check them against your expectations

It’s the grammar visualization that I forgot to include. I mentioned parse tree visualization a few blog posts ago, but this is different — one is visualizing how a bunch of text got parsed, the other is visualizing the grammar itself.

It still has room for improvement, but here is what my grammar visualization currently looks like for JSON. You can see an NFA for each one of your rules, a DFA for each state of lookahead, and the DFAs that do the lexing.

The latest code from Git (note that I recently moved from repo.or.cz to Github) can generate these grammar dumps — just pass ‘-d’ to gzlc.

Observation #2: we will very soon have more cores than we know what to do with.

Prediction: fully concurrent garbage collection is the future of automatic memory management. I’m talking garbage collectors that run in other threads and clean up after me without ever stopping me in the middle of what I’m doing.

It will almost certainly be more expensive in terms of total CPU time, and probably can’t be as aggressive in terms of what it can reclaim at any point in time, but for most applications the latency guarantees will far outweigh.

Discuss.

Is this really the boilerplate necessary for creating even the simplest thread in Python?

This is making me miss Ruby, for which the equivalent is:

P.S. Gazelle 0.2 is making a lot of progress, but unfortunately won’t hit the 1 month mark I hoped for. Surprise surprise. But when it does come, it’s going to be awesome.

While you’re sitting tight waiting for Gazelle 0.2, I have a challenge I’m putting to my readers. I want my program gzlparse to parse some input text and output the parse tree in some useful format. What is the most useful format for visualizing a parse tree?

I want both a good text-based format and a good graphical format, if possible. For text formats there’s:

• XML (ugh. I didn’t want to say it, but I knew someone else would if I didn’t first. I’ll probably support it, but I’ll put ambivalent emoticons in the source code).
• S-expressions. Maybe I’ll win over some LISPers.
• ??

A good useful text format would be nice, but a good graphical format could be groundbreaking. I could always draw it as a tree, but I’m wondering if there isn’t something better. Something that keeps the text in its original format, but uses color or borders or something like that to represent the parse tree structure.

Here’s an example of the kind of visualization I think is really great and innovative. It’s the way that Lurker displays an email thread:

What’s so brilliant about this view is that it shows you both time-order of the messages and complete threading information in an attractive way. Of course, this is simpler than a parse tree, and such a nice view of a parse tree might not be possible. But what I’d really love to see is a parse tree visualization that:

• kept the original text recognizable (viewing it purely as a tree throws away the original text formatting completely)
• shows the parse tree structure somehow
• major bonus: can be rendered in a browser using a DOM. like, would allow me to write JavaScript to create this DOM inside the browser.

If this were possible, then I could write a web-based syntax analyzing text box that parsed your text as you typed it and showed you beautiful graphical representations of the parse. Something like this extremely awesome interactive regex visualizer, but for full context-free grammars. Or something like what ANTLRWorks provides, but on the web.

That would be SO HOT.

I got a little scared when people said they were going to start trying Gazelle out now, because it immediately made me think of how many things I’ve been meaning to fix before I unleashed it on anybody else! But on the other hand, it gives me all the more motivation to get it to a point where other people can try it out. And I’m never more motivated or get as much done as when I know people are waiting for me!

So here’s my line for the moment. Don’t try out Gazelle just yet. There are too many things for me to fix at the moment that I know are broken. But I want to fix those things ASAP and get Gazelle 0.2 out the door, so I can finally have a release that I can recommend people try out.

When will Gazelle 0.2 come? I’m hoping no more than a month. Here’s the target feature set:

• complete Strong-LL(k) lookahead support. I have the code to generate Strong-LL(k) lookahead, I just need to support this at the bytecode and runtime stage.
• a command-line compiler program (gzlc) that takes reasonable options and is simple enough to use by reading its --help
• a “tour” section for the manual
• a command-line program (gzlparse) that can output the parse tree in a useful format, so you can see how Gazelle parses your input text.
• a test suite, so that when people report bugs I can add the bugs to the test suite and not regress. this will be important for keeping my sanity.
• (stretch): make Gazelle self-hosting, so that the parser is more robust and easier to understand than the hand-written recursive descent parser I’m currently using. I don’t want people to have to deal with corner-case parser bugs.

Think of it as “the anti-fluff piece.” And as a bonus, when Gazelle is actually ready for general consumption, it will have a great manual ready-to-go.

I’m strongly considering porting Gazelle from Lua to JavaScript. I haven’t fully decided whether I’ll do this, and to be honest I’m not terribly thrilled about the switch. I’m going to lay the case for switching out and leave it to you, my dear readers (all 3 of you) to weigh in about whether this is a good idea.

This isn’t just about Gazelle — it’s really addressing the bigger question of what language best fills the niche I’m going after. I would describe the niche as: “small, fast, flexible language for embedding.”

Sounds just like what Lua was designed for, no? Indeed, Lua has been something of a dream to use. It’s fast, flexible, easy to learn, easy to embed, tiny, and has a tiny JIT available (disabusing me of the notion that JITs can only accompany bloated monstrosities like the JVM). So why switch from Lua?

Three main reasons. The first is that, as Steve Yegge noted when he explained why he ported Rails to JavaScript, JavaScript is the most accepted language of its type inside Google, and I want to see Gazelle take on a life of its own inside Google. I want to be able to write tools using it and have at least the slightest chance that my colleagues will take it seriously. If the imperative language that accompanies Gazelle is JavaScript, there will be an element of familiarity, and hopefully it will be possible to find someone to hack on it with me. Lua is unknown in comparison.

The second reason is much like the first: JavaScript is just more familiar to programmers in general. People know it from web browsers (even if they still think that the language itself is to blame for their browser compatibility nightmares). It is based in “curly brace syntax,” which gives Java, C, and C++ programmers warm fuzzies. It’s kind of lucky that a language as totally decent as JavaScript is set to become one of the most widely used languages, at least for the next several years.

The third reason is that it gives me a strategy for making Gazelle work on the JVM. There isn’t any real Lua implementation on the JVM, and Java heads HATE anything that uses JNI to talk to the outside world (because then it isn’t “100% pure Java”). But Rhino, on the other hand, is a pretty reasonable JavaScript implementation and it is 100% pure Java. So using JavaScript gives me a good JVM portability strategy.

So why am I less than thrilled about making the switch? The main problem is that there aren’t any JavaScript implementations that are remotely as nice as the Lua interpreter. Lua is idyllic.

• the entire Lua interpreter download is 200Kb
• it compiles in 4 seconds on my desktop
• it compiles to a 200Kb executable (130Kb stripped)
• it runs “Hello, World” in 3ms (startup time is nearly 0)
• it runs nontrivial benchmarks among the fastest of the scripting languages
• it has a JIT available that adds only 32Kb of code (compiled) to the core, and compiles most functions in microseconds (very low overhead and memory footprint)
• it has a really good and well-documented embedding/extension API

I believe there is a profound benefit to making software as lightweight as it can possibly be while still accomplishing its task. Most software fails miserably by this criteria. I often think of Pascal’s quote “I would have written a shorter letter, but I did not have the time” and think how much better off we would be if software developers had the same attitude. Lua is one of those rare jewels that takes this to heart. But let’s take a look at Tamarin, which is the favored next-gen JavaScript implementation. This is from the documentation about it’s garbage collector:.

MMgc is not only a garbage collector, but a general-purpose memory manager. The Flash Player uses it for nearly all memory allocations.

Oh fantastic! Because definitely the one thing that my application doesn’t already have is a memory manager. I am so glad that embedding JavaScript into my program is going to drag in a 15,000 lines of code implementing heaps, spin locks, memory barriers, and complicated C++ macros. Couldn’t you have taken the time to write a shorter letter — err, language implementation? Does JavaScript really need that entire 15,000 line memory manager? Let’s get some perspective on what you can do in 15,000 lines:

• Lua is 14,000 lines total. That includes the lexer, parser, vm, compiler, byte code format, extension APIs, all of the standard libraries, and the garbage collector.
• Gazelle is currently 5000 lines total. That includes the parser, LL lookahead calculation, NFA construction, NFA to DFA conversion, DFA minimization, code to read and write byte code in Bitcode format, and code to do the actual parsing.

If you add the code in Tamarin core, you’re now up to 70,000 lines of code. Add in the regular expression library and you’re up to 95,000. Sure, JavaScript as a language isn’t as minimal as Lua, but is it really 6 times more difficult to implement?

I know that Tamarin is a gift from Adobe, and that I shouldn’t kick a gift horse in the mouth. Their software from which they have taken Tamarin is probably happy to use MMgc everywhere. I just wish that our profession gave greater value to the virtue of brevity.

You might write me off as a sad, strange man to care so much about this, but are you so willing to write off Steve Yegge? Code’s worst enemy is size, he says. So there! Proof by Steve Yegge.

Back to JavaScript and Lua. I think JavaScript is a fine language for my porpoises. I just wish there was an implementation of it that was as good as Lua. Size, speed, flexibility, choose any three. No? But Lua does!

I’ve looked over all of the JavaScript implementations listed on Wikipedia. Spidermonkey is small-ish and flexible, but slow. Of the other implementations I like NJS the best (but it looks unmaintained), and SEE looks ok too. But none of these is nearly as nice and small as Lua. It will be hard to let it go.

Buffalo approaches the question from an educator’s perspective:

The real million dollar question in my mind, Josh, is what are these super-programmers doing that others are not? Even if there’s just something irreproducible in their genes - shouldn’t this be the sort of thing we could detect somehow? Or if there are strategies involved, could we use them to make the 90 percentile 5x better than everybody else?

[…]

Which of course brings me around to my research - say you had a group of the 50 cs students who enter an average program around the country. And say your goal was to get absolutely as many of them as possible into that 5% and screw everything else. What would you say to them? What would you do with them?

I guess it’s depressing to put this way, but my true belief is that programmers are born, not made. So I think the absolute number one thing a CS education can do for its students is help them understand whether they were born programmers or not.

This isn’t a binary thing, of course, and I don’t even think it’s single-dimensional. Everyone has areas of strength and weakness. And it’s a big world out there — succeeding at Amazon or Microsoft or Google is different than succeeding in the consulting world or at a startup or the technology department of an unrelated industry. I know at least one person who is a decent programmer but is doing quite well by combining good business understanding and fantastic people skills with his competent but not outstanding technical background. There are a lot of paths for people to take.

Then there is the question of what productivity is. In a corporate setting, productivity is making progress toward keeping your customer happy, whoever your customer may be and however bizarre their wants. Well I doubt that writing a self-compiling C compiler or figuring out how to calculate pi was at the top of any customer’s wishlist. Fabrice Bellard, for all his badassery, could end up being fairly unremarkable when you put him in a cube and tell him to write web applications. Or he might find it excruciating and spend the whole time inventing a framework that lets him write web apps the way he thinks. After all, that’s what Paul Graham did.

So disclaim, disclaim, disclaim. My point so far is just to highlight what is probably already obvious: that the landscape of talent is complex and multidimensional.

That said, I think it’s really key that students understand their strengths and weaknesses as early as possible. It will help them decide whether CS is right for them, and if it is what direction they should go within CS. And the best way to achieve this understanding is to expose students to lots of different things. As a bonus, this is also exactly what the overachievers need and want too; more problems to feast their minds on. So if I could sum it up in a word, I think the number one thing that university can do is expose young programmers to lots of topics and give them feedback about how they do in each one.

As to how we can help budding programmers be in the super-productive elite? I’m somewhat hesitant to answer that question, because I feel the answer reveals what the answerer thinks makes him/her a member of this elite class. But my biased answer is that the #1 most valuable trait a programmer can have is resourcefulness. It’s knowing what’s out there — what tools, strategies, file formats, blogs, or best practices — and knowing how they apply to the problem in front of you. If some programmers truly are outperforming others by 20x, it’s not because they’re typing 20x faster — it’s because they have such a solid understanding of the problem’s context that they have laser-like focus on the shortest possible path between them and their goal.

On the flip side, I think the worst thing that can happen for programmers is for them to get caught in their own little worlds where they only know one way of dealing with problems. Java-only programmers are the easiest to pick on example of this vice, but not the only one. You can probably get by only using/liking Java if you work at an all-Java shop where you are a cog in a machine, but you’ll never have any perspective of the big picture. I think we all fall prey to getting comfortable with our favorite languages/tools and thinking of everything through those lenses, but the more we can avoid that, the better we do IMO.

So that’s my answer: show students what’s out there, and encourage them to be resourceful. If there is any trace of programmer in them, it will come to life and demand that you feed it more. Anyone who doesn’t self-motivate at that point is not destined to be a programmer. They might not have the talent, or they might have the talent but not the motivation; either way, they probably should find something else instead.