Posted by josh at July 25th, 2007

My most recent posts have been far too constructive and boring. A fun little rant is overdue. And Con Kolivas’s Interview “Why I quit”, which is hot on the net right now, is the perfect subject matter for doing so.

Con Kolivas is a Linux Kernel hacker who’s had it with the Linux development process. He’s seen his good ideas repeatedly get stonewalled, then assimilated into competing patches from more senior Linux developers. That reeks of kernel development stagnating into an old boys club, but that’s not really what I’m here to talk about today. I want to write about my resounding agreement with Con’s opinion of desktop computing today. From the interview:

[T]he desktop PC is crap. It’s rubbish. The experience is so bloated and slowed down in all the things that matter to us. We all own computers today that were considered supercomputers 10 years ago. 10 years ago we owned supercomputers of 20 years ago.. and so on. So why on earth is everything so slow? If they’re exponentially faster why does it take longer than ever for our computers to start, for the applications to start and so on? Sure, when they get down to the pure number crunching they’re amazing (just encode a video and be amazed). But in everything else they must be unbelievably slower than ever.

I’ve been saying this to anyone who will listen for over a year now (and all my friends are tired of it and give me the hand whenever I start). But I want to take just a moment and spell out why I think current operating systems AND user-space (GUI frameworks and such) suck so badly on the desktop.

GUI frameworks (and the applications that use them) suck

Here’s how 99% of desktop applications today work. There’s an event loop in the app’s main thread. It works something like this:

while(1) {
  event = GetNextEvent();
  if(event.type == NEED_REDRAW) do_redraw();
  else if(event.type == USER_CLICK) process_user_click();
  else if(event.type == DO_SOMETHING_SLOW) something_slow();
  ...
}

In your run-of-the-mill application, the app will happily sit there doing something_slow() in the main thread (which could involve I/O to the disk or network, or just something CPU-intensive), blissfully ignoring USER_CLICK events, NEED_REDRAW events, etc. until it finishes. It could even be something as benign-seeming as a malloc(), which in stock implementations will grab a process-wide mutex and trounce around the heap reading bookkeeping information (which, mind you, could trigger page faults and force pages to be swapped in).

And while this is happening, what do you the user get? That’s right, the spinning beach ball, which I think would be more accurately rendered as a hand flipping you the bird, because that’s the message the OS is sending you at that point. You as a user are totally shut out at that point; your only option is to wait for it to subside or to kill the process completely.

Even if the something_slow() task doesn’t take long enough to show the beach ball, it’s still a delay before the UI can be redrawn. Delays like this are the cause of stuttering, which makes a UI feel sluggish even if the UI is perfectly caught up every time it does redraw.

I know what you’re thinking. “OK, so some applications suck and aren’t multithreaded. What’s your point?” The point is that we can’t pin the on application authors — I think the fault lies squarely with the GUI frameworks. Sure, if you’re really good at threading and invest a lot of effort, it’s possible to write applications that never ever block the event loop, but does the framework make it easy or provide the tools for easily doing so? Mainly, does the framework provide a threading model that is well-supported by the framework?

Only one GUI framework that I’ve ever encountered does this: BeOS. Notably:

Every BWindow object spawns a thread (in the application’s address space) where it receives and responds to interface messages from the server. –The InterfaceKit Intro

The programmer doesn’t have to be diligent, the programmer doesn’t have to have experience with the threading library; but automatically, without even trying, the app gets a separate thread for every window that is distinct from the main event loop. That means that even if the event loop spends some time doing something expensive, the GUI will continue to redraw, resize, render button clicks, etc. just as smoothly as if nothing were happening.

That’s a big part of what made the BeOS GUI faster on late 90s hardware than Windows, Linux, OR OS X are on today’s hardware.

So problem #1 is that today’s GUI frameworks suck. They let the GUI become unresponsive far too often by failing to isolate the UI’s responsiveness from long-running work that the application should be doing in the background.

Neither OS’s nor applications have any freaking clue how to do real-time

Real-time computing is one of my favorite topics, because nothing else brings out so many self-proclaimed experts regurgitating the BS they learned when their OS class spent two days studying it. (I never took an OS class and therefore clearly cannot be such a person, so ha!) Such people don’t think a process is “real-time” unless a missed deadline will launch a nuclear missile or stop somebody’s heart. I’m not kidding: here’s an example of the sort of thing that gets modded +5 on Slashdot when real-time computing comes up. Someone asked the question “would new real-time features in Linux help audio/video playback in Linux,” and the +5 modded response is:

Desktops and most servers do not get any benefit from a RTOS. RT makes it so that the system purposefully downgrades less-useful things like user input for maximum priority things like, say, polling a fetal heart monitor every n milliseconds or responding to an automobile collision to deploy an airbag.

Please, don’t fall for people like this who have such a shallow understanding of what real-time means.

Multimedia playback is absolutely real-time: it has deadlines, and the system will malfunction if those deadlines are missed. “Hard” vs. “soft” real-time is a pretty meaningless distinction; it’s an arbitrary and uninteresting way of classifying how bad it is to miss a deadline.

Now that we have that out of the way, we can move on to how current OS’s fail to provide the tools to write reliable RT applications (like, say, a movie playback application).

A movie playback application’s real-time needs are so very simple. Say I’m the application:

  • I need to read the next chunk of movie data from disk in time to feed it to the decoder.
  • Once I have the data, I need to decode it in time to feed it to the graphics card.

Every time your movie playback stutters, it’s because the OS failed to satisfy these two simple needs. Let’s look at them one at a time.

The first need is a disk I/O deadline. The app needs to read a chunk of data (of known size, or at least known upper bound) by a certain time. In other words, the OS needs to allow the movie player to reserve a certain amount of disk bandwidth, and guarantee that it got first dibs on that bandwidth. I was about to claim that no commercially-offered operating system provides this, but I was just surprised to learn that apparently it has been added to Vista. Let me save you the suspense: Linux doesn’t have it.

The second need is a CPU deadline. The app needs a fixed amount of CPU time by a certain time, or it won’t be able to decode the video in time.

In the bad old days before Linux got PREEMPT_RT (which was not very long ago), a leading reason why this part would fail was simple scheduling latency. Say the movie player is waiting for the short window of time when it needs to blast the new frame into the framebuffer. The time comes, but…! the kernel has unfortunately acquired a lock or disabled interrupts or done something else uninterruptible. My movie player, even if it is the highest priority task in the whole flipping system, won’t run until the kernel finishes what it was doing. By that time it may be too late.

The single accomplishment of the PREEMPT_RT patch is to make the two highest scheduling priorities in the system (SCHED_FIFO and SCHED_RR) actually work as advertised: processes with these priorities will actually, truly be run as soon as they are runnable, without being held hostage by obliviously latent routines in the kernel.

But do you know what’s missing from The PREEMPT_RT overview? Any mention of deadlines. In other words, there is no facility for telling the OS what a process’s deadlines are, and therefore no intelligent way for the OS to pick between scheduling two real-time processes that are both runnable. Sorry folks, but without deadlines, it ain’t real-time.

OK, so there’s one little exception. If you only have one real-time task in the system, and that ONE real-time task is scheduled SCHED_FIFO or SCHED_RR, then it will always be run when it’s runnable, and will therefore make all its deadlines provided there is enough CPU time available for that to be possible.

But TWO real-time processes? The best you can do is given them equal time using SCHED_RR, but that will fail in many cases that a proper real-time scheduler would cover.

Also remember that a process scheduled SCHED_FIFO runs to the exclusion of absolutely everything else. You’d better hope that it doesn’t get into an infinite loop, or it will lock your entire computer. A true real-time scheduler could allow real-time tasks only a limited amount of highest-priority time per period.

So problem #2: existing operating systems don’t give programmers the tools they need to write good RT applications. As a result, it is impossible to write something as simple as a movie player that will never glitch.

No one realizes it’s all about I/O

Everyone continues to be fixated on CPU cycles as a measure of speed. But I/O is an ever-present bottleneck that no one really talks about. Think about it:

  • every cache miss is a pipeline stall. so yes, binary size does matter: 32k of L1 and 4MB of L2 don’t go very far.
  • every time you page fault on a page that isn’t in memory, your application blocks on a disk read. fat lot of good your SCHED_FIFO priority does now!
  • even for meticulously-written applications, these days you just gotta move a lot of data around! a user’s photo library or music library could be hundreds of megabytes in size, and they shouldn’t have to wait just to scroll through it all.

There’s more I could write about this subject, but this entry has gotten pretty long already. The gist of it is that few OS’s or applications recognize the integral part that I/O should play in the overall design. For example, an OS should know better than to schedule a process that is swapped out — such a process can’t do a thing until it’s swapped back in. But it’s beyond the scope of most scheduler designs to take a thing like that into account.

Scheduling of non-RT tasks sucks sucked

I won’t belabor this point, because apparently the Staircase Deadline (SD) Completely Fair Scheduler solves this problem. But seriously, was it that hard to figure out that a scheduler should aim to give equal-priority processes an equal amount of CPU time? Just sayin.

Closing remarks

In summary, I agree with Con Kolivas about how sub-par an experience the Linux desktop of today is. I really want to put my money where my mouth is and write something better (and I have some plans for doing just that), but first things first: parsing.