cananian

We used Google Wave this past weekend during a practice for January's Mystery Hunt. It was an interesting experience, but Wave needs better group support before it's really usable for large-group collaboration. Further, wave threads with lots of discussion start becoming unwieldy: what usually ends up evolving is a collaboratively-edited document/summary/link forest on top, and lots of discussion and comments trailing off below. This forces you to keep scrolling between the "useful stuff" at top and the end of the comment thread, rapidly vanishing off below. The terrible scrollbar implementation doesn't help. This seems like a basic UI problem — and one that wikis have to some degree, too: anyone who's done any collaborative editing of a wiki knows the temptation to insert little inline comments, which spawn discussions that threaten to overwhelm the text. I suspect the solution is some sort of split view or toggle so you can keep an eye on both the discussion and the document at once — think of the little chat window integrated into a collaboratively-edited Google Spreadsheet, but with history and linking and all that wave goodness.

So, as presently constituted, the web Google Wave app is Not Quite There Yet (there are also some instability, compatibility, and slowness issues, but they seem more straightforward to solve). But the underlying Wave technology remains very exciting, and I believe it's something Sugar should build on, as I've written before.

The somewhat oddly-technical (and often slightly-wrong) middle section of the Chrome OS presentation veered off at one point about filesystem partitions (!) and how having a read-only partition is a novel feature of their OS. Read-only partitions are one of the oldest security mechanisms — my boot floppy had the write-protect notch taped over back when I was booting DOS — and a near-universal feature of thin-client deployments (which Chrome OS certainly is). OLPC maintained a mostly-read-only root, but primarily to extend the lifetime of the flash disk (flash lifetime was not touched on in the Chrome OS presentation). Litl mounts a read-only root with a writable unionfs on top, which actually works much better in practice: improved maintainability because all the various Linux system daemons can still "do their thing" as they expect, but you can wipe the top partition whenever you like to return to a clean state (at litl we do this on every update). (If you haven't hacked the lower levels of a Linux distribution, you'd probably be surprised at how many system daemons assume they can write various places in the root partition, and you really don't want to maintain hacked versions of all of them.) Since ChromeOS gave an Ubuntu shout-out at one point, I strongly suspect the unionfs scheme is actually what they are doing as well — and, for that matter, what all the other Ubuntu Mobile downstreams are doing. Not new.

The emphasis on a read-only root partition is rather misleading from a security standpoint (as much of the middle portion of the presentation was). If you're not storing your files locally, it doesn't mean that you suddenly have no security concerns. It just means you have different security concerns. Cross-site scripting attacks give a malicious website access to your Google account through your web browser: these sorts of things are the malware for a WebOS. You have a different attack surface, but a vulnerability in your browser or flash plugin still gives access to private data. Mentioning that they encrypt the data on disk seems to be pure snake oil: your browser has access to the unencrypted data, and that's your most vulnerable surface anyway.

Overall, Chrome OS is a nice validation of some of the cloud-computing ideas we've been working on at litl, and it's always nice to see more legacy-free hardware (like the OLPC XO-1 and the litl webbook), but the presentation was oddly underwhelming. They're not really "reimagining the PC" — they're just removing all the applications on the PC except for Chrome. You still interact with "the PC" the same way you currently interact with Chrome. For reimagination, watch the videos at litl.

Update: Lucas has written up some additional technical details — and he mentions our update system, which was one of the first bits I worked on when I arrived. We heavily dog-food the updater, using buildbot to push out the latest bits to developers every night.

Update 2: Welcome, slashdot! The videos at http://litl.com/support/ give a good idea of what the UI is like.

Update 3: Non-technical press coverage: Wired, WSJ, Xconomy, CrunchGear

Update 4: More words: dignacio on litl's server side, J5, twitter

litl launches today, so I can finally talk a bit about the cool technologies behind our software stack.

On the server side, litl is a cloud computer, built on Google App Engine, Amazon S3, and Django — all of which are fantastic technologies. All machine data is stored in the cloud, so you can have a gorilla stomp on your litl, pick up another one, log on and instantly recreate your environment. (Since we developers are always abusing our prototype hardware, we've tested this a lot!)

On the client side, the litl software (naturally, code-named "big") is built on gjs, which is the smallest possible wrapper around JavaScript necessary to make it a large-scale programming language. I've really enjoyed programming in JavaScript, which might seem odd to people who (a) have had frustrating experiences with global variables and crazy incompatibilites trying to make JavaScript work on the web, and (b) know that I'm a static types and compiler-bondage sort of guy. So I'll spend a little time here talking about gjs.

From a language standpoint gjs adds just one feature: a minimal module/namespacing mechanism built on a single top-level definition: the name "imports". Modules are imported using (typically constant) definitions, such as:

const StringUtil = imports.stringUtil;

s = StringUtil.sprintf("%d", 3);

The dynamic stringUtil property of the imports object is an object whose properties are the top-level definitions in the file stringUtil.js, found on the import path. Subdirectories are additional dot-separated components, as in Java package names; imports.util is a dynamic object representing modules found in a util directory on the path. You may need to compare this to the namespacing mechanism in the abandoned ES4 proposal to appreciate how small and elegant this is.

Further, this module system integrates with the GObject system via GObject introspection annotations for library code. This allows easy integration with libraries written in C or any other introspectable language. For example:

const Gtk = imports.gi.Gtk;
Gtk.init(0, null);

let w = new Gtk.Window({ type: Gtk.WindowType.TOPLEVEL });
w.connect('destroy', Gtk.main_quit );

let button = new Gtk.Button({ label: "Hello, world!" });
button.connect('clicked', function() { w.destroy(); } );
w.add(button);
w.show_all();

Gtk.main();

The gjs system is built on the SpiderMonkey JavaScript engine, the one used in Firefox, so JavaScript execution benefits from all the JIT and performance work done upstream. Further, it means that we can code in JavaScript 1.8, Mozilla's dialect of JavaScript with lots of bells and whistles (mostly borrowed from Python):

gjs> function range(n) { for (let i=0; i<n; i++) yield i; }
gjs> [i*2 for (i in range(5))]
0,2,4,6,8

(In a later post perhaps I'll describe how you can use the yield expression to build a continuation-based system for writing asynchronous callbacks for UI code in a natural manner.)

Overall, JavaScript as a system programming language feels a lot like Lisp must have for the programming generation before mine: minimal syntax, very powerful and orthogonal core abstractions, and (dare I say it) not much type-checking or busy-work to get in your way. (gjs is not a homage to Sussman, but it should be!) JavaScript is a programming language for those who know what they're doing and aren't afraid of a little functional abstraction. (Now if only there was a way to disable semicolon insertion!)

OK, enough about gjs. The litl software stack is based on Canonical's distribution of Ubuntu for mobile/embedded devices, and the Canonical folks have been great partners. It's been a joy to get changes integrated upstream, and Canonical has done a lot of excellent work accommodating their distribution to litl's unique needs. On the UI side, we use X11 and some GTK widgets, but implement our own (very simple) window manager. Most of the actual look and feel is done with Clutter (hardware-accelerated 3d animated effects, whee!), and we have a Flash-based API for external developers. We also have hardware-accelerated h.264 video.

Regardless of the technical fun, though, I have to say that the best thing about working at litl is its management: developing with all the other rockstars here is made the more fun by the knowledge that management will help ensure that our goals are realistic and that we'll be able to hit our targets, with time left over to polish before release. It's just much more fun to code when you know you'll be proud of the end result.

Dave Woodhouse in a recent article calls shenanigans on hardware translation for flash devices. I agree: flash memory is a Horse Of A Different Color, and trying to gussy it up to look like a rotating disk of magnetized rust is using a bad and leaky abstraction that will only end in tears. But I don't think the engineers' better judgment will prevail: the use of hardware translation is driven by Windows/DOS compatibility concerns, since Microsoft (to my knowledge) has shown no desire in writing a new filesystem for flash drives. OLPC used a raw flash device in the XO-1, but in their follow-on had to switch to a hardware-translation device because market/scale economics were making those devices cheaper and cheaper while the original raw flash device was (a) not increasing in volume (aka, getting relatively more expensive), (b) not increasing in size (no one wanted to make new ones), and (c) getting discontinued (aka, impossible to buy). The best one can hope for is that a raw interface be offered in addition to the standard "Windows-compatible" one, for specialized embedded or high-performance applications — but the chicken and egg problem applies: until there are compelling gains, these interfaces won't be purchased in sufficient volumes to yield reasonable prices, and no one is writing the optimized filesystems because you can't find reasonably-priced flash devices to run them on. The end result is likely to be that Worse Is Better, and we'll be left with another set of legacy chains. Given enough time and transistors, the hardware may eventually grow Heroic Measures to work around the bad abstraction (see: the x86 instruction set).

If your filesystem is large enough and the amount of data being rewritten small enough, the flash problems "probably" won't bite you until after the obsolescence of the device — flash storage doesn't have to be good, it just has to be "not too bad" until, say, 3 years after purchase. Like non-removable rechargeable batteries that slowly degrade over time, you'll find your filesystem slowly degrading — one more reason to eventually buy a new one, and I've never known manufacturers to be especially sorry about that. Heroic Measures may never be needed/taken.

Leaving amateur market economics (and despair), let's revisit a cryptic and probably overlooked paragraph in my olpcfs proposal:

Implementations tuned for flash memory can use Adaptive Packed Memory Arrays for efficient CAS. This is an open research topic, but there is some evidence that a high-performance scalable flash-based filesystem can more profitably be implemented using cache-oblivious B-Trees and APMAs, rather than more "traditional" filesystem structures.

Here I'm trying to build a little bridge between filesystem designers and functional data structure researchers, two groups who probably rarely sit down together for a beer. I think Packed Memory Arrays are the "B-trees of flash filesystems": a better way to build an on-disk index given the peculiar characteristics of flash memory. Just as BeOS demonstrated that your filesystem could be "B-trees all the way down", I believe you could build a compelling filesystem using PMAs as the primary data structure. Ultimately, I suspect that the development strategy Dave Woodhouse describes — small incremental changes to existing filesystems whose philosophies are "close enough" — will probably prevail over a ground-up PMA rewrite. Incremental improvements and shared codebases are the Right Strategy for successful open source projects: you get more developers and testers for those parts which aren't flash-specific, and you've already gotten yourself out of the Cathedral with some working code to kick things off.

But if anyone's interested in attempting a clean slate design, PMAs are my advice. Maybe you'll come up with something so wonderful it will make a compelling book (and inspire folks like me), even if ultimately you don't win in the marketplace.

(But maybe you'll win! Flash storage and your optimized filesystem will prevail, and one day we'll think of rotating media in the same way we now think of core memory, floppy disks, tape drives, and the x86 instruction set... er, hmm.)

It seems I missed a nice little Slashdot/Negroponte/Krstić OLPC/Sugar row while I was away in Europe savoring the efficient intercity rail. While Tomeu and OLPCnews choose to blame particular words or phrases ("Sugar", "$100 laptop"), both J5 and gregdek seem to think the Real Problem was the fact that OLPC wouldn't upstream its patches.

I beg to disagree. As I wordily tried to explain in a comment to gregdek's post, I think Ivan is mostly right here: OLPC tried to do "seven new things" (as Mary Lou explained to me when I was hired) -- and "new things" end up costing a lot of debugging and development time, in one of the Iron Laws Of Writing New Code And Making New Hardware. But another problem was just Picking The Wrong Partners. With the exception of the display (one of the few unalloyed successes of the XO hardware), most of our hardware and software partners were working at cross purposes. Red Hat didn't really want to build an embedded OS product, "mesh networking" to Marvel meant household networks between your TV and your stereo with maybe 10 participants, the Geode was an orphaned offering from AMD, the display and flash NAND controller was a unloved one-off, etc. Success is found by aligning your partners' interests with your own.

At Litl our OS partner has many other embedded-systems clients, and has developed the toolsets to handle forks and customization without all the angst I'd grown used to at OLPC. We just say, "we need feature X turned on/off" or "set to Y" or just "somehow Z needs to happen" and it's done. We're not fighting, we're not destabilizing their core OS, we don't waste time with elaborate justifications why This Is The Right Thing To Do. If the change is appropriate to upstream, they upstream it, and maybe the work benefits the other embedded-systems clients. There's no drama, because We All Want The Same Thing.

I think "upstreaming" is entirely the wrong hero for OLPC here. If anything, I think OLPC's best hope is to continue to aggressively innovate -- no one buys a computer because its code has been upstreamed. Unfortunately, I don't think OLPC has enough current funding to do anything but follow the upstream, so maybe this current round of praise for Fedora-ization is just the old cliché: making a virtue of necessity.

So, yesterday I posted a entry discussing how Google Wave actually implements the collaborative vision Sugar (and OLPC) were working towards. (It's a shame we didn't have better contacts with Google while I was at OLPC; Google was actually on OLPC's board, and beta-ing Waves would have been a very fruitful partnership.)

If you agree with the premise: what should the next steps in a Waves-ification of Sugar be? Eventually Google promises to release "the lion's share" of its source code, for both server and client, so getting the google server installed on the school server is one task — but not one which can be done immediately. Implementing Network Principles is another necessary precondition, in order to use Wave's DNS-based participant naming system ("SuzyQStudent@lima.peru.schoolserver.laptop.org" or whatever). That's something which can be done now. What else?

Eventually, when the source code drops, making the waves client work offline would be important, since Waves (and embedded gadgets) basically replace Write and Sugar's bulletin board. Waves edit XML trees, so porting existing activities to use XML-based file formats will go far in integrating them into a new Wave World Order. I haven't seen any demo of a waves-based drawing activity/gadget (tuxpaint is a favorite of most kids), so Waves Paint would be a fun project if you want to start playing with the Waves extension APIs.

More controversially, work on Waves-enabling a next-gen Journal could be interesting. As predicted by proponents of the Journal for some time, the "wave of the future" (so to speak) is filesystem-independent storage. Waves in Google's demo are titled and searched like email messages, not as "documents" in a filesystem hierarchy. However, we had repeated requests to unify Journal storage and traditional filesystems, for (a) better interoperability with existing systems, and (b) sneakernet collaboration. In my mind, this would mean exporting a waves-like view of an existing filesystem, as I proposed for the Journal, where directories are interpreted as slightly-special tag markers. One could imagine implementing a "Wave Server" based around this idea, in effect using the filesystem as the wave database. With the magic of Wave Federation, these "filesystem" waves can interoperate with other wave clients and servers. This standalone file-based server might also server as the basis for "one child under a tree" wave editing. (For that matter, a robust sneakernet implementation of the Wave Federation Protocol would also be very useful!)

Exciting times! Wave promises to bring OLPC/Sugar's vision of ubiquitous collaboration to life at long last. Google has the funding and development resources to tackle what is in effect a bottom-up reorganization of the software stack. OLPC/SugarLab's role is peripheral but vital: strongly lead the development of offline or "flakey connectivity" aspects of this technology so that it can be used everywhere from the solar-powered jungle to the dense urban cities, and to provide the educational software on top of the platform so that kids can *learn* as they collaborate and create.

Google announced Google Wave today, an "attempt to imagine what email would be like if it were invented today". It has a robust collaboration infrastructure strongly grounded in distributed systems theory. If I were (re)starting the OLPC project today, this is certainly what I'd base Sugar on — with as much corporate support from Google as possible, since the project is strongest when it has strong allies.

I believe you could make each school server into a Wave Provider, using the Network Principles I drafted while employed at OLPC to ensure appropriate DNS-style naming. Any document format based on XML can be made collaborative using the Operational Transform mechanism. The journal's large scale history/versioning mechanism could be based on the same principles, as So6/LibreSource demonstrated. And the UI demonstrated in the keynote (which I haven't even fully watched yet) would be an exciting way to implement the neglected Bulletin Board feature.

It would be an exciting start to the Sugar project! But given the existing code base, is there now too much bathwater around the baby to consider swapping the child out?

I started work at litl yesterday, did all the necessary administrivia and got a build environment set up, and am typing this today using their software stack. There are a lot of similarities between my old and new jobs: both target poorly served "nonexpert" audiences — in litl's case, families at home — and are taking advantage of the opportunity to build a clean-slate integrated hardware/software solution that's new and different and not another rehash of the Common Desktop Environment of 1993. Both litl and OLPC write highly design-driven software and employ some of the same design firms, so there are more subtle aesthetic similarities as well.

The most sobering difference (for me) is the effective organization at litl. After spending years fighting uphill battles at OLPC, it's refreshing to find what seems a model development environment: clear and explicit specifications for new features, release notes that incorporate the specs (with green annotations indicating future features or deviations), and test cases. Lots of test cases — both in the code for machine use, and written in English for people to use.

This isn't rocket science. But writing specifications for a complete software/hardware system is not a little work, and as you become more ambitious the difficulty increases. Both OLPC's Sugar and litl reinvent the standard desktop paradigm, so you can't take even "the program launches when you click on it" for granted: you have to define every such interaction (what does "the program" mean?), how it should work, and test that it does so. Software at OLPC never quite made the transition from self-directed research project to production code. It saddens me to think what might have been if OLPC had been able to either properly develop Sugar, or successfully contract out the work — if only it was *Sugar* with the careful 200-page spec, exact lists of features implemented/in-process, heaps of test cases, and proper staging of features...

As part of a much larger company, litl also avoids the self-hosting trap. Email, HR, site hosting, databases, bug trackers, and collaboration support for the devel team are all sourced out to companies who do it well, so precious development time isn't wasted by having your rock star developers installing ubuntu for the 13th time on some new server, upgrading, doing backups, moving machines from one network to the next, tuning trac, etc. Some of this is inevitable, and some is actually necessary -- sometimes there *is* a vital business need for some tool which it's easier to have in-house and patch — but OLPC was suckered into bad decisions by the lure of fast free bandwidth from MIT. At OLPC we didn't have to pay for hosting or bandwidth as long as we kept our servers under our desks — so we ended up paying far more than we saved in lost developer hours as we tinkered.

Anyway, I'm looking forward to the change of pace and to actually delivering the features in the spec! At least one of my OLPC friends will be joining me, and litl is hiring (although they self-describe as "a small team of rockstars"); I can point you in the right direction if you want to learn more about working here.

OLPC laid off most of their software staff yesterday -- 50% of the staff overall, but the cuts were deepest in software. That means, as of Friday, I am available to work -- for you!

I am hearing rumors that there might be a source of seed funding for continued development of Sugar, the educational software stack shipping on all OLPC XOs. If so, I'd love to be able to continue the work that's consumed me for the past 18 months. Fingers crossed.

In October, SJ and I travelled to Peru for OLPC. (Ob. Plug: buy a G1G1 machine this holiday season! It helps get laptops to needy kids and helps fund us to develop more and better software for those kids.)

Anyway, my favorite story from our time in Peru was when SJ stood up in front of a classroom of university kids and announced, in Spanish, "Now we are going to make drawings with pee-pee." He actually was talking about Pippy, the python-programming-for-kids activity written by Chris Ball (with patches from yours truely and others) -- but there's a reason why Pippy is translated "Peppy" in Spanish =).

But I've found another contender for that crown: a cookbook. Best book title ever. (David Friedman not only found that book, he made it an "employee pick" when working at a "large chain bookstore".)

So. Pippy and Winnie-the-Pooh. Closing a scatological circle.

(While I'm in the neighborhood, I have to plug David Friedman's Murder in the Hundred Acre Wood as well. And 10 Lessons from the Movies. Ok, go resume your productive lives now.)

Last bit: the video from Friday's Sugarcamp sessions (2008-11-21) is now up. Ogg-format video in both full and small sizes is available at download.laptop.org. There are "all of Sugarcamp", Wednesday, Thursday, and (new) Friday playlists on YouTube.

Friday has talks on Sugar UI and Collaboration. (These links are to the "small" Ogg-format files, reasonable for downloading.) I came in late to the Sugar UI brainstorm session, so the video is missing the first part. Sorry about that =(.

That's all the sugarcamp video! Hope you've enjoyed watching along!

Part two! The video from Thursday's Sugarcamp sessions (2008-11-20) is now up, you-tube-ized, etc. As before, ogg-format video in both full and small sizes is available for download at download.laptop.org. In addition to the existing "all of Sugarcamp" playlist, I've now created separate Wednesday, Thursday, and (empty, but not for long) Friday playlists on YouTube if you want to skip to a specific day of talks.

Thursday includes talks on Sugar on a (USB) Stick, Sugar, Resara, and the LTSP, and a grab bag of miscellaneous proposals. (These links are to the "small" Ogg-format files, reasonable for downloading.)

One more day of talks to come...

I just finished uploading, encoding, and you-tube-izing all the video from Sugarcamp sessions on Wednesday 2008-11-19. Ogg-format video in both full and small sizes is available for download at download.laptop.org, but the YouTube playlist might be the easiest way to watch:

Included so far are talks on Desktop compatibility, "What was missed", "Activities as building blocks", Discoverable gestures, the School Server, Internationalization, and Community. (These links are to the "small" Ogg-format files, reasonable for downloading.)

And this was all just one day at Sugarcamp! More to come...

It may be old news to you, dear reader, but I discovered OverStream last night and was very impressed. It lets you take web video (say, from YouTube) and create caption streams for it, emitting a standard SRT file which you can then re-import into YouTube, all using web-based tools.

I used this to translate a brief video on OLPC's Arahuay pilot in Peru (ogg version w/o captions):

Help wanted: there's lots of video from Peru in my YouTube channel. You can create an account over at OverStream to caption it -- and if you send me the resulting SRT file, we can make this media more accessible to non-Spanish speakers.

On my last trip to Peru, I brought back a DVD of video material which Peru's Ministry of Education has created to document and promote their One Laptop per Child project. I've uploaded it now to http://download.laptop.org/content/media/Peru-MoE/, converted it to the free Ogg Theora video format, and put some of the videos up on YouTube for easy viewing.

My favorite is the Children's Song (ogg), which sings about the joy of learning with an XO. The first Peru program (ogg) contains the song subtitled in English, which some of you may enjoy more:

The second Peru program (ogg) is a mashup of various press features on OLPC in Peru, starting with a very nice BusinessWeek piece (in English), followed by a piece for infordata.tv (subtitled in English), and ending with a piece for a Japanese program (at least that's what I think it is, I can't read the subtitles!).

More to come!

Looking ahead to our very ambitious goals for the next Sugar/OLPC release, I think it's important to keep Richard Hamming's question in front of us:

What's the best thing you could be working on, and why aren't you?

Today I finish my first week of my new job at One Laptop Per Child. It's true, you can get paid for writing free software (and saving the world) -- who'd have thunk?

At the moment, I'm working on IPv6 networking and Linux kernel support for autoconfiguration.

Helpful Wikipedia articles to explain some of the above: OLPC, IPv6.