JavaScript in asm.js (and a little rust)
cananian

Over on twitter, Tim Caswell mentioned, "I think high-level scripting language on top of something like rust.zero would make for an amazing OS." and that set me off a bit. Twitter isn't a great place to write a reasoned discussion of programming languages or implementation strategies, so let's take a shot at it here.

As I've written about on this blog, I've been tinkering for years with TurtleScript, a small learnable JavaScript subset in the spirit of Alan Kay. Over in that twitter conversation David Herman mentioned rusty-turtle, my TurtleScript bytecode interpreter written in Rust. The rusty-turtle codebase includes a REPL which runs TurtleScript's tokenizer, parser, bytecode compiler, and standard library (all written in TurtleScript) through the Rust interpreter. It's quite cute, and I implemented much more of the JavaScript semantics than I strictly needed to (with the side-effect that the behaviors in the JavaScript wat talk now appear quite sane and sensible to me).

I wrote rusty-turtle as a personal warm-up: I was considering taking a job with the fine folks at Mozilla (OLPC having run out of money again) and wanted to understand the technology better. I described a number of further research projects I thought would be interesting to pursue in the rusty-turtle README, including cilk-style fork/join parallelism or transactional memory support (the latter being the subject of my thesis), and a JIT backend using rust's llvm library bindings.

But the true turtles-all-the-way-down approach would be to rewrite the backend using asm.js, which can be trivially JIT'ed (using llvm bindings). Then you've have an entire system from (pseudo-)assembly code up, all written in a consistent manner in JavaScript. To that end, I wrote single-pass type-checker/verifier for asm.js in TurtleScript, discovering lots of issues with the spec in the process (sigh). (I justified this as more "Mozilla interview preparation"! Besides, it was fun.)

Tim Caswell, to finally answer your question: I think that this "JavaScript all the way" system would make an amazing OS. The Rust stuff is just a distraction (except as needed to bootstrap).

In the next post I'll rant a bit more about Rust.

ps. In the end I over-prepared (!): Mozilla's feedback was that I seemed to "know too much about Rust to work on Servo" (Mozilla's experimental web layout engine, written in Rust). Mozilla seems to have reached that awkward size where it can not longer hire smart people and find places for them to contribute; new hires need to fit into precise job descriptions a priori. That sort of place is not for me.


Reading Project Talk (and slides)
cananian

An unruly tag team of OLPC folks gave a long talk on the Literacy Project today for attendees at this year's OLPC-SF Community Summit. It was streamed live on Ustream: Part 1 (Matt Keller, Richard Smith), Part 2 (Richard Smith, Ed McNierney, C. Scott Ananian, Chris Ball, questions from the audience). We've posted the slides: Matt Keller, Richard Smith, C. Scott Ananian.

You can try out some of the apps mentioned in the talk. Nell's Balloons and Nell's Colors will run in any reasonably-recent Google Chrome or Mozilla Firefox browser. They will also run as a Firefox webapp on Android devices, using the latest Firefox nightly for Android. For deployment we use a slightly-tweaked build of Firefox (adding expanded webapp storage quotas and the ability to use plugins from inside webapps), and a custom plugin to hook up the Funf logging framework. Source code is available on github: nell-balloons; nell-colors. In addition, Chris Ball's "Matching" app for Android is available: apk; source.


Small systems... and distributed ones
cananian

Today I stumbled across some very interesting projects by Nickolay Platonov which I'd like to discuss in an OLPC context.

I've been hacking away at TurtleScript fueled partly by a drive for minimalism: a small system is a learnable system. To that end, the language is based on Douglas Crockford's "Simplified JavaScript" (as recognized by this top-down operator precedence parser) which tries hard to include only JavaScript's "Good Parts". For example, the initial code for the TurtleScript system uses prototype inheritance (via Object.create) rather than classical class-style inheritance. In fact, the new operator isn't even included in the Simplified JavaScript/TurtleScript language.

In a discussion of TurtleScript Alan Kay mentioned, "It is very tricky to retain/maintain readability (so the first Smalltalk was also an extensible language not just semantically but syntactically)." And today I stumbled across the Joose library, which gives a very readable syntax for traditional class-based inheritance. It backs this up with a robust meta-object protocol, introspection, and lots of nice features borrowed from Perl 6, CLOS, and Smalltalk. The syntax ought to work fine with the limited tile set of TurtleScript, although I might have to add a tile for the new operator.

However, adding Joose raises some questions. Is the increase in readability worth the addition of such a large library to the system? What impact will this have on understanding problems and debugging? Is a return to class-based inheritance a positive change? (There have been arguments that the make-a-clone-and-change-it practice of prototype inheritance is easier to understand for new learners.) Can a larger overall system actually be easier to understand?

And once we're looking at libraries... Nickolay Platonov is now working on Syncler, based on the Bayou system. Unobstructive replication mechanisms would certainly make it easier to construct the sorts of collaborative applications we've wanted for Sugar. I have two concerns with Syncler's current state. First, the use of explicit continuation-passing style greatly impairs readability. The JavaScript yield keyword helps a lot when writing asynchronous code. (It's not supported by all JavaScript engines, but yield wouldn't be hard to add to TurtleScript.) Second, Syncler's event model uses explicit callbacks. I've been greatly impressed with the Flapjax event model (and its strongly-typed functional cousin). Both of these changes ought to make asynchronous code much more readable—and isn't that an important part of grokking a system?


Turtles All The Way Down
cananian

Inspired by Bert Freudenberg, Ian Piumarta, and Walter Bender, I started hacking on "Turtles All The Way Down" (aka TurtleScript) on the plane back from Uruguay. Now there's a nice rendering demo to show what a tile-based editor for JavaScript might look like, as well as a bytecode compiler and interpreter for the language. The bytecode instruction set is still too large; encouraged by Craig Chambers' work on SELF I think I ought to be able to replace all the unary and binary operators, conditional jumps, and slot selectors by a single mapof operator. I can put a better object model on the interpreter, too; I've written some notes on the matter.

The question is: does this really have educational value? "Turtles all the way down" is a great slogan, and a fine way to teach a graduate-level class on compiler technology, but I feel that the higher-level UI for tile-based program editing is the really useful thing for tablet computing. I'm a compiler geek and love the grungy underbelly of this stuff, but I keep reminding myself I should really be spending more time building a beautiful fluffy surface.


Next Steps for New Technologies
cananian

I've reached the end of the month. I've accomplished my Android and NativeClient-related goals, but didn't get the time to do as much mesh and python investigation as I'd wanted. Here are some ideas for next month's work. (Next week I'll be in Uruguay for EduJAM.)

GObject Introspection (Android or NaCl)

  1. Start by porting libffi. An android port would be straightforward, but since libffi involves code generation (ARM, x86), this is going to require a bit of assembly magic and the new "JIT"/"shared library" support in the NaCl plugin.
  2. Then port gobject-introspection. GObject-Introspection relies on libffi for its guts, but the hard part of this port will be refactoring g-i's build process, which is not cross-compilation friendly. Might need to rewrite some tools. If targeting NaCl, you might consider finishing the code allowing execution of unsandboxed NaCl binaries.
  3. Turn gobject-introspection on its head: generate GIR and a C binding for the platform "native" interface. For NaCl, this would be a GObject-using C-level binding of the browser-native DOM; for Android, this would be a GIR binding of the native Android APIs. These bindings should be mostly automatically generated, since they will need to continue tracking successive native platform releases/HTML5 features.
  4. Demos! Change browser DOM from Python, write native Android apps in Python. Add a gobject-introspection binding to cforth, then do the same from forth. (Forth might be a simpler place to start than Python. Or not.)

GTK (Android or NaCl)

  1. Build on the cairo/pango port to proceed to a full GTK backend for Android/NaCl. These backends ought to be upstreamable. The NaCl port should be based on the broadway work: the cairo canvas would be drawn to more directly, but a lot of the mechanism which captures JavaScript events and translates them into the GTK event loop could probably be reused.
  2. Demo: "Hello GTK world" in Android/NaCl.

Sugar partitioning.

Bring Sugar closer to being a true multi-language multi-library platform.

  1. Refactor sugar modules (for example, sugar toolbar widget) as standalone C libraries. Basic idea is to embed Python and export a C API, while preserving as much of the code as possible. Python libraries now invoke this library via g-i-r instead of directly. The python embedding tool is probably a useful standalone product.
  2. Rewrite "Hello, Sugar" activity in C (or vala), using #include for import and GObject inheritance instead of python inheritance. Use this as a guide to pull apart sugar into modules (as above) to make this code actually work as written.

Miscellanous topics

  1. ChromeOS w/ touch support.

    Find an appropriate machine, do an installation, what are the roadblocks/rough spots? Can we install on XO-1.75 as a testbed?

  2. TurtleArt as JavaScript viewer/editor.

    Revisit TurtleScript work, but skip over the time-consuming "construct an editor" step by reusing the (excellent) TurtleArt code.

  3. Mesh: android olsrd frontend, build testbed, research 802.11 DCF issues.

Summary

There are four rough topics here; I might try to continue the breadth-first search by spending a week on each. It might be more satisfying to downselect two of these issues and spend two weeks on each.


CoffeeScript and TurtleScript
cananian

Via reports on the OSCON 2010 Emerging Languages Camp, I recently discovered CoffeeScript, a very interesting "dialect" of JavaScript. The original idea for CoffeeScript seems to be to clean up the syntax of JavaScript, while preserving direct correspondence as much as possible. Over time, it seems to have grown more and more "neat features" which have increasingly-hairy desugarings into JavaScript — but the JavaScript translation of a CoffeeScript program is still very readable.

This has some relationship to my TurtleScript project, which also sought to clean up JavaScript to some degree. In TurtleScript I've tried to pare down as much of the language as possible, using the smallest number of syntactic forms and JavaScript features as possible, with the smallest possible parser/compiler/runtime system, inspired by Crockford's Simplified JavaScript.

CoffeeScript has some very elegant simplifications: for example, everything is an expression, which reduces the statement/expression syntactic distinction in C-like languages. Making the body of a function into an expression removes unnecessary return statements. Making for and while into expressions is a cute means to introduce (the power of) array comprehensions without additional syntax. CoffeeScript also has a nice way to express functions both with and without lexical this binding (a JavaScript skeleton in the closet).

Unfortunately, the full CoffeeScript language includes many many syntactic forms — even more than full JavaScript. Some of these can trivially be moved into a library, at a slight cost in compactness. While having varied syntax that precisely and concisely expresses programmer intent is certainly admirable, it makes the language harder to learn and, in my opinion, less elegant. For a scalable, learner-friendly language system I like something like NewSpeak more: a language whose goal is to grow smaller, not larger.


Adopting an IFRAME
cananian

Warning: heavy geek content.

Google recently rolled out a new Gmail feature: if you pop open a new little compose or chat window, and then close your main window, the small child window doesn't go away.

This is a pretty esoteric little tweak, but the underlying functionality is quite profound: in order to make this feature work, the entire document context (all the JavaScript responsible for running Gmail's UI) needs to get teleported from the main window out into the child window. For speed and memory reasons, you don't want to run a copy of the main code in the child window — no, you want to wait until just before you close the main window and then instantly teleport the guts of the main window over to the child. How's Google doing that?

In the browser model, separate windows are usually separate security contexts, safely sandboxed from each other. So this magic teleportation trick is also creating a wormhole between the different security domains. My curiosity was piqued, but Google (both blog and search engine) was strangely silent on how their new trick was pulled off.

Here's the story I eventually discovered. Google had long been thinking about a way to make this feature work. Their initial proposal was something called GlobalScript (or sometimes, "SharedScript"). This proposal met resistance and a new simpler solution was found: the "Magic IFRAME".

The guts are hidden inside bug 32848 in webkit's bug tracker. You put all of your application's good stuff inside an <iframe> element — we've guessed that already. You pass your child window a copy of that IFRAME, something like:

function doSomethingCoolThatNeedsANewWindow() {
     var childWin = window.open(...);
     childWin.onload = function() {
         childWin.functionThatTakesScriptContext(mySharedIFrame);
     }
 }

Now, the crux: when your main window is about to close, you just adoptNode the shared <iframe>:

// adoptNode in this case does removeChild() internally
// but does not unload the content
childWin.adoptNode(mySharedIFrame);
childWin.body.appendChild(myShareIFrame);

Voila! This used to completely unload and reload the iframe, erasing the existing application context, but with this one small hack Chrome (and now Safari, too) suppresses the reload and allows the untouched <iframe> context to teleport over to the child.

This doesn't work on Mozilla yet. And it's a pretty ugly hack, tweaking the behavior in just this one narrow case. (And Mozilla is a little bit cranky about adoptNode() being used without an prior importNode().) But this hack also allows work-arounds for two other long-standing iframe-reparenting "bugs". It suggests that <iframe> is now the <module> tag Crockford wanted, especially now that the <iframe> can be sandboxed in various ways as well. By putting each piece inside an <iframe> you can now build a robust module system for browser JavaScript.


litl's technical secrets revealed!
cananian

Update: Lucas has written up some additional technical details — and he mentions our update system, which was one of the first bits I worked on when I arrived. We heavily dog-food the updater, using buildbot to push out the latest bits to developers every night.

Update 2: Welcome, slashdot! The videos at http://litl.com/support/ give a good idea of what the UI is like.

Update 3: Non-technical press coverage: Wired, WSJ, Xconomy, CrunchGear

Update 4: More words: dignacio on litl's server side, J5, twitter

litl launches today, so I can finally talk a bit about the cool technologies behind our software stack.

On the server side, litl is a cloud computer, built on Google App Engine, Amazon S3, and Django — all of which are fantastic technologies. All machine data is stored in the cloud, so you can have a gorilla stomp on your litl, pick up another one, log on and instantly recreate your environment. (Since we developers are always abusing our prototype hardware, we've tested this a lot!)

On the client side, the litl software (naturally, code-named "big") is built on gjs, which is the smallest possible wrapper around JavaScript necessary to make it a large-scale programming language. I've really enjoyed programming in JavaScript, which might seem odd to people who (a) have had frustrating experiences with global variables and crazy incompatibilites trying to make JavaScript work on the web, and (b) know that I'm a static types and compiler-bondage sort of guy. So I'll spend a little time here talking about gjs.

From a language standpoint gjs adds just one feature: a minimal module/namespacing mechanism built on a single top-level definition: the name "imports". Modules are imported using (typically constant) definitions, such as:

const StringUtil = imports.stringUtil;

s = StringUtil.sprintf("%d", 3);

The dynamic stringUtil property of the imports object is an object whose properties are the top-level definitions in the file stringUtil.js, found on the import path. Subdirectories are additional dot-separated components, as in Java package names; imports.util is a dynamic object representing modules found in a util directory on the path. You may need to compare this to the namespacing mechanism in the abandoned ES4 proposal to appreciate how small and elegant this is.

Further, this module system integrates with the GObject system via GObject introspection annotations for library code. This allows easy integration with libraries written in C or any other introspectable language. For example:

const Gtk = imports.gi.Gtk;
Gtk.init(0, null);

let w = new Gtk.Window({ type: Gtk.WindowType.TOPLEVEL });
w.connect('destroy', Gtk.main_quit );

let button = new Gtk.Button({ label: "Hello, world!" });
button.connect('clicked', function() { w.destroy(); } );
w.add(button);
w.show_all();

Gtk.main();

The gjs system is built on the SpiderMonkey JavaScript engine, the one used in Firefox, so JavaScript execution benefits from all the JIT and performance work done upstream. Further, it means that we can code in JavaScript 1.8, Mozilla's dialect of JavaScript with lots of bells and whistles (mostly borrowed from Python):

gjs> function range(n) { for (let i=0; i<n; i++) yield i; }
gjs> [i*2 for (i in range(5))]
0,2,4,6,8

(In a later post perhaps I'll describe how you can use the yield expression to build a continuation-based system for writing asynchronous callbacks for UI code in a natural manner.)

Overall, JavaScript as a system programming language feels a lot like Lisp must have for the programming generation before mine: minimal syntax, very powerful and orthogonal core abstractions, and (dare I say it) not much type-checking or busy-work to get in your way. (gjs is not a homage to Sussman, but it should be!) JavaScript is a programming language for those who know what they're doing and aren't afraid of a little functional abstraction. (Now if only there was a way to disable semicolon insertion!)

OK, enough about gjs. The litl software stack is based on Canonical's distribution of Ubuntu for mobile/embedded devices, and the Canonical folks have been great partners. It's been a joy to get changes integrated upstream, and Canonical has done a lot of excellent work accommodating their distribution to litl's unique needs. On the UI side, we use X11 and some GTK widgets, but implement our own (very simple) window manager. Most of the actual look and feel is done with Clutter (hardware-accelerated 3d animated effects, whee!), and we have a Flash-based API for external developers. We also have hardware-accelerated h.264 video.

Regardless of the technical fun, though, I have to say that the best thing about working at litl is its management: developing with all the other rockstars here is made the more fun by the knowledge that management will help ensure that our goals are realistic and that we'll be able to hit our targets, with time left over to polish before release. It's just much more fun to code when you know you'll be proud of the end result.


JDoctest 1.5 released
cananian
I've released JDoctest 1.5, with support for integrating Javascript doctests of your Java code into the JUnit test framework. I use this to implement continuous testing.