Ubiquitous Programming with Pen and Paper

Programming ideas strike at any time – while taking a walk, or standing in line. Meetings, for me, are a common location for programming ideas. A few slides or phrases catch my interest in an otherwise droll and predictable commentary. They mix with leftover thoughts and echoes of yesterday. New ideas are born. The desktop programming environment never seems so far away – nor so slow-to-start – as when a new idea is suffocating, dying silently, unable to take its first breath of life and scream for the world to hear.

If we’re quick, we can at least preserve the idea – write it down before it’s forgotten! – then return to it later. But sometimes the moment has passed, the spark is gone, it can’t be recovered. We must seek a new inspiration to nurture. Today, people have twitter – not so great for programming ideas, but excellent for those short natural language witticisms and thoughts that would otherwise go unheard and unappreciated.

I believe there is a similar niche for small programs and tiny programming sessions in the range of thirty seconds to a few minutes. There may even be a similar role for social interaction: observe streams of programs in development, and provide suggestions, refutations, extensions, integration.

Ubiquitious programming describes an ability to program anywhere, anytime, even in those tiny temporal windows. The phrase is a nod to the much grander vision of ubiquitous computing. I am by no means the first to use the phrase ubiquitous programming, but I have not seen it used elsewhere in its literal sense (as opposed to “programming of ubiquitous computing systems”).

I’ve been thinking much about ubiquitous programming this year – a consequence of being flooded with ideas. For me, 2012 has been an exciting year in HCI. In March, I received a hands-on demonstration of Golden-i. A couple weeks later, I received press announcements of Project Glass. In August, I purchased my first smartphone (losing my great-grandfather of a service plan – voice and text at $60 per year). I read a Bret Victor article on hands and watched a video on Disney’s Touché. The Leap Motion technology was announced, and got me thinking more about gestures. Recently, I read an article on a wrist-worn Kinect (which could do with some miniaturization). A few of my thoughts were recorded in my abandoning commitment article and a discussion with the augmented-programming group.

Anyhow, since purchasing my smartphone, I’ve been wondering how to use it for programming. I was short of ideas until a friend interpreted an interpretation of an interpretation of the excellent Learnable Programming essay as Bret Victor attacking his precious notation. “NooOOoo! Don’t take away my notation!” is not what my friend said, but does offer the right impression. That got me thinking:

What is the best way to support notation in an ubiquitous programming environment?

Pen, Paper, and Camera

The very idea of writing code on a smartphone’s touchscreen nauseates me. Even writing natural language, I feel cramped visually, physically, spiritually, and emotionally. My big hands block the screen, and have difficult placing a cursor. An on-screen keyboard further blocks the screen. If I were to squeeze in an IDE – suggestions, annotations, quick links, etc. – there’d be no play space left for my heart and mind and comments and code. A tablet or laptop is a non-starter for other reasons. They are too heavy, require too much preparation, and are too difficult to whisk out at a moment’s notice. I don’t have tablet-sized pockets (and I don’t imagine sitting comfortably with them even if they existed).

When I considered glasses, however, I found a lot of promise.

Augmented reality glasses have an embedded camera that will see what I see. If I’m looking at notation, then I have a means of loading said notation into a program. And oh! while more awkward (requiring dedicated hand to lift the camera), I could leverage a smart phone camera in a similar manner: capture an image of notation as the primary input model, shifting the primary editing off the touchscreen.

Keeping a small pad of folded scrap paper and a pen is easy enough to do in a pocket or wallet. And a more prepared person might grab a composition book or moleskin.

Primary input device becomes pen, paper, and camera. The program “canvas” – the paper – is as large or small as we spread it. A fine pen is far more precise and elegant an input device than my finger. (Some people might favor pencil.)

Use of paper for editing leaves the smartphone’s screen free for other rich development aspects: annotations, suggestions, requests for clarification, warnings, errors, establishing relationships to other program. For glasses, similar elements might be rendered in overlay or at the periphery.

Modularity, Linking, Sharing

Subprograms from different canvases cannot be linked directly. (They might be on different pages of the same notebook, for example, and we don’t want to tear it apart.) Thus, relationships between subprograms are established in a virtual space. The author will represent a desired link in the canvas – e.g. by drawing a rectangle and adding a little context (e.g. a title or other label, maybe a date or version number or author name). Given this representation as a hint, and more implicit hints (such as a history of programs viewed or browsed by the author), the phone or glasses can present a narrow set of options.

By selecting options, establishing relationships, the program ultimately becomes much more than its canvas – more meaningful, precise, logical, correct, integrated. Development and editing becomes a mixed bag of virtual manipulations and physical extensions (or occasionally crossing out or erasing an element). The actual program is a virtual object maintained by the programming system, preferably in a cloud (whether public or private) and synching with local storage for “occasionally offline” programming. History and multiple versions of every virtual program object would be accessible via the cloud.

However, the original canvas continues to provide access to the associated virtual object. It will do so until damaged or degraded beyond recognition. This is achieved by indexing programs on their content and visual fingerprints. Virtual programs are indexed on many properties: author, date, location, book, content, comments, relationships, histories, and those visual fingerprints.

If you have read-access to my cloud, then I could hand my notebook off to you and you’ll be able to read the programs – not just my semi-legible handwriting, but also the relationships and clarifications. And my library becomes accessible to you. Ubiquitous programming with pen and paper is sharable in this wonderful, tangible way – like any book. Of course, you could still get the e-book experience by browsing the cloud directly. That probably loses a lot of favor.

A good programming model might also press for certain indirection in links, to avoid entanglement. But I find ubiquitous programming to be an orthogonal issue.

Free-form Diagrams and Notations

Modern programming is primarily text because that’s what a keyboard affords us. We could use a mouse for freeform input, but that’s too painful in my experience. With a pen and paper, we are afforded full use of graphs, diagrams, arrows, functional curves, sketches (e.g. of characters, world maps, interactions), of ad-hoc mathematical symbology, etc.

This is a freedom that I love in my note taking, when I’m actually thinking about ideas. I’m that person who buys composition books and handfuls of pens and just wanders off to a park or restaurant. I often find it difficult to shift back to rigid program text.

I like scratch space. I scratch notes in margins. I connect ideas together. I’m not fond of structured editors. But this notion of an “unstructured” edit space that happens to represent a structured virtual program object (achieved via clarifications and constraints) appeals to me greatly.

Word’s Up!

Usually, my composition books end up in some disorganized trunk that I never look through again (after I have internalized the concepts). But if all these ideas, sketches, diagrams, and information are indexed, searchable, accessible, replicated in a manner resistant to fire damage – these are many nice properties to add to written word (regardless of whether it was written for programming). People often take pictures of whiteboard diagrams developed at meetings. Why isn’t this already being stored and indexed? Is there already a tool out there? <– Possible killer, money-grubbing application here for anyone who has the time and inclination to pursue it.

REPL, Widgets, Live Coding

Whether we use phone or glasses, we can get a lot of interactive feedback from our pen and paper programs. This includes evaluation of expressions. The notion of writing out a series of equations and getting feedback about their values? Straightforward, once programs are input from pen and paper. And this has a fringe benefit that people can say “That’s not true in my book!” much more literally.

An interesting possibility is to lift REPL concept to full widgets. Instead of a simple value, an expression must evaluate to a tangible value or naked object or similar that renders as interactive widgets. The phone or glasses can then render the widget, making it observable and manipulable. Such widgets might be manipulated voice or gesture.

Widgets developed in such a manner can control state in real systems, and it may prove a useful basis for ubiquitous HCI. Open your moleskin book to the right page. Tweak some virtual sliders and checkboxes. Take some looseleaf paper and write out a few programs or program-references. Tape them to your environment, and now you have ubiquitous widgets.

Hmmm. Widgets represented in this manner have a sort of spell-books and painted runes feel to it. Shall we call this brand of ubiprogrammers ‘Widges and Widgards’? ^_^

The notion of live coding is orthogonal to ubiquitous programming, but I think they interact nicely at the boundary between programming and HCI – for developing these widgets and REPLs.

Leveraging Color?

With pen and paper programming, we lose syntax highlighting. Developers who want multiple colors would need a handful of pens (or crayons). For ubiquitous programming and computing, I am disinclined to support semantic color for at least six reasons:

• If the colors have different semantics, then I must waste precious idea-time sorting my pens.
• Keeping multiple pens of different color means carrying many more pens, but more than one and a spare is unacceptable for my pocket space.
• Computer vision algorithms will more easily handle simple contrast.
• Colors don’t handle well under varying light conditions (esp. colored lights).
• For ubicomp/HCI, invisible inks (IR, UV, crystalline diffraction) might become popular, but generally lack the flexible range of color information.
• Simple contrast will work in many more ubiquitous programming scenarios.

But color might still find a few niches – not for ubiquitous programming, but perhaps for a more artistic programming or compilation.

Next Steps

I think ubiquitous programming with pen and paper is powerful, feasible today, and an exciting possibility. While glasses would be preferable (especially while standing or moving), the phone is more widely distributed, more accessible, and would be a better place to start. Computer vision algorithms we have today are generally sufficient to the purpose. A default cloud could be provided easily enough (preferably open source).

What I have not discussed, and what needs to be done to make this a reality, is integration of the flexible syntax with a programming model. It doesn’t need to be RDP (though I’d appreciate it!). I spent most of 2007 thinking about syntax, and my general conclusion was to put it off as long as possible while preserving static comprehension of code.

I wonder if an approach to user-defined syntax might be the way to bootstrap ad-hoc visual and diagrammatic syntax. Combined with the virtual program object, we don’t even need to declare the syntax ahead of time – we can just start writing then clarify afterwards (syntax as yet another subprogram relationship!).