Ubiquitous Programming with Pen and Paper

Programming ideas strike at any time – while taking a walk, or standing in line. Meetings, for me, are a common location for programming ideas. A few slides or phrases catch my interest in an otherwise droll and predictable commentary. They mix with leftover thoughts and echoes of yesterday. New ideas are born. The desktop programming environment never seems so far away – nor so slow-to-start – as when a new idea is suffocating, dying silently, unable to take its first breath of life and scream for the world to hear.

If we’re quick, we can at least preserve the idea – write it down before it’s forgotten! – then return to it later. But sometimes the moment has passed, the spark is gone, it can’t be recovered. We must seek a new inspiration to nurture. Today, people have twitter – not so great for programming ideas, but excellent for those short natural language witticisms and thoughts that would otherwise go unheard and unappreciated.

I believe there is a similar niche for small programs and tiny programming sessions in the range of thirty seconds to a few minutes. There may even be a similar role for social interaction: observe streams of programs in development, and provide suggestions, refutations, extensions, integration.

Ubiquitious programming describes an ability to program anywhere, anytime, even in those tiny temporal windows. The phrase is a nod to the much grander vision of ubiquitous computing. I am by no means the first to use the phrase ubiquitous programming, but I have not seen it used elsewhere in its literal sense (as opposed to “programming of ubiquitous computing systems”).

I’ve been thinking much about ubiquitous programming this year – a consequence of being flooded with ideas. For me, 2012 has been an exciting year in HCI. In March, I received a hands-on demonstration of Golden-i. A couple weeks later, I received press announcements of Project Glass. In August, I purchased my first smartphone (losing my great-grandfather of a service plan – voice and text at $60 per year). I read a Bret Victor article on hands and watched a video on Disney’s Touché. The Leap Motion technology was announced, and got me thinking more about gestures. Recently, I read an article on a wrist-worn Kinect (which could do with some miniaturization). A few of my thoughts were recorded in my abandoning commitment article and a discussion with the augmented-programming group.

Anyhow, since purchasing my smartphone, I’ve been wondering how to use it for programming. I was short of ideas until a friend interpreted an interpretation of an interpretation of the excellent Learnable Programming essay as Bret Victor attacking his precious notation. “NooOOoo! Don’t take away my notation!” is not what my friend said, but does offer the right impression. That got me thinking:

What is the best way to support notation in an ubiquitous programming environment?

Pen, Paper, and Camera

The very idea of writing code on a smartphone’s touchscreen nauseates me. Even writing natural language, I feel cramped visually, physically, spiritually, and emotionally. My big hands block the screen, and have difficult placing a cursor. An on-screen keyboard further blocks the screen. If I were to squeeze in an IDE – suggestions, annotations, quick links, etc. – there’d be no play space left for my heart and mind and comments and code. A tablet or laptop is a non-starter for other reasons. They are too heavy, require too much preparation, and are too difficult to whisk out at a moment’s notice. I don’t have tablet-sized pockets (and I don’t imagine sitting comfortably with them even if they existed).

When I considered glasses, however, I found a lot of promise.

Augmented reality glasses have an embedded camera that will see what I see. If I’m looking at notation, then I have a means of loading said notation into a program. And oh! while more awkward (requiring dedicated hand to lift the camera), I could leverage a smart phone camera in a similar manner: capture an image of notation as the primary input model, shifting the primary editing off the touchscreen.

Keeping a small pad of folded scrap paper and a pen is easy enough to do in a pocket or wallet. And a more prepared person might grab a composition book or moleskin.

Primary input device becomes pen, paper, and camera. The program “canvas” – the paper – is as large or small as we spread it. A fine pen is far more precise and elegant an input device than my finger. (Some people might favor pencil.)

Use of paper for editing leaves the smartphone’s screen free for other rich development aspects: annotations, suggestions, requests for clarification, warnings, errors, establishing relationships to other program. For glasses, similar elements might be rendered in overlay or at the periphery.

Modularity, Linking, Sharing

Subprograms from different canvases cannot be linked directly. (They might be on different pages of the same notebook, for example, and we don’t want to tear it apart.) Thus, relationships between subprograms are established in a virtual space. The author will represent a desired link in the canvas – e.g. by drawing a rectangle and adding a little context (e.g. a title or other label, maybe a date or version number or author name). Given this representation as a hint, and more implicit hints (such as a history of programs viewed or browsed by the author), the phone or glasses can present a narrow set of options.

By selecting options, establishing relationships, the program ultimately becomes much more than its canvas – more meaningful, precise, logical, correct, integrated. Development and editing becomes a mixed bag of virtual manipulations and physical extensions (or occasionally crossing out or erasing an element). The actual program is a virtual object maintained by the programming system, preferably in a cloud (whether public or private) and synching with local storage for “occasionally offline” programming. History and multiple versions of every virtual program object would be accessible via the cloud.

However, the original canvas continues to provide access to the associated virtual object. It will do so until damaged or degraded beyond recognition. This is achieved by indexing programs on their content and visual fingerprints. Virtual programs are indexed on many properties: author, date, location, book, content, comments, relationships, histories, and those visual fingerprints.

If you have read-access to my cloud, then I could hand my notebook off to you and you’ll be able to read the programs – not just my semi-legible handwriting, but also the relationships and clarifications. And my library becomes accessible to you. Ubiquitous programming with pen and paper is sharable in this wonderful, tangible way – like any book. Of course, you could still get the e-book experience by browsing the cloud directly. That probably loses a lot of favor.

A good programming model might also press for certain indirection in links, to avoid entanglement. But I find ubiquitous programming to be an orthogonal issue.

Free-form Diagrams and Notations

Modern programming is primarily text because that’s what a keyboard affords us. We could use a mouse for freeform input, but that’s too painful in my experience. With a pen and paper, we are afforded full use of graphs, diagrams, arrows, functional curves, sketches (e.g. of characters, world maps, interactions), of ad-hoc mathematical symbology, etc.

This is a freedom that I love in my note taking, when I’m actually thinking about ideas. I’m that person who buys composition books and handfuls of pens and just wanders off to a park or restaurant. I often find it difficult to shift back to rigid program text.

I like scratch space. I scratch notes in margins. I connect ideas together. I’m not fond of structured editors. But this notion of an “unstructured” edit space that happens to represent a structured virtual program object (achieved via clarifications and constraints) appeals to me greatly.

Word’s Up!

Usually, my composition books end up in some disorganized trunk that I never look through again (after I have internalized the concepts). But if all these ideas, sketches, diagrams, and information are indexed, searchable, accessible, replicated in a manner resistant to fire damage – these are many nice properties to add to written word (regardless of whether it was written for programming). People often take pictures of whiteboard diagrams developed at meetings. Why isn’t this already being stored and indexed? Is there already a tool out there? <– Possible killer, money-grubbing application here for anyone who has the time and inclination to pursue it.

REPL, Widgets, Live Coding

Whether we use phone or glasses, we can get a lot of interactive feedback from our pen and paper programs. This includes evaluation of expressions. The notion of writing out a series of equations and getting feedback about their values? Straightforward, once programs are input from pen and paper. And this has a fringe benefit that people can say “That’s not true in my book!” much more literally.

An interesting possibility is to lift REPL concept to full widgets. Instead of a simple value, an expression must evaluate to a tangible value or naked object or similar that renders as interactive widgets. The phone or glasses can then render the widget, making it observable and manipulable. Such widgets might be manipulated voice or gesture.

Widgets developed in such a manner can control state in real systems, and it may prove a useful basis for ubiquitous HCI. Open your moleskin book to the right page. Tweak some virtual sliders and checkboxes. Take some looseleaf paper and write out a few programs or program-references. Tape them to your environment, and now you have ubiquitous widgets.

Hmmm. Widgets represented in this manner have a sort of spell-books and painted runes feel to it. Shall we call this brand of ubiprogrammers ‘Widges and Widgards’? ^_^

The notion of live coding is orthogonal to ubiquitous programming, but I think they interact nicely at the boundary between programming and HCI – for developing these widgets and REPLs.

Leveraging Color?

With pen and paper programming, we lose syntax highlighting. Developers who want multiple colors would need a handful of pens (or crayons). For ubiquitous programming and computing, I am disinclined to support semantic color for at least six reasons:

  • If the colors have different semantics, then I must waste precious idea-time sorting my pens.
  • Keeping multiple pens of different color means carrying many more pens, but more than one and a spare is unacceptable for my pocket space.
  • Computer vision algorithms will more easily handle simple contrast.
  • Colors don’t handle well under varying light conditions (esp. colored lights).
  • For ubicomp/HCI, invisible inks (IR, UV, crystalline diffraction) might become popular, but generally lack the flexible range of color information.
  • Simple contrast will work in many more ubiquitous programming scenarios.

But color might still find a few niches – not for ubiquitous programming, but perhaps for a more artistic programming or compilation.

Next Steps

I think ubiquitous programming with pen and paper is powerful, feasible today, and an exciting possibility. While glasses would be preferable (especially while standing or moving), the phone is more widely distributed, more accessible, and would be a better place to start. Computer vision algorithms we have today are generally sufficient to the purpose. A default cloud could be provided easily enough (preferably open source).

What I have not discussed, and what needs to be done to make this a reality, is integration of the flexible syntax with a programming model. It doesn’t need to be RDP (though I’d appreciate it!). I spent most of 2007 thinking about syntax, and my general conclusion was to put it off as long as possible while preserving static comprehension of code.

I wonder if an approach to user-defined syntax might be the way to bootstrap ad-hoc visual and diagrammatic syntax. Combined with the virtual program object, we don’t even need to declare the syntax ahead of time – we can just start writing then clarify afterwards (syntax as yet another subprogram relationship!).

This entry was posted in Distributed Programming, Language Design, Modularity, Open Systems Programming, Stability, State, UserInterface and tagged , . Bookmark the permalink.

7 Responses to Ubiquitous Programming with Pen and Paper

  1. Guy Leblanc says:

    I was using pen and paper to do programming in the 80. I had then to retype everything at the end when I was ready to test my program but most of my programming was done with pen and paper.

    Today I use Freeplane on my laptop, iToughtsHD on my iPad with a python scrypt to generate files from a mind map. I complete that with emacs when I do work that can be help by using a real programming editor like multiline block edit. I call it mind map programming. It give me the possibility make visual link, use color and bold text on my code. I use markdown in my mind map to write text that i will want to share with other. Everything is in one file.

    This approach is far from perfect for the moment, and it is not as natrural as using pen and paper but it does work very well for me.

  2. Pen and paper will do great – or some other medium that doesn’t distract us. The important thing is that we get a time-slice of undistracted concentration.

    On the other hand, playing with my kid or making jokes with my spouse triggers my creative cells. So I guess it’s not all black and white.

    Our brains work amazingly don’t they?

  3. Somehow your story reminds me of Rich Hickey’s talk.
    Hammock driven development.

    Although I like your story better. Rich’s story is too solitary.

  4. Jonathan says:

    Great ideas. Might think back to APL and its tradition of considering notation as enhancements to thought for ideas. If I was using pen and paper to write programs I wouldn’t confine myself to a linear ASCII text representation. I have a feeling code would look more like mathematical notation than verbose text.

  5. Blaise Pabon says:

    What a great idea! This could leverage the benefits of handwriting and the power of layout symbols to create the bridge between the image on the page and the meaning it represents.

    I was a late bloomer with handwriting. As a child I moved through a series of educational systems, British, US (private), Spanish (public), Puerto Rican (Catholic) so my handwriting was a mess with each one trying to get me to write their way. As an adult I bought a calligraphy kit and it 15 minutes I realized I had misunderstood the structure of the roman alphabet. Script is merely the repositioning of a set of primitive shapes into more complex ones, *not* the drawing of an individual letter!

    Later, reading Scott McCloud’s *Understanding Comics*, I learned that comic books use conventions to indicate the relationship between frames and also between elements inside a frame. It’s conceivable that this could be included in your notation. Markdown is a related tool which comes to mind. If the “glass” had api which provided abstraction layers for script and notation, it’s conceivable that it would be easy to adapt other writing systems and subject-specific notations.

  6. Pingback: Programming with Augmented Reality | Awelon Blue

  7. Thomas Fitzpatrick says:

    This could be done with the Anoto dot pattern technology. It has an optical reader that reads a dot pattern that can track positions uniquely despite skew/angle. This requires buying special paper right now from Livescribe – but they could just print a piece of IR filter plastic with tiny holes for all of the dots. Adding this to any screen would make it a writing surface when combined with a the bluetooth optical reader pen allowing you to trace the outline of the screen to register with any screen. Carbon paper and a stylus would achevie the same for paper. Too bad this technology is locked down in patents. My friend tried to give Livescribe this idea for months and never even got talk to an engineer.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s