A hand-sketched UML diagram.
I first became aware of sketch-based user interfaces as an undergraduate in 1996 when I saw James Landay's faculty candidate presentation on a sketch-based tool for user interface design. I was intrigued by the idea that computer interfaces could be as simple as pen and paper, but it didn't immediately grab me as something I would want to work on.
Several years ago, while working on problems in information visualization and design technology, I developed an inflammation in my hands that made it difficult for me to type. I began to off-load as much programming as I could onto the people I was working with at the time. I did this by one-on-one meetings where I would specify object-oriented systems by drawing UML diagrams on a whiteboard, explaining and discussing as I drew.
Afterward, my collaborators would implement what we had talked about in software. Over time, this process evolved into a very efficient way to design software systems--efficient for me, that is, but tedious for the person who was actually implementing the code. Through this experience I became increasingly convinced that if there was a way to automatically capture, interpret, and process hand-drawn UML diagrams (to generate the skeleton code of an implementation, for example), it could change the way software is designed and implemented.
Although it was the semantically-rich application of UML that initially sparked my interest in sketch-based user interfaces, I have since become obsessed with semantically-poor user interfaces. Could there be a small set of syntactic constructs that, if supported well in a sketch-based editor, would make digital paper preferable to pen and paper? And is there a robust yet extensible way to perform recognition of these constructs? I am currently exploring these issues in the context of a sketch-based version of Microsoft's PowerPoint presentation software.