In defense of drawing by coding

Just some half-baked thoughts in repsonse to Bret Victor’s talk, Drawing Dynamic Visualizations (along with his addendum here). As usual, it is a fun and inspiring talk, so if you haven’t seen it I highly recommend it; but I will summarize it here.

He starts by surveying the state-of-the-art in options for the creative scientist who wants to visualize some data. The options he outlines are:

  1. Use some program like Excel which has a standard repertoire of graphs it can generate. The problem with this approach is that it completely stifles any creativity and freedom in visualizing data.

  2. Use a drawing program like Illustrator or Inkscape. This gives more freedom, of course, but the process is tedious and the results cannot easily be modified.

  3. The final option is to write some code in a framework like Processing or d3.js. The problem here, Victor says, is that you are just staring at a mass of symbols with no immediate, dynamic feedback.

He then goes on to demo a really cool prototype tool that allows drawing using a graphical interface, a bit like Illustrator or Inkscape. But the similarity is only surface deep: where those programs are restrictive and inflexible, Victor’s is richly interactive and editable. Instead of drawing concretely located lines, circles, and so on, it infers the relationships between things you are drawing, so updating the characteristics or positioning of one element automatically updates all the others which depend on it as well. In other words, one can construct a generic, editable visualization just by drawing one particular example of it.

Victor is quite negative about option (3) above—drawing by coding—referring to programming as “blindly manipulating symbols”: “blind” because you can’t actually see the picture you are creating while writing the program.

What I would like to point out is that in fact, despite his negativity about drawing by programming, when using his graphical tool Victor is still programming! It’s just that he has a graphical interface which allows him to construct certain sorts of programs, instead of writing the programs directly. In fact, you can see the programs he constructs on the left side of the screen in his tool. They appear to be structured imperative programs, consisting of sequences of drawing instructions together with things like loops and conditionals.

The problem is that this kind of higher-level interface cannot provide for all possible circumstances (unless you somehow make it Turing complete, but in that case it probably ceases to be at all intuitive). For example, Victor impressively drags and drops some spreadsheet data into his application. But what if I want to use data which is structured in some other format? What if I need to preprocess the data in some computationally nontrivial way? Or on the drawing end, what if I want to draw some shapes or compute some positions in a way that the interface does not provide for? We can’t completely get away from the need to write code in the service of visualization.

What we really need is a more inclusive idea of “programming”, and a continuum between direct manipulation of images and manipulation of symbols to produce images. Symbolic methods, of course, can be incredibly powerful—there is nothing inherently wrong with manipulating symbols.

More specifically, I am proposing something like the following:

  1. First, I am all for making elegant and powerful high-level graphical interfaces for constructing interactive, editable drawings—and not just drawings but code that generates drawings. This can probably be pushed quite far, and there is lots of HCI research to be done here. There are also some very interesting questions relating to bidirectional computation here: making an edit via the graphical interface corresponds to some sort of edit to the code; how can this be done in a sensible and consistent way?

  2. Recognizing, however, that sometimes you do need to actually write some code, how can we make the underlying language as beautiful as possible, and how can we make the interaction between the two systems (code and higher-level graphical interface) as elegant and seamless as possible? The ideal is for a user to be able to flow easily back and forth between the two modes, ideally spending much of their time in a high-level graphical mode.

Of course, if you hadn’t guessed by now, in the long term this is the sort of direction I would love to go with diagrams… though I need to finish my dissertation first (more on that subject soon).

About Brent

Assistant Professor of Computer Science at Hendrix College. Functional programmer, mathematician, teacher, pianist, follower of Jesus.
This entry was posted in diagrams and tagged , , , , , , , . Bookmark the permalink.

3 Responses to In defense of drawing by coding

  1. Derek Elkins says:

    You are violently agreeing with Bret. I think you have pigeon-holed what he is doing and trying to do.

    Bret is definitely not against programming which is why he uses the term “coding” which he is against. See the FAQ for this talk located here: The first (of two) questions is “Is this ‘programming’?” The answer of which begins, “Sure. The artist is certainly creating a computer program.”

    If you haven’t already, I’d read these fairly short notes from the talk “Media for Thinking the Unthinkable” to provide context on where Bret is coming from and what he is aiming at overall.

    Finally, he’s not advocating a purely point-and-click approach to programming. This is addressed from one angle in the other question of the FAQ, but he also gives an example of the same concepts that turn “coding” into “programming” in the context of a textual representation in his essay: albeit still in a context where there’s a relatively clear direct visualization. As you’ll recall, “Inventing on Principle” had some examples of him attempting to address more abstract domains.

    • Brent says:

      I think you’re right that we probably agree more than I thought. But I think there is still an important point of difference, namely, that I am not against “coding”. Maybe that just means Bret is more far-sighted than me and my work will fall by the wayside; oh well.

  2. Tobias Florek says:

    i am amazed that bret does not seem to know of tools like geogebra which have existed since a _very_ long time.

    but to get to the point: i’d like to see a visual tool in which you can play around with as in geogebra (which is superior to bret’s tool at least for now) but can also include from within (say) haskell to feed it data. it would be even better if you could somehow include your own data manipulation “code” into your sketching using the tool. i am looking forward to what you have in mind!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.