Code as Collage

One of the interesting processes that I’ve been undergoing as I learn to code is thinking critically about the graphic user interface. While I always knew about the command line, I never really connected with it on a personal/intimate level (now we super intimate!). I’ve been thinking recently about how much the interface we use to connect with our computers shapes the ways we think about them, think through them, and what we think is possible.

I’ve always been very interested in language, the gaps between understanding, and the ways that form influences our understanding of function (for example, metaphors as an interface that shapes our understanding). I thought a lot about this when I studied anthropology of law, a life-changing course I took in college that not only taught me a lot about language and the different ways of establishing truth, meaning, fact, etc, but also turned me into a morning person (it was at 8:30am)! Valuable life lessons. There’s an interesting anthropologist of metaphor at Johns Hopkins named Emily White, who looks at these questions. She has one essay that looks at the metaphors we use in biology textbooks. For example, when describing sex cells and the egg and sperm, many textbooks use gendered metaphors and images that are biologically inaccurate, but tie into our obvious cultural relationships with these processes. I think it’s clear that there can be implications for how people learn biology and think critically about the paradigms of the field and what is possible via the language they understand it in.

Until now, I’ve never questioned the GUIs of my computer (and other devices), nor thought about how they might affect me and how I think on these devices. It was interesting to hear my teacher Avi speak recenly about how he thinks via text editor. This question of how you think digitally is one that has been on my mind recently. I was speaking to a friend, and I mentioned that I thought I would never be able to really think on an iPad or other tablet, and need a computer to write, think, and analyze. He said that perhaps this was just because that was the form I was used to thinking on, had learned how to think on, etc. People who have tablets since they are really young have learned to think on them, just as I learned to think on my computer. This was really interesting, and I actually have noticed that I’ve been better at thinking on my phone since thi conversation. It made me realize that (at least to an extent), thinking on a device is not limited just to the functionality horizons of the device, but also to the user’s history and experience with the tool. Since I’ve been learning to program, I’m much better at thinking in text editor, instead of in Microsoft Word or a blogging platform.

Another interesting comment Avi made around the same time was in response to a question a student had about how to start writing code. I think many of us feel, at least some of the time, that we understand the concepts we’re learning and even the code, but then when we sit down to actally write code and solve a problem, we don’t know where to begin. Avi advised us to just start somewhere, and spoke about how when he was writing, he would use the same paragraph as his first paragraph, and then be able to start writing his story without worrying about the first paragraph. By the time he finished the story, he took out the paragraph. However, as he was speaking about this, I was wondering how this may have affected his stories, the themes they circled around, etc. It seems inevitable that they would be somewhat affected, just as when you read something, you start to get the rhythm in your head and reflect it in your writing.

Similarly, visual imagery, sensory experience, and interactive quality of the GUI must affect the ways we think in it.

I found a really interesting blog that asks these kinds of questions called “The Graphical User Interface. Time for a Paradigm Shift?” You can find it here: http://sensomatic.net/gui/ . One of the most interesting parts of this blog is the timestamp – it was written by Christine Zmoelnig, who did her masters in Hypermedia Studies in London in 2000.

Everything about this is fascinating. First of all, what is “hypermedia”?

Upon further research, I found that hypermedia is an extension of the concept of hypertext into other forms of media to generate a non-linear medium of information. Hypermedia is different from multimedia, which is more linear. Hypermedia is often used in reference to electronic literature. The term was first used in an article called “Complex information processing: a file structure for the complex, the changing and the indeterminate” in 1965 by Ted Nelson, whom the internet calls an “IT sociolosopher). He coined “hypertext” and “hypermedia” (as well as “transclusion, virtuality, intertwingularity, and teledildonics”…) Interestingly, he called Tim Berners-Lee’s creation of the internet as an oversimplification of his ideas: “HTML is precisely what we were trying to PREVENT— ever-breaking links, links going outward only, quotes you can’t follow to their origins, no version management, no rights management.” Nelson was influential force for a lot of thinkers. Douglas Adams, author of Hitchhiker’s Guide to the Galaxy, made a documentary in 1990 called Hyperland about “hypertext and surrounding technologies” (I’m quoting from Wikipedia, but I love the idea of surrounding technologies). This movie sounds amazing – like a combination of “Science of Sleep” and the Beatles movie “The Point!”. It’s a fantasy documentary that begins with Adams asleep by his fireplace with his TV on. He starts dreaming, and finds himself in his dream taking the TV to the garbage dump, where he meets Tom, a “software agent” (which apparently means a computer program that acts on your behalf). Tom shows him the future of TV: interactive multimedia. This leads them on a journey, which includes interviews with a number of people experimenting with interactive media in 1990. The film ends with a vision of how information might be accessed in 15 years, which for the film is 2005. Upon retrospect, the film is remarkably prescient for a work that came out about a year before the first web browser was released. All this makes sense in some way, as I think hazily back to when I read all five books of “Hitchhikers Guide to the Galaxy,” which starts out linear, and by the end becomes very, very non-linear.

All this is a useful framework as I look at Zmoelnig’s GUI blog, thinking about her background and approach at the time of writing. Some of her blog post topics include “Visions of the new paradigm”, “History of the GUI”, “The GUI’s Impact on Society”, “Mac OSX ‘Aqua’: A paradigm shift in GUI design?”, “3-dimensionality as a quality factor for the user?”, “The new ‘look-again-look’”, “‘Alternative Models’ Time for a different approach to GUI design”, and “A new medium for GUIs: the Internet”. It’s a very inspired, idealistic, and flexible vision of the GUI. In some way, I think that perhaps because there was so much change happening in that space at the time she was writing (not that there isn’t change today, but we have a lot more history/heft to go with it) that she was able to think in this flexible way, and ask questions I haven’t seen asked explicitly in many other places about the GUI. On one hand, UX/UI and interaction design are really important spaces for innovation at the moment. However, at least in what I’ve come across, they start from level 2, with assumptions taken for granted about the GUI already embedded within them, versus asking those questions and exploring those pillars explicitly. Many of the search results I found when searching “alternative GUIs” are older pages, or about GUIs for very specific components of an operating system, which may strengthen this point. Furthermore, there’s the question of how alternative an alternative GUI would be. There’s the level of changing colors, shapes, etc (i.e themes), and then there’s the level of changing the associative nature of our computers. Zmoelnig presents one of the few search results I found that explores the latter. She looks at approaches to changing the human-computer interaction via 3d, virtual reality, natural language processing, and the ways that the internet presents some new ways of thinking about these questions.

It’s interesting to look at her description of the internet. She talks about the internet and time: “the internet also brought a dynamic dimension into interface design: Time-related navigation concepts like back-buttons and chronological file-structures have only come up with the internet.” This is interesting for me to read – I’ve never thought about the internet as explicit chronology before, and this also makes me think about other spatial and temporal relationships the internet does, does not, and could potentially set up. It’s also interesting to think how this has changed since 2000.

She also talks about the desktop, which is another thing I never thought critically about until I read her work. The metaphor of the desktop as our homebase for computing via the GUI is interesting, as it references a working site very different from the main one we use today. The spaces and forms we work in are also constantly evolving. Going back to metaphor, language, and description, it’s interesting how ideas from one era bleed into and shape our conceptualization of the next.

There’s a lot of interesting stuff going on in her work. As I read through it, I observed a few interesting things. When I started reading Zmoelnig’s site, I thought of it as a blog. As I continued to explore the differnet links and refernces, I started to realize that it was actually her dissertation in hypermedia. In some way, the framework of hypermedia that I sought to understand in order to contextualize her thinking led me identify her work as such. Hypermedia shaped my exploration of the concepts she presents, and it shaped my research of the hypermedia about hypermedia. The process of this research is an interesting framework in itself.

Secondly, this exploration opened up my eyes to a few different interfaces I have always taken for granted in my computing history, from the desktop to internet chronology (i.e. the back button).

This all stemmed from what has been happening as I learn to code from the command line, and how that has been changing the way I think about computing, and about breaking down processes more generally. Command line coding is both extremely hypermedia and totally singular in its point of origin. I’m not sure what all this means yet (or ever, probably) but I do think it’s important to understand the implications of both what we code and the places we code in. By making these explicit, it enables us to better understand about how our thinking occurs within them. From there, we can take understandings from different fields and apply them to our limitations within the medium. So, in sum – code as collage!