I. Introduction — Speaking in Code
In the Djinni’s lair they speak in code. Their Djinnish language does not reflect a subservient nature as they announce “your wish is my command.” They have mastery over elements of our mortal realm, their words cause events to occur. What people hear in their words is their imperative language translated into our human tongues. They speak in commands, power words with concrete effects. “Open sesame” opens doors, a wish becomes tangible possibility. Humans translate thoughts into operations using computational media. The languages humans use to express these commands affect what they produce using these media. The process of translating from ideas into imperatives has profound consequences. Aladdin’s tale narrates one example of this process, the tale here tells another.
Computational media reflect the previous generation of media. They also offer new characteristics unseen in earlier technological media. What are these new characteristics and how can they be examined? There are many varieties of computational media and it is quite difficult to isolate a particular subset to begin investigation. However, the tools used to present and create media art lie behind every media artwork. The theory of programming languages is a useful means by which to characterize these media. Formal languages offer broad insight into the nature of computational manipulation and specific organizational structures of imperative languages reveal reflections of these structures in media software. This is a natural reflection because the theory of languages expresses organized models for executing algorithms and structuring data, which are the types of manipulations human creators perform on media when treating it as computational data. Finally, this is a means to characterize only formal aspects of media manipulation, not the semantic. The ideas here are not presented in a technological determinist vein — stating that it is programming language theory that caused this restructuring of media. The influences of media and culture upon each other are mutually dependent. This will be shown by reviewing examples of these concepts of formal manipulation from both art and computing. The reflection of programming languages in computational media should be no surprise, and using the tools of programming language theory to consider such media provides illuminating insight.
II. Interfaces Influence Art
I sit now in an accursed palace, corrupted by the literal and malicious nature of a Djinn. I am a humble man, made from earth as we all are. Still, that the language of the Djinni, of those beings composed of fire, should be so malevolent, is a fact that I shall always lament. When he emerged from the lamp and acknowledged my three wishes, when I wished for the fulfillment of my artistic vision as a lavish palace of lacquered coral and gold, alas I did not realize the venom dripping from fate’s bite. Had I been a stonemason and crafted this castle with my own two hands the ceilings would not be so low that my back scrapes them as I shuffle, bent over, to my throne. Had I been a carpenter my satin canopied bed would not be so splintery that it takes thirty minutes for me to slip safely from it at dawn. Had I been a master of the olfactory arts of oil crafting and perfumery I would have been able to banish the aroma of sulfur and replace it with myrrh laden zephyrs wafting through every corridor. Alas, the tool I used was a contractually obligated, but ungrateful and rancorous, Djinn. The words “make for me a lavish palace” were made reality by the Djinn’s language and all that I had left unspecified was decorated by his cruel tastes. My consolation is that my request was not spoken: “make me a lavish palace” (as the colloquial version would have been worded) lest instead I would now be the inhabited abomination rather than be inside of it!
Artistic works are related to the tools used to create them. The mark of the tool is apparent, whether the work has an explicit relationship to the tools as in tromp l’oeil painting which reveals it effect only when the mark of the tool becomes recognized and the illusion is broken, photography which in the past has been presented by news media as a transparent tool that represented a window to reality — an aura which continues now despite its double nature now as a purveyor of false images, or Nam Jun Paik’s cyborg materials to evoke his “cybernated” art concept. This paper considers primarily screen based media artwork generated on a personal computer. Typically these works are created using commercial software such as Macromedia Director, Adobe Photoshop, Adobe Premiere, etc. Of course some minority of artists also create their own computational tools. Regardless, even these artists are influenced by the medium of their art and it is important to understand what this influence is when engaging their work.
A starting point in this examination is the recognition of the role of the interface as being intimately tied to content. Douglas Englebart was one of the early innovators of the contemporary computer interface. He designed a networked environment designed to support collaborative interaction between people using computers. Englebart is known as the inventor of the mouse, windows, email, and the word processor. In his times even the idea of a computer as a monitor connected to a console with an input device such as a keyboard was novel. It transformed the relationship between people and computers in the fifties and sixties. Englebart proposed “a way of life in an integrated domain where hunches, cut-and-try, intangibles, and the human “feel for a situation” usefully co-exist with powerful concepts, streamlined terminology and notation, sophisticated methods, and high-powered electronic aids.” Echoes of the contemporary computer interface reverberate from his speculative way of life. When describing a tool that an architect could use to describe a working list of specifications and considerations he wrote: “the lists grow into an ever-more detailed, interlinked structure, which represents the maturing thought behind the actual design.”
Englebart describes the language of his augmentation medium as “the way in which the individual parcels out the picture of his world into concepts that his mind uses to model the world, and the symbols that he attaches to those concepts and uses in consciously manipulating the concepts (“thinking”).” Process hierarchies represent the hierarchical approach that humans use to solve problem. He viewed symbol structuring as a crucial part of language, a new means by which people could “begin experimenting with compatible sets of structure forms and processes for human concepts, human symbols, and machine symbols.” The cumbersome acronym H-LAM/T (Human using Language, Artifacts, Methodology, in which he is Trained) reveals the connection between human work and machine symbols underlying Englebart’s work. The primacy of language as a structuring device is apparent and the key idea of associating concepts and names within a hierarchical structure is a main theme.
Another example of the connection between the conception of media production and the user interface is apparent in the work of Alan Kay. He invented the object-oriented language “Smalltalk” and many innovative features used in contemporary interfaces. Kay’s slogan is “Doing with Images makes Symbols.” In this case, he associates the symbolic with the imperative object oriented programming language Smalltalk (the mouse is associated with “doing,” and “images” are associated with icons and windows). Object oriented programming means that objects know what they can do. In this model it is possible to select an object and send it a message asking it to fulfill the user’s desires. The paintbrush knows what colors it is allowed to paint with, all the user has to do is tell it to use one. The concept of the symbolic is what allows manipulation of objects in this way. Kay describes the symbolic as that which enables humans to tie together long chains of reasoning. In this regard, the reasoning process of the user of the interface is expected to correspond to this symbolic model embodied in the programming language paradigm.
Working in a realm defined by what the program and its programmer see as the user’s mode of working influences the method by which the user engages those using the tools. The user of a computational interface can then ask “where lie romantic notions of art process?” “What of inspiration and random or spontaneous methods?” “What of non-hierarchical thinking and procedure?” Certainly conventions of artistic production have not disappeared in media production software, in fact artistic paradigms have been as hard-coded into these tools as programming conventions. Adobe Premiere does not force users to think in terms of frames and transitions that are inherited from conventions of cinema or constraints imposed by hardware in older media technology (such as frames and linear time progression). Despite the fact that the influence of artistic production and technology is a two-way street it is useful to examine the direct influence of software upon artistic production. To understand the assumptions made by user interface designers it is informative to look at programming language theory. Then it is possible to step back and look at how software such as Photoshop or QuickTime leave their marks upon media production.
III. Formal and Programming Languages
I do not wish to credit myself too greatly. I was a humble fisherman, and a dabbler in algebra and alchemy, before my discovery of the lamp. My knowledge of fish far exceeds my knowledge of unearthly creatures. Still I cautiously praise myself that, in my elaborate, if cramped, gardens I planned my next wish for months. I crafted my language perfectly. I would be completely literal and unambiguous in my next request for the Djinn. I invented a language. It would specify first the number of requests and subservient wishes embedded within my one wish. It was not an attempt to greedily exceed the bound of two more wishes. My desire was only enough precision so that I would not be thwarted and cursed once again rather than finally granted boon. This language would translate my clay human tongue to the language of spirits. My plan had grace and elegance, it put my knowledge of the algebra to its greatest test. In the end, I succeeded at that! Why then, reader, do you detect the disconsolate tone in these words. Because as I prepared to say my formal, refined, and completely specified wish for a bride that would inspire clouds to gather and disperse, waves to roll more quickly, and my heart to infinite joy — I realized that in uttered word my wish would take me four thousand and ninety six days to pronounce.
A formal language is defined as a set of finite strings, each made up of atomic symbols. Formal languages can be denoted by a mathematical structure called a grammar. The main form in a grammar is a “production” which describes a rule by which one string in a language can produce another. Languages are classified according to the types of productions allowed in them, and this is one way of characterizing the power of a language. A very simple type of grammar is defined by regular expressions. Regular expressions allow for composition of strings in a few very particular ways. These are: alternation, concatenation, and closure. To make an analogy with natural language: alternation means choosing one expression from two lists of expressions in English, composition means joining two expressions together to form one, and closure means describing expressions in part (e.g. looking at only expressions that contain the phrase “media art.”)
More complicated formal language systems allow naming of separate parts of language and specifying the ways in which they are put together. This is analogous to describing the English language in terms of paragraphs, words, or letters. It is possible to describe rules for how these pieces fit together. More powerful formal language systems allow use of these rules to describe the function of the language and actually make deductions using the language. These types of languages are an abstract way to categorize programming languages. Taking a step back reveals that the methods of composition used in formal languages are mathematical descriptions of some processes humans use to think and create artifacts. Making film consists of concatenation of frames or clips onto one another. Editing music uses the principle of alternation as clips are spliced together from different sources. Compositional elements are often labeled to aid in the process of recombination. Rules are defined for how these elements should be recombined — personal rules, rules of convention, and physical rules limited by the physical means by which the media may be assembled.
In computational media these rules can be automated and implemented algorithmically. Moreover, it is possible to force compliance to a particular set of rules to assure that a particular organizational strategy is followed. It is possible to concatenate two film clips using any of a variety of dissolves or wipes. One may pick images to insert in word processing documents as easily as one can cut and paste words from a variety of different sources. One can label these individual segments, paste them in layers over one another, perform searches through them, or substitute one element for another using convenient thesaurus features. Formal languages provide a concrete way to talk about these processes that people engage in with texts. One can go further, however, and look at some of the constructs of modern imperative programming languages. Nearly all commercial software is constructed with these languages and the structure of the languages is directly reflected in the structuring of the software.
Programming languages are designed to fit very particular criteria. A good language is designed for: abstraction, orthogonality (features should be free from unexpected interaction), simplicity, regularity, consistency, and ease of translation. Not every language fits these criteria, but as abstract goals it can be seen that these features also carry through to software interfaces. The goals of regularity and consistency are evident in the means for cutting and pasting provided in many user-interfaces. The method of selecting “cut” from a menu is the same as the means for selecting “paste.” This may seem obvious, but another choice could have been to make the means of cutting analogous to the real world of cutting and the means of pasting analogous to the real world experience of pasting resulting in different modes of interaction. Instead, however, the mindset associated with the design of traditional interfaces has been influenced by the design of programming languages since the early days of Douglas Englebart’s research in the fifties. Hierarchical organization and categorization within lists of information are often the underlying means by which media production software is forced to structure artistic information. Despite the presence of a “cut” command, there is no easy analogy using commercial software to the cutting of a text into pieces, tossing those pieces into a shoebox, and recombining them at random. There is no organizational feature in Adobe Photoshop analogous to dumping a pile of photos onto the floor and running ones hands through them until s/he finds the one that “calls out to her/him.” There are no commands to force a font to forgo its natural irregularity and take on an expressionist texturing or to become blurred by teardrops such as is possible when writing a handwritten letter.
It seems unreasonable to expect such individualized features to be available when working with computational media, yet it is far from unreasonable to acknowledge that artists, scientists, and casual creators of media work in extremely diverse and personalized ways. It is far easier to achieve a lack of orthogonality when working in paint (mixing in strange fluids to create unexpected interactions) than it is in commercial software. It takes an extraordinary amount of effort to create one’s own software production tools that do not enforce the features of a programming language. The argument being made here is that these features of formal and programming languages are not unique to technological media, but their enforcement and implementation are. The restrictions imposed by these features and the power of structuring information and media in complex patterns and manipulating it algorithmically are not easily separable from the creation of computational media or the experience of being an audience of it.
IV. Characteristics of Programming Languages
I was happy for a time, I had forsaken my wish for a bride and was married instead by love, providence, and whatever fortunate design allowed my simple person to appeal to my mate. My second wish was for a ring fit for her desires, and somehow divine predestination allowed this simple request to be fulfilled. My final wish, I concluded, would be grand. A wish for love, a wish for our future, a wish for forever, a wish that would make each day as fresh as sipping from a chalice brimmed with morning dew, fresh as new moments after a rainshower. So I toil still, to make my perfect language more concise, to let it be spoken naturally. I must let it be composed as neatly as a fern’s leaves, but it must be as infinitely expressive as the conch’s spiral. It must signify wonderfully like Mowlana Jalaluddin Rumi’s poems. It must embody the clear thought of Abu Ja’far Muhammad ibn Musa al-Khwarizmi’s mathematics. In these words I realize that it is not the task of one man to create this language. I do not pretend to be the superior of these thinkers and feelers that loom across time. I am in their shadows. With the diligence of women and men across many times, throughout many lands, and with the proper blessings, I hope one day my dreamed language will be expressed. Until then, I reserve my love-inspired wish and ponder more the structure underlying the speech of the beings made of fire whose words cause action and reshape our earthbound experience of this world.
Following this account of programming languages and their appearance within media creation tools it is possible isolate other relevant characteristics of programming languages. Important characteristics include: reference, control, abstract data structures, block structure, and polymorphism. There are direct connections between each of these concepts and computational manipulation of media.
Reference is the idea that informational elements and their names are separable within a computer. Naming systems can be introduced to describe a text in some media and fluidly change names and content independently. In a trivial example, sections of a story can be labeled as: “beginning”, “middle”, “climax”, and “ending. ” Then it is possible to swap out the information labeled “climax” with revised information, thus retaining the story structure. This example is simplified but not far-fetched. Professional graphic designers use such features in software regularly, as do professional sound technicians.
Control means that it is possible to structure the flow of information in a system. An example of a control structure in a programming language is a loop. In both music and video processing software artists can create explicit media loops. In a great deal of modern popularized music computational media are used to create drum loops as a basic unit of music composition. The video and installation artist Bill Viola is vitally aware of this as he states that “viewing becomes exploring a territory, traveling through data space.” Control structures represent the means used to shape these paths through media data space. Bill Viola continues his description of computational media manipulation asserting that “editing will become the writing of a software program that will tell the computer how to arrange (i.e., shot order, cuts, dissolves, wipes, etc.)”
Abstract data structures are the means by which data spaces are organized. Lists and arrays of information are basic compound data types, but one can combine base data types in any variety of abstract ways to represent current needs. Circles become numbers: integer coordinates for the center point and a floating point radius, dates become several fields of integers. Basic programming structures became revolutionary structures when introduced explicitly to media art. This can be seen in work such as Luc Courchesne’s “Family Portrait” which uses laserdiscs indexed by HyperCard stacks to allow users to respond to questions asked by members of a family to create startling sensitive interactions. When describing ways to construct video artwork, Bill Viola refers to “visual diagrams of data structures already being used to describe the patterns of information on the computer video disc.” The medium of a videodisc can be seen as merely a digital update of former videotape media, but when looked at for the actual way in which the information is structured a quite distinct nature is revealed. The structure of the computer programming language has been used to restructure the information of video data. Data types such as matrices, linked lists, and records have become the status quo for the organization of media in digital formats.
Block structuring allows for hierarchical organization of computer programs. It also allows each block level to be manipulated separately. This structuring is pervasive throughout media production software. Text can change font or color at the letter level, word level, sentence level, paragraph level, page level, or document level with precision and ease. Photographs can change onionskin layer at a time, affecting lower levels but not higher levels. These levels are nested into hierarchies with ease and can be imported into other such hierarchies at will. Whereas the nature of a photographic manipulation could be seen as based in continuous fluid and light when done using traditional darkroom techniques, professionals using media software often work using block structures. The software is structured in such a way as to discourage and inhibit other methods of use.
Polymorphism is the idea that data or functionality can change depending upon its context. The extension of this idea is that structures for manipulating one data type can manipulate data of another type. It is possible to use a jpeg image just as easily within a word processing document as it is to composite the image in a photographic document. The data here is not actually polymorphic, but the concept of using the data in such a variety of settings is. Cutting and pasting a graph is done using the same mechanism as cutting and pasting a paragraph. The same “play” button on a Quicktime player starts playing an audio file the same way as it starts a video file. This software exhibits the traits of polymorphism. This type of media fluidity, unknown in previous media, is becoming an expected trait of the experience of computational media.
It is certainly possible to look at various models of programming languages in order to elicit more parallels in computational media. Those listed here represent some of the most important characteristics of computational media that are often enforced by software. The idea behind such programming languages extended far beyond computer science and in some sense are general concepts to organize information. The concentration of all of these modes of operation and ways of thinking, their rigorous enforcement by a machine, and the compliance of media artists to their tough strictures (by default of the tools they work with) is something new.
The havoc unleashed by Djinni’s granting ill-considered wishes by humans reveals the Djinni’s lack of concern for the environment in which s/he is operating. The considerate Djinn probably could understand that the human languages cannot properly express wish fulfilling words. Likewise, humans should realize that computational languages have effects upon what we create using them. This essay is meant to use accounts from theory programming languages to reveal a glimpse into the computational medium and the ways in which a meta-medium is not a consolidation of previous media but has its own recognizable traits and languages. No creation by any artist can escape this chain and transcend the nature of its medium in a material sense. The expressive, analytical, evocative, or otherwise subjective interpretation or intent of the work can certainly transcend the computer, but in a concrete material sense the mark of the programming language as a primary characteristic of computational media is always evident.
 Nam June Paik. “Cybernated Art,” Manifestos, originally published in Great Bear Pamphlets (New York: Something Else Press, 1966). Reprinted in Theories and Documents of Contemporary Art: A Sourcebook of Artist’s Writings, Berkeley, CA: University of California Press, 1996) and Multimedia: from Wagner to virtual reality, edited by Randall Packer & Ken Jordan, New York: W.W. Norton and & Company, Inc., 2001. p. 39-43.
 Douglas Englebart. “Augmenting Human Intellect,” originally published in “The Augmentation Papers”, Bootstrap Institute, 1962. Reprinted in Multimedia: from Wagner to virtual reality, edited by Randall Packer & Ken Jordan, New York: W.W. Norton and & Company, Inc., 2001. p. 64-90.
 ONLine System (NLS). developed at the Augmentation Research Center of the Stanford Research Institute and unveiled in 1968.
 Alan Kay. “User Interface: A Personal View,” The Art of Human-Computer Interface Design, edited by Brenda Laurel (Reading, MA: Addison-Wesley Publishing Company, 1989. Reprinted in Multimedia: from Wagner to virtual reality, edited by Randall Packer & Ken Jordan, New York: W.W. Norton and & Company, Inc., 2001. p. 121-31.
 Lev Manovich. The Language of New Media, Cambridge, MA: The MIT Press, 2001. p. 140-41.
 Noam Chomsky. Syntactic Structures. The Hague: Mouton, 1995.
 Ryan Stansifer. The Study of Programming Languages. Upper Saddle River, NJ: Prentice Hall, 1995. Chapter 2.
 Ibid. p. 5.
 Mowlana Jalaluddin Rumi is recognized internationally as one of the world’s great literary figures.
 Abu Ja’far Muhammad ibn Musa al-Khwarizmi is generally recognized as the “father of algebra.”
 Bill Viola. “Will There Be Condominiums in Data Space,” originally published in Video 80, no. 5 (Fall 1982). Reprinted in Multimedia: from Wagner to virtual reality, edited by Randall Packer & Ken Jordan, New York: W.W. Norton and & Company, Inc., 2001. p. 287-98.