multimodal

As scholars experiment with collaborative, multimodal approaches to analyzing electronic literature, the tools, methods, and practices of such collaboration become increasingly an issue. How do we share, edit, archive, and publish arguments that address and evolve across multiple types of data, platforms, and disciplines? How can the approaches (data visualization, code analysis, textual explication, bibliographic history, etc.) be shared in ways that other scholars can engage not just with the final interpretations but also with the processes that lead to them? Recent publications such as 10 PRINT CHR$ (205.5 + RND (1)); : GOTO 10, represent the value of such collaborative efforts in combining media archaeology, platform studies, software studies, and Critical Code Studies. Our own work in collaboratively close reading William Poundstone’s “Project for Tachistoscope: [Bottomless Pit],” which we presented at ELO 2010 (held at Brown University) and are now developing as a book for Iowa UP, has prompted us to reflexively consider how the processes of our own collaboration might prove generative to other scholars. Supported by an ACLS Collaborative Scholarship Fellowship 2012-2013, we are developing an open-access scholarly website to facilitate collaborative critical interpretations of digital art, a platform for digital humanities scholarship focused on born-digital poetics. The goal is to produce a workbench where scholars can apply critical tools to works of electronic literature and share the results of their investigations. We propose to present this website, in its nascent stages, and discuss its ambitions and affordances to producing complex, multimodal, and collaborative critical readings.

ENGLISH SUMMARY Digital Poetry: Aesthetic analysis and the role ofmediality in the communication of artwork Digital poetry (language-based digital art) is a global, interdisciplinary movement consisting of poets, artists and programmers who study and develop opportunities for programmed writing. Digital poetry combines writing with animation, images and sound. There are moving letters, interaction and autogenerative programming. Some digital poems also consist of actual programming code. Digital poetry can be colourful, expressive, technologically advanced, organic, delicate and minimalistic. The thesis consists of analyses of selected examples of digital poetry and investigates, discusses and demonstrates how digital poetry can be analysed. This results in a wide range of theoretical issues concerning genre and intermediality, media philosophical questions regarding technologies of writing and issues related to programming, materiality, temporality and agency. The thesis is a methodological reflection on which concepts should be applied and what new set of questions should be asked in the analysis of digital poetry and contemporary digital art in a broader sense. The methodological approach is based on the theory of enunciation. This means that rather than focusing on the artwork as object or on the experience of the artwork, the analysis focuses on the relation between object and recipient and investigates the specific conditions for experience provided by the artwork. Throughout the thesis, this analytical approach is supplied with investigations that examine issues related to medial issues and their effect on the communication of artwork. The thesis contributes to the research field of digital literature with aesthetic analyses of digital poems. It argues that the analysis of operational logics (i.e. formal studies of code) and hermeneutic traditions fail to provide adequate tools to analyse the potential experiences and effects of digital poetry. Digital poetry is in the thesis characterised as a diaspora in continuation of historical literary avant-gardes, but it is also considered important to include comparative perspectives on other art forms and genres than the literary and in general to move away from literary entrenched logics by, among other things, using the more inclusive terms ‘work’ and ‘recipient’ instead of ‘text’ and ‘reader’. The thesis consists of an introduction to digital poetry, as well as to the methodology, questions and concerns of the research project. This is followed by six chapters and a conclusion. The first chapter is called ‘MO [VE.MEN] TION – Code, Materiality and Concretism in Digital Poetry’. The Australian poet mez and her work practice in which programming languages are combined with phonetic English are analysed. This raises questions of programming language versus natural language, and drawing on the theories of N. Katherine Hayles and Nelson Goodman, among others, questions concerning materiality are explored. How is materiality complex in the digital field where works should be regarded as processes and events rather than as objects? This procedural nature is made explicit in the digital poem ‘La série des U’ where the letters move, and it is investigated how that affects the meaning. The chapter finally investigates issues of concretism through a short outline of historical concrete movements in various art forms, and it discusses why digital poetry is not concretistic in the same way; historical concrete works usually experiment with the limits of the work's own art form, while digital poetry is too complex a mixture of art forms to be determined at all. Digital poetry is distinctly multimodal, which among other things means that you cannot operate with notions such as ‘writing’ or ‘text’ as the smallest medial units. This fact is important for the development of a multimodal approach to the analysis of digital poetry. Chapter two is named ‘Mediality and Historical Language Technologies’. Drawing on Walter J. Ong and Friedrich A. Kittler's analysis of historical language technologies the chapter argues for the use of a broad concept of media. As W. J. T. Mitchell and Mark B. N. Hansen have argued, the collective singular media could be used as a third term capable of bridging, or ‘mediating’, the binaries (empirical versus interpretive, form versus content, etc.) that have structured media studies until now. This bridging is exemplified by how analyses of ‘moveable writing’ are interested in the meaning as well as the effects hereof. However, analyses should not exclude empirical interest in the digital computer as a ‘language technology’ that determines the moving letters. Based on the broad media concept, chapter three, ‘Art Form, Mixture, Hybrid – The Role of Multimodality in The Communication of the Artwork’, develops an analytical approach that helps to avoid notions such as ‘writing’ and ‘text’, as the smallest medial units, by instead operating with Lars Elleström’s model of the modalities of media, in which all media consist of material, spatiotemporal, sensory and semiotic modalities. This terminology is applied in an analysis of the Swedish poem ‘Väljarna’ [The Electorates] by Johannes Helden. It is argued that traditional art forms can be defined by their specific combination of the four modalities, but that digital poetry as a genre is so composite that each new work will constitute a new combination of the four modalities. This is used as an argument to move the model from a descriptive level to an analytic one to be used on types of works where the combination of modalities is precisely ‘new’ and therefore can be said to be explored at the level of signification. The mode of investigating how the medial (in this case the multimodal) affects the communication of the works is an important part of the methodology of the thesis, and it is repeated in the last three chapters which focus on other medial elements: issues concerning programming, temporality and distributions of agency, respectively. Chapter four is called ‘Limits of Sensing, Incestuous Interaction and Breathing Letters – On Secrets of Programming and its Role in the Communication of the Work’. The chapter analyses David Jhave Johnston's digital poem ‘Human-Mind-Machine’ and discusses how knowledge of programming can be incorporated in the analysis if relevant characteristics are incomprehensible on the phenomenological level. In continuation hereof the differences between human and machine ‘senses’ and issues of interpretation and agency are investigated, followed by a discussion of whether a concept such as ‘liveness’, which is otherwise attributed to human bodies, can be used to denote the performance of digital programmes. The issue of secret programming is also discussed as a cultural issue relating to secret surveillance of data. Chapter five bears the title ‘WHEN NOW IS MORE NOW THAN NOW - On the Role of Temporality in the Communication of the Work’. By focusing on specific temporal organisations and their significance, the chapter analyses ‘Mémoire Involuntaire no. 1’ by Braxton Soderman, ‘Dada Newfeed’ by Eugenio Tisseli and ‘Last Life: Your life. Your time’ by Gregory Chatonsky as well as other types of works and digital artefacts. The analyses explore how the works thematise issues of presence, memory and trace, and focuses on how the temporal organisation determines different senders and subjects. How does it, for instance, affect the significance of pronouns in a digital poem where the words move about? The chapter makes use of Paul Ricoeur’s differentiations between cosmic, phenomenological and historical times, Bernard Stiegler’s theory concerning the relation between time, technology and memory and his concept of tertiary memory, and Mark B. N. Hansen's concept of ‘diachrone things’. The analyses, among other things, determine how moving letters (also in artefacts that are not poetry or art) can ‘outsource’ the communication in the sense that a statement, even though it has a specific sender, has never been formulated by a subject. This interest in the relation between medial forms and the determination of a subject is continued in the thesis’s sixth and final chapter titled ‘Cyber- identities and Economies of Communication - on the Role of Distributions of Agency in the Communication of the Work’. The chapter's analysis is, among other things, motivated and inspired by Bernard Stiegler’s criticism of contemporary communication technologies that the user is unable to understand, influence and develop. Through analysis of ‘_cross.ova.ing ][4rm.blog.2.log 07/08 XXtracts_.’ by mez, it is studied how agency is distributed in works where the medium or the technology appears to control the communication or where it is obvious that a sender has been ‘communicating’ with the technology before communicating with us. This analysis provides an opportunity to discuss issues related to the interpellation of communication technologies and further discuss possibilities for various Internet identities and their correlations with medial conditions. The thesis is a contribution to the research field of digital literature, but it is also a contribution to intermediality studies, using Elleström’s model of the modalities of media to describe modalities and their composition in addition to talking about arts (e.g., literature and visual arts) or ‘basic media’ (e.g., text and image) and their combinations. Furthermore, it is argued that intermedial and multimodal dimensions should be treated not only on a descriptive level when they are essential to the creation of meaning and therefore should be analysed. Hence, the thesis also contributes to the development of methods of aesthetic analysis by supplementing them with a medial sensibility. The mindset behind the broad mediality concept and the model of the modalities of media can contribute with analyses that avoid dichotomous differences between human and machine performances, between analogue and digital media, between ‘reality’ and ‘Internet’. At the same time, the broad mediality concept and the model of the modalities of media provide opportunities for an analytically accurate identification of these phenomena and their distinct differences. It is an approach that has far-reaching potential for further developments, e.g. in connection with studies of relations between communication and identity in different media.

Starting with Homer’’s Odyssey through R. Larsen’’s The Selected Works of T.S. Spivet literature describes journeys, wanderings and world-explorations in which spatial realms provide the basic dispositives for the series of narrated events. While theories of literature from G. E. Lessing’’s Laocoon to K. Hamburger’’s Logic of Literature had conceived of literature as a particular way of perceiving time, M. M. Bakhtin’’s theory of the ““chronotope”” made space into a central constituent affecting the perception of the models in literary scholarship.

In recent years, current electronic media have prompted wide-ranging considerations on the importance of space for socio-cultural processes (the ““spatial turn””). With the application of mobile media devices such as mobile phones, GPS and PDAs and the development of mixed reality environments in museums, galleries or research labs, new combinations of physical, virtual and symbolic spaces have been realized. Metaphorically, we might even say that literature, after having passed through the needle’’s eye of book culture, seems to be reverting back to the multimodal patterns of action and the forms of antiquity, of the Middle Ages, or of the Renaissance. This, however, is taking place on a completely changed media-technological level: texts, objects, bodies and spaces combine in a largely uncharted way; electronic media take ““body language”” to a new level since more and more often the whole body is involved in the media activity. Increasingly complex sensors (integrated into vehicles, clothes and environments) ““realize””——in other words: measure——the movements of the body, its mimics and gestures. This ““multimodal”” body itself then also exchanges information with the ““products”” of this kind of technology. Such medial couplings and framings enable the cooperation of non-symbolic activities, natural language activities and algorithmic processes of computer systems.

Of special interest for the analysis of literary developments today are environmental, exterior or urban projects, the so-called ““Locative Narratives”” using the previously-mentioned locative media such as GPS-tools, PDAs or others, aestheticizing each of them in a quite unexpected turn that inverts the traditional processes of literarization from the ““head”” back to the ““feet;”” they adapt literary patterns like travel-, adventure-, love-, or detective narratives, returning their imaginary movements into real ones again. Among these are projects like Jean-Pierre Balpe’’s Fictions d’’Issy, Stefan Schemat’’s Wasser, Gabe Sawhney’’s [murmur], works of the collaborative artists’’ group ““34 North, 118 West,”” narrative online journals like the Madison Avenue Journal (http://www.madisonavenuejournal.com), and also projects like Worldwatchers by Susanne Berkenheger and Gisela Müller (http://www.worldwatchers.de/) that study the growing intensification of social control via electronic systems of observation. This contribution attempts to outline an initial overview theoretically situating these projects.

(Source: Authors' abstract for ELO_AI).