multimodality

During the last ten years the digital publishing industry has gone through an important development in terms of the creation and adoption of new software. E-books that are widely known and used in PDF format can now be entirely designed with new software and enriched with multimedia and multimodal features, which can also embed Augmented Reality and Virtual Reality technologies.

Since the design of the layout and the software of enhanced e-books are both created with the same technology used in the web, these enhanced e-books can be analyzed and described with theories and concepts used in the field of web technology. The innovative features that these e-books present include practices termed as Multimedia Literacy, i.e. multilinearity, multimedia, multimodality. For these reasons the analysis of the works of fiction and poetry published in this format can be carried out through these theories and research methods, already elaborated in the field of electronic literature.

The method of media-specific analysis conceived by Katherine N. Hayles, as well as her comparative study on the relationship between print books and works of electronic literature, the studies on software and new media aesthetics by Lev Manovich, and Johanna Drucker’s work that encompasses print books, artists’ books and digital aesthetics are among the most important resources.

Despite the popularity of e-books, the practice of multimedia literacy and the increasing interest in AR and VR technology in various industrial sectors as well as in the arts, the market growth and development of the technology for both the creation and the preservation of digital books has been remarkably low. Enriched e-books, defined by editors and experts as “enhanced e-books”, or referred to with the acronyms EPUB (electronic publishing) and DPUB (digital publishing), may also fail in popularity because of similar terminology used in online and web- based publishing.

In my presentation I will summarize a survey made in 2018 among six digital publishers from Italy, all founders of publishing companies or projects in digital publishing. The survey covers a time span from 1971, the year Project Gutenberg was founded to 2017 when the IDPF was combined with the World Wide Web Consortium.

In my interviews I asked questions concerning the creation, distribution and preservation of these types of books. Moreover, the interviewees gave their personal views on the acceptance and awareness of this technology among their customers in their respective fields. The topics addressed range from works of literature, art and school education.

The aim of PO.EX: A Digital Archive of Portuguese Experimental Literature (http://po-ex.net/) is to represent the intermedia and performative textuality of a large corpus of experimental works and practices in an electronic database, including some early instances of digital literature. This article describes the multimodal editing of experimental works in terms of a hypertext rationale, and then demonstrates the performative nature of the remediation, emulation, and recreation involved in digital transcoding and archiving. Preservation, classification, and networked distribution of artifacts are discussed as representational problems within the current algorithmic and database aesthetics in knowledge production.

(source: abstract DHQ)

This dissertation arises from an interest as a research fellow at the project “PO.EX'70-80 – Digital Archive of Portuguese Experimental Poetry” which has in its essence the problematic of digital archives for variable and multimodal media. Sharing is nowadays a word contested in many ways, and it is this same fact that underpins this thesis too. The distribution of information and the works of art in various digital files is our topic in scrutiny. With this sharing it is our intention to inform about distinct authors and art forms, making them present in a world where the multimodal and the cultural diversity seem to walk together, but also seem to be quite ephemeral in its productions. Assuming that the digital files are critical to the preservation and dissemination of artistic content, the aim of this dissertation will be to know a little more of this world of participative, multimedia and hypertextualized culture, and ending with an overview of the forms of archiving that digital makes possible today. This research also includes the analysis and qualitative assessment of five digital archives: PO.EX'70-80, ELMCIP (Electronic Literature as a Model of Creativity and Innovation in Practice), VirtualArt Database, ELD (Electronic Literature Directory) and NT2 (Nouvelles Tecnhologies Nouvelles Textualités). We investigate and propose a autonomous model of analysis and evaluation, aiming to understand the pros and cons of these archives and how they are made known to the public and their users.

(source: abstract in repository)

The 21st century is awash with ever more mixed and remixed images, writing, layout, sound, gesture, speech, and 3D objects. Multimodality looks beyond language and examines these multiple modes of communication and meaning making.

Multimodality: A Social Semiotic Approach to Contemporary Communication represents a long-awaited and much anticipated addition to the study of multimodality from the scholar who pioneered and continues to play a decisive role in shaping the field. Written in an accessible manner and illustrated with a wealth of photos and illustrations to clearly demonstrate the points made, Multimodality: A Social Semiotic Approach to Contemporary Communication deliberately sets out to locate communication in the everyday, covering topics and issues not usually discussed in books of this kind, from traffic signs to mobile phones.

In this book, Gunther Kress presents a contemporary, distinctive and widely applicable approach to communication. He provides the framework necessary for understanding the attempt to bring all modes of meaning-making together under one unified theoretical roof.

This exploration of an increasingly vital area of language and communication studies will be of interest to advanced undergraduate and postgraduate students in the fields of English language and applied linguistics, media and communication studies and education.

(Source: publisher catalog copy)

This presentation will link the trope of “digital ekphrasis” (as articulated by Cecilia Lindhé) and the developing of platforms for “augmented reality” to argue that one probable future for electronic literature lies in the interweaving of “born digital” and print texts in ubiquitous layers of mediation. It will examine three instances of “augmented” print – the multimodal performance of ekphrastic poetry, the AR comic book Modern Polaxis, and the AR epistolary romance Between Page and Screen – all of which demonstrate the power of “intermediation” (Hayles) and foster a critical perspective on it. Looking at these amalgamations of print and digital textuality through the lens of digital ekphrasis reveals that electronic literature will most likely always arouse ambivalence, just as the trope of ekphrasis in traditional media has, for better or worse, provoked a sense of the uncanny through its interweaving of visual, auditory, tactile, verbal, or haptic experiences.

I will first establish the aesthetic and rhetorical theories of ekphrasis that will frame my discussion of augmented reality and electronic literature. In writing of the centuries of ambivalence associated with intermedial “picture poems,” W.J.T. Mitchell has outlined the “dangerous promiscuity” of ekphrasis (155), how its “mutual interarticulation” (162) – with words helping to determine the significance of images and vice versa – threatens the stability assigned by audiences to each medium, at the same time that it provokes the “hope” that each medium’s limitations can be overcome. Lindhé has rehearsed and extended that discussion to highlight specifically “the interaction between visual, verbal, auditive and kinetic elements in digital literature and art” (Lindhé Par.13). She makes the case that the more comprehensive theory of ekphrasis in rhetoric allows us to understand and appreciate the intermedial functions of digital textuality in new ways: “digital literature and art align with this concept of ekphrasis, especially in the way that its rhetorical meaning is about effect, immediacy, aurality, and tactility. The multimodal patterns of performativity in the rhetorical situation stage a space-body-word-image-nexus with relevance for how we could interpret and discuss digital aesthetics.” Lindhé’s concept of digital ekphrasis has much to offer as we think about the power of electronic literature, but I will argue that the ambivalence the trope has always elicited is just as important to remember.

In the final section of the presentation, I will demonstrate how various AR texts court a sense of the uncanny and thereby serve as paradigmatic examples of the multi-layered future of electronic literature. After noting the precedent of Caitlin Fisher’s Andromeda (Electronic Literature Collection, V.2), I will examine the remediation of print poetry through multimodal AR performances and locative poetry (Berry and Goodwin). Next, I will offer a close analysis of Sutu’s AR comic book Modern Polaxis, which employs both the palimpsest effects of AR and the tropes of science fiction (time travel, body snatchers, the automaton) to encourage us to “learn to taste the tea on both sides” of an uncanny reality. Finally, I will end with a discussion of Borsuk and Bouse’s Between Page and Screen, to my mind the most ambitious use yet of AR for literary expression. Like Lev Manovich’s thoughts on “the poetics of augmented space,” Borsuk’s work (both the book and her essay on “words in space and on the page”) shows us, it is more fruitful to think of AR as a cultural and aesthetic practice than as a technology. The platforms for AR may change from smartphones to wearables, and beyond, but AR itself will persist in, among other things, an uncanny electronic literature not just “born digital,” to use Strickland’s phrase, but cached in the world around us.

(source: ELO 2015 conference catalog)

The digital turn brings about not only changes in young adult literature considered as aesthetic artifacts and literary works but also changes in the perception and reception of the reader. Digital young adult literature is increasingly multimodal and interactive, and it integrates elements from game aesthetics. When young adult literature navigates between media, new analytical approaches are required to explore the way in which it operates among various aesthetic strategies and medialities and the way it affects the young adult reader. With this development it becomes essential to combine different fields of research, e.g. research in literature and media science; thus, the focus of this paper will be research in children’s literature in an intermedial perspective. The analytical approach can be either diachronic when the object is the study of how various aesthetic expressions (text, picture, sound, etc.) have been used to create the literary artifact, or the approach can be synchronically based when the object is studying the categories which cut across the aesthetic expressions with the aim of transgressing conceivable media specific borders, and the latter will be the focal point here.

The pivotal point of this paper will be exploring how transgressing analytical categories, e.g. rhythm, sequentiality, time, space and dialogue with the reader, can shed light on the formation of meaning in a specific digital young adult literary work, i.e. Tavs (Camilla Hübbe, Rasmus Meisler and Stefan Pasborg 2013) which prompts different reading methods, paths, and types of interaction. The analysis will focus on selected analytical categories in order to explore the integration of various art forms and sensory appeals, viz. visual, auditory, and tactile modalities. In other words, the paper will investigate the ‘denaturalization’ of the reading process and it will attempt to investigate and offer analytical categories which can be used also by young readers so that they can become competent cross media readers of young adult literature in a digitalized and medialized landscape of texts.

Theoretically, the presentation will be based on theory on digital literature and media (Hayles, N. Kathrine Electronic Literature. New Horizons for the Literary. Indiana: University of Notre Dame Press 2008, Simanowski, Roberto, Jörgen Schäfer and Peter Gendolla (ed.) Reading Moving Letters. Digital Literature in Research and Teaching. Bielefeld, Trancript Verlag 2010, Bell, Alice, Astrid Ensslin and Hans Kristian Rustad Analyzing Digital Fiction. New York: Routledge 2014) and theory on picturebook (Nikolajeva, Maria and Carole Scott (2006) How picturebooks work. New York: Routledge).

(source: ELO 2015 conference catalog)

Electronic literature exists in a perpetual state of flux, due to its reliance on digital technology; with the rapid progression of processing power and graphical abilities, electronic literature swiftly moved from a reliance on the written word into a more diverse, multi-modal form of digital arts practice. The literariness of early electronic literature is manifest: the work was primarily textual, the centrality of reading paramount. The current crop of electronic literature--with its audio-visual, multimodal nature--calls into question the literariness of this work, however, as is evidenced by this year's call for papers. I propose that this ambiguity as regards literariness and written textuality in electronic literature disadvantages the field, in both academic circles and in the search for a wider reading audience. If electronic literature as field is to assert and validate its position within the greater literary tradition, links between electronic literature and past literary achievements need to be uncovered and illuminated. In this paper, I will explore the connections between works of postmodernism (in particular, experimental authors such as William Gaddis and Thomas Pynchon, and authors explicitly critiquing their own craft, such as Paul Auster), and electronic literature. In doing so, I hope to uncover a richer, more nuanced background for the literary in the traditions/practices of print and oral cultures, rather than in arts where the literary may or may not be present. Literature qua literature is currently present only as a minority element in works claiming the status of electronic literature, and it is therefore unlikely that literary studies will set itself up for reading just this sub-genre within an as yet minority arts practice. Turning to the works of Tom LeClair and Joseph Tabbi, postmodern novels are seen as coming into existence at a time when the world was on the brink of the globalized, world system. LeClair writes of the art of excess, in which the "recognition of radically new and massive information in the world and the impulse to represent this information are. . . the ultimate motives for the art of excess" (48) in postmodern novels. Tabbi, building off of LeClair, understands Gaddis and Pynchon as not only accurately representing a globalized world of excessive information, but also as representing the cognition developing out of this world system. My paper will explore how we can understand our own contemporary world system through works of electronic literature. If postmodern authors turned to their own material supports--the written language marked on a page through the course of a particular writer's thinking--then how are we to understand electronic literature as representing a certain mode of consciousness, especially considering electronic literature's distinct marginalization of the written word? My paper will offer close readings of earlier works of hypertext fiction and new works of electronic literature (published in 2012), in an effort to uncover the uniquely literary accomplishments of chosen texts/authors writing in a medium in which the literary is not always evidently manifest.

(Source: Author's Abstract)