semiotics

The paper would be a presentation, with video, on the subject of Splatter Semiotics, and The Semiotics of Splatter, which is concerned with 'messy' digital lit / digital literacy. It will discuss Trump's tweets, Russian hacking, blockchain, and disruptive technologies which possess 'spread' as, not only a form of digital literature, but also a new and dangerous cultural horizon, one that threatens the very foundations of democratic institutions. This work stems out of my thinking about 'gamespace / edgespace / blankspace' that I've presented at other conferences (including ELO); the three terms reference gaming and habitus as forms of political, social, and artistic thinking; I would include a summary of this work and indicate its relationship and dissolution in splatter. ('Gamespace' is defined as a rule-governed domain; the term applies to anything from a chessboard to a school community. 'Edgespace' is the borderlands of the gamespace; it's always problematic, and might exist within competing regimes. In the motion capture work I've done, edgespace references the boundaries of the architecture for capture, and what happens at the boundaries appears to 'break' the capture representation. 'Blankspace' then indicates how edgespace is 'filled in,' how the imaginary operates there. I use the terms in considerations of Arctic and Antarctic mappings, virtual worlds, and so forth. Finally, the semiotics of splatter considers splatter as world-breaking and fast-forward tendencies towards mobile boundary closures; this leads to splatter semiotics, where the terms form a field that remains always already ruptured. This is the semiosis of the overloaded or hacked network, the network of fake news and fake apps, the explosive and turbulent behavior of the mediasphere itself. I see this field as a form of politicized digital literature, where words lose meaning, become puncta (Barthes) or tokens, where language splays. The talk will be laptop-dependent; the concepts are easiest to grasp through text-imagery, which will include maps, tweets, word grids, virtual-world video, and so forth.

The 21st century is awash with ever more mixed and remixed images, writing, layout, sound, gesture, speech, and 3D objects. Multimodality looks beyond language and examines these multiple modes of communication and meaning making.

Multimodality: A Social Semiotic Approach to Contemporary Communication represents a long-awaited and much anticipated addition to the study of multimodality from the scholar who pioneered and continues to play a decisive role in shaping the field. Written in an accessible manner and illustrated with a wealth of photos and illustrations to clearly demonstrate the points made, Multimodality: A Social Semiotic Approach to Contemporary Communication deliberately sets out to locate communication in the everyday, covering topics and issues not usually discussed in books of this kind, from traffic signs to mobile phones.

In this book, Gunther Kress presents a contemporary, distinctive and widely applicable approach to communication. He provides the framework necessary for understanding the attempt to bring all modes of meaning-making together under one unified theoretical roof.

This exploration of an increasingly vital area of language and communication studies will be of interest to advanced undergraduate and postgraduate students in the fields of English language and applied linguistics, media and communication studies and education.

(Source: publisher catalog copy)

In his theory of gestural manipulations, Bouchardon (2014; see Ensslin 2014: 82-83) starts from the assumption that “all clicking is interpretive” (160). He proposes a refined view of how readers interact with digital media through a “repertoire of gestures” (161-162), the use of which depends on the physical device used to interact with a particular text. In his view, analyses of gestural manipulations happen at five distinct levels: (1) the gesteme, which is an individual move, such as a key stroke or a mouse click, linking haptic move and interactive interface item; (2) the acteme, which refers to a sequence of individual gestemes combining to form a larger unit of gestural meaning, such as drag-and-drop; (3) the semiotic unit of manipulation (“SUM”), which is the sum of identical or similar actemes and their semiotic function; (4) media coupling, which denotes specific functions and meanings of SUMs in their medial contexts; and (5) interactive discourse, which happens at the superordinate level of digital text in context, and relates to the meanings of gestural interactions against this larger backdrop. This paper aims to complement and build upon Bouchardon’s (2014) theory in that it offers an examination of the ways in which innovative, experimental gestural manipulations give rise to new vocabularies of haptic interaction. I will report on a small study performed with a group of students, in which they induced a list of verbal expressions for specific actemes and their semiotic transpositions from interacting with Cannizzaro & Gorman’s Pry (2015), Erik Loyer’s Strange Rain (2011), and Steve Jackson’s Sorcery (2013). Against the backdrop of Bouchardon’s platform-independent theory, I will propose a tentative and fluid “gestionary” for mobile app fictions that reflects both the complexities of “spatialized relations” (Drucker 2011: 10), as well as the conceptually widening scope of gestural manipulations that is at reader-players’ disposal, as well as the multiple iconic, symbolic, and indexical meanings reader-players can perform with them in the storyworld, via touchscreen interfaces. The paper will end on an inevitably critical note, pertaining to the limitations of black-box media for expanding gestural vocabularies into algorithmic modes and into truly collaborative maker spaces (Emerson 2014).

This paper traces the development of a new semiotic mode, kinetic typography. Kinetic typography began with the experiments of filmmakers like Len Lye and Norman McLaren. Later, film title designers like Saul Bass and Pablo Ferro drew on the shapes of letters with inventive metaphors – serifs, for instance could make letters walk, because they can stand for shoes as they are elongated horizontals on which something stands. In Saul Bass’ titles for Hitchcock's Psycho, the splitting of letters became a metaphor for the split mind of the film's main character. Such inventions eventually became part of a lexicon of clichés drawn on by designers across the world. Eventually, researchers and software designers began to formalize and systematize the language of kinetic typography, and the fruit of their work is now widely available, not only to specialists, but also to anyone who uses PowerPoint or Adobe AfterEffects, even though users may not always be aware of the lexico-grammatical rules which underlie the menus they choose from. And computers, being agnostic as to the kind of “objects” their operations operate on, apply these grammars to letters as well as other graphic forms, thus consolidating the multimodality of the language of movement. The second part of the paper discusses these formalizations, drawing on the kinetic design literature. Based on M.A.K. Halliday's transitivity theory, it sketches the outlines of a systemic grammar of movement that can make the meaning potential of kinetic typography explicit. The paper concludes with an analysis of art works created by David Byrne which use PowerPoint as a medium. Using PowerPoint's relatively simple movement grammar, Byrne has nevertheless succeeded in using movement creatively, giving us a glimpse of a future of creative writing which has kinetic typography at its very centre.

(Source: Authors' Abstract)

The talk reflects on the theoretical and practical aspects of collaboration in e-literature. Firstly a model of digitally enhanced collaboration that could encompass both its past and future instances is proposed. Matching several groups of categories (for example “production / negotiation / creation” against “material / story / discourse”) the model demonstrates that e-literature – even if we are really witnessing the end of it now – maintains its status of an important laboratory for any collaboration in digital environment.

Alongside acclaimed collaborative works (Forward Anywhere, The Unknown, A Million Penguins) several less known examples from Poland will be presented: Digital Green Eye (2012) and Bałwochwał (2013) – collaborative adaptations of Polish avant-garde classics – as well as Piksel Zdrój – a hypertext project by 8 authors published in 2015. The aim of the first part is to introduce both a universal analytical model and some rather unknown examples of e-literature to the international audience.

The second part, in which I draw from my own experience as an author and producer of several collaborative e-lit efforts, reflects on available tools. I will demonstrate that popular collaboration tools hardly match the complexity of teamwork fiction writing aimed at delivering not only a product, a perfect “text”, but also a cohesive world with events and characters that start “living” their own lives

As it turns out, even in the world of ubiquitous computing the ultimate, working models for collegial writing are to be found in the universal social activities that had long proved to be storytelling friendly. These archetypes of literary communication (for example the road trip, the campfire chant, the round table debate) might be as much important for setting up a good collaborative environment, as technological affordances of software and hardware. Lastly, I will try to shortly predict possible directions in digital collaborative writing.

(source: ELO 2015 Conference Catalog)

Pedro Barbosa recalls in this interview his memories of the first studies and works of electronic literature back in the 1970s when he was a student at the University of Porto. Starting from considerations about his collaborative works he makes a comparison between printed literature tradition and the age of new media focusing on the paradigmatic change of this very transitional period with live in and the differences of the creative work. Furthermore he makes an interesting statement on regard of the aesthetics of new media by comparing works of electronic literature with the oral tradition. In the end he mentions some of the milestones of electronic literature that he considers important.

While discussion of the relationship of image and word has been prominent in the discourses surrounding new media writing, the role of sound is rarely addressed in this context, even though words are sounds and sounds are a major component of multimedia. This paper explores possibilities for new theoretical frameworks in this area, drawing on musico-literary discourse and cross-cultural theory, and using ideas about semiotic and cultural exchange as a basis. It argues that words and music in new media writing create emergent structures and meanings that can facilate ideas to do with boundary crossing, transnationalism and cross-cultural exchange.

The paper will examine the different types of sound in new media writing from voicescapes to soundscapes to musical composition. Building on my previous work on affective intensities in new media writing (Smith 2007 ; Smith 2009), and the manipulation of the voice to create cultural effects such as sonic cross-dressing (Smith 1999), I will discuss the ways in which sound plays a distinctive role in new media writing. I will draw up a typology of different kinds of conjunctions between sound and words in this area (e.g. parallelism, co-ordination, semiotic exchange, algorithmic synaesthesia and heterogeneity). I will also, constructing the term musico-literary miscegenation, explore the cultural effects of these word-sound blends, and how they can interrogate ideas about gender or ethnic identity.

The paper will refer to word and sound relationships in classic electronic literature works such as John Cayley’s Translation, Young Hae Chang Heavy industries Operation Nukorea and MD Coverley’s Afterimage. It will also discuss the exploration of different types of synergies between word and sound by the Australian sound and multimedia group austraLYSIS — of which I am a member. In particular it will feature some of my own work with composer Roger Dean, and our recent collaborations with video artist Will Luers.

The paper will take the form of a talk and powerpoint presentation.

(Author's introduction)

This paper explores what I define as a “masqueraded complexity”, a term that refers to the way
children’s electronic literature disguises its multiple features to a formative reader (the child/young adult) in order to maintain/assert the whole range of semiotic and narratological creative approaches allowed in this new literary scenario. The paper paper also examines the lights and shadows of children's digital literature's inherent properties from an educational perspective. To support this exploration, I combine theoretical approaches to digital literature (Ryan, Murray, Hayles, Landow, etc.), the exploration of the digital literature landscape for youngsters and recent studies on children's literary education (Chambers, Colomer, Tauveron etc.). Some of my own research group ongoing case studies with real young digital readers will also be used to illustrate the outcomes.
Despite its obvious heterogeneity, electronic literature presents a series of common complexities
intrinsically connected to the constructive properties that define it, such as multimodal expression, non-trivial engagement, and narratological disruption. Children's literature could also be understood as a polymorphic set of texts; however, the formative reader (the child) becomes an anchor during the creative act that assembles this set of texts. The awareness of this reader’s existence and her role forces authors to find ways to express their art within certain required limits in the complexity. In combination with the particular contexts where this group of texts are received, the resulting poetics denote a closeness between the electronic literature's properties and those of children's literature (e.g. interactivity and immersion as ways to maximize readers emotional engagement and the seduction of reading). This provocative and stimulating scenario is increasing scholars' interest towards children's e-lit in a very hopeful way but the rawness in the formative and definitional process of this new corpus is still quite obvious, far removed from mainstream electronic literature's solid evolutive paths (e.g. avant-garde expressions of e-poetry, the different hyperfiction generations and so forth).
Nevertheless, children’s e-literature is already managing to find a way to deal with this required
dialogue between these new literary features and the fact that its users are still not fully developed as readers/spectators. This status quo urges to reflect on children and young adult's electronic literature’s future as well as become aware that a new approach to our literary education is necessary. This new approach should account for the electronic literature's formative potential for a contemporary reader. (Source: authors abstract)

Blueberries (2009) by Susan Gibb, published in the ELO (Electronic Literature Organization), invites the reader to travel inside the protagonist’s mind to discover real and imaginary experiences examining notions of gender, sex, body and identity of a traumatised woman. This article explores the verbal and visual modes in this digital short fiction following semiotic patterns as well as interpreting the psychological states that are expressed through poetical and technological components. A comparative study of the consequences of trauma in the protagonist will be developed including psychoanalytic theories by Sigmund Freud, Jacques Lacan and the feminist psychoanalysts: Melanie Klein and Bracha Ettinger. The reactions of the protagonist will be studied: loss of reality, hallucinations and Electra Complex, as well as the rise of defence mechanisms and her use of the artistic creativity as a healing therapy. The interactivity of the hypermedia, multiple paths and endings will be analyzed as a literary strategy that increases the reader’s capacity of empathizing with the speaker.

Analyzing Digital Fiction offers a collection of pioneering analyses based on replicable methodological frameworks. It offers analyses of digital works that have so far received little or no analytical attention and profiles replicable methodologies which can be used in the analyses of other digital fictions. Chapters include analyses of hypertext fiction, Flash fiction, Twitter fiction and videogames with approaches taken from narratology, stylistics, semiotics and ludology. Essays propose ways in which digital environments can expand, challenge and test the limits of literary theories which have, until recently, predominantly been based on models and analyses of print texts.

Chapters:

1.Introduction: From Theorizing to Analyzing Digital Fiction Alice Bell, Astrid Ensslin and Hans Kristian Rustad

Section 1: Narratological Approaches

2. Media-Specific Metalepsis in 10:01 Alice Bell

3.Digital Fiction and Worlds of Perspective David Ciccoricco

4. Seeing into the Worlds of Digital Fiction Daniel Punday

Section 2: Social Media and Ludological Approaches

5. Playing with rather than by the Rules: Metaludicity, Allusive Fallacy and Illusory Agency in The Path. Astrid Ensslin

6. 140 Characters in Search of a Story: Twitterfiction as an Emerging Narrative Form Bronwen Thomas

7. Amnesia, the Dark Descent: The Player's Very Own Purgatory Susana Tosca

8. Wreading Together: The Double Plot of Collaborative Digital Fiction Isabell Klaiber

Section 3: Semiotic-Rhetorical Approaches

9. (In-)Between Word, Image and Sound: Cultural Encounter in Flight Paths. Hans Kristian Rustad

10. Figures of Gestural Manipulation in Digital Fictions Serge Bouchardon

11. Hyperfiction as a Medium for Drifting Times: A Close Reading of the German Hyperfiction Zeit für die Bombe Alexandra Saemmer

Afterword Roberto Simanowski.