mixed media

The 21st century is awash with ever more mixed and remixed images, writing, layout, sound, gesture, speech, and 3D objects. Multimodality looks beyond language and examines these multiple modes of communication and meaning making.

Multimodality: A Social Semiotic Approach to Contemporary Communication represents a long-awaited and much anticipated addition to the study of multimodality from the scholar who pioneered and continues to play a decisive role in shaping the field. Written in an accessible manner and illustrated with a wealth of photos and illustrations to clearly demonstrate the points made, Multimodality: A Social Semiotic Approach to Contemporary Communication deliberately sets out to locate communication in the everyday, covering topics and issues not usually discussed in books of this kind, from traffic signs to mobile phones.

In this book, Gunther Kress presents a contemporary, distinctive and widely applicable approach to communication. He provides the framework necessary for understanding the attempt to bring all modes of meaning-making together under one unified theoretical roof.

This exploration of an increasingly vital area of language and communication studies will be of interest to advanced undergraduate and postgraduate students in the fields of English language and applied linguistics, media and communication studies and education.

(Source: publisher catalog copy)

"Once you get into the flow of things, you're always haunted by the way that things could have turned out. This outcome, that conclusion. You get my drift. The uncertainty is what holds the story together, and that's what I'm going to talk about." -- Rhythm Science The conceptual artist Paul Miller, also known as Dj Spooky that Subliminal Kid, delivers a manifesto for rhythm science -- the creation of art from the flow of patterns in sound and culture, "the changing same." Taking the Dj's mix as template, he describes how the artist, navigating the innumerable ways to arrange the mix of cultural ideas and objects that bombard us, uses technology and art to create something new and expressive and endlessly variable. Technology provides the method and model; information on the web, like the elements of a mix, doesn't stay in one place. And technology is the medium, bridging the artist's consciousness and the outside world. Miller constructed his Dj Spooky persona ("spooky" from the eerie sounds of hip-hop, techno, ambient, and the other music that he plays) as a conceptual art project, but then came to see it as the opportunity for "coding a generative syntax for new languages of creativity." For example: "Start with the inspiration of George Herriman's Krazy Kat comic strip. Make a track invoking his absurd landscapes...What do tons and tons of air pressure moving in the atmosphere sound like? Make music that acts a metaphor for that kind of immersion or density." Or, for an online "remix" of two works by Marcel Duchamp: "I took a lot of his material written on music and flipped it into a DJ mix of his visual material -- with him rhyming!" Tracing the genealogy of rhythm science, Miller cites sources and influences as varied as Ralph Waldo Emerson ("all minds quote"), Grandmaster Flash, W. E. B Dubois, James Joyce, and Eminem. "The story unfolds while the fragments coalesce," he writes. Miller's textual provocations are designed for maximum visual and tactile seduction by the international studio COMA (Cornelia Blatter and Marcel Hermans). They sustain the book's motifs of recontextualizing and relayering, texts and images bleed through from page to page, creating what amount to 2.5 dimensional vectors. From its remarkable velvet flesh cover, to the die cut hole through the center of the book, which reveals the colored nub holding in place the included audio CD, Rhythm Science: Excerpts and Allegories from the Sub Rosa Archives, this pamphlet truly lives up to Editorial Director Peter Lunenfeld's claim that the Mediawork Pamphlets are "theoretical fetish objects...'zines for grown-ups." (Source: Google Books)

Mixed-media artists Joellyn Rock and Alison Aune offer a hands-on visual art workshop on collage, paper-cutting, silhouettes and digital compositing. What does this have to do with electronic literature you ask? Well... In Rock and Aune's multimedia installation, Fish Net Stockings, which will be exhibited at the Hybridity Exhibition at ELO 2015, the little mermaid story unfolds with multivalent versions echoing folk art patterns and digital iterations. Bifurcating imagery, like that made by folding and cutting, plays a role in the aesthetics of the work. Hans Christian Andersen was known for his live scissor writing. His version of scherenschnitte was an improvised performance art with paper cut imagery, integrating the haptic visual experience into his storytelling. Andersen’s cut paper collages anticipate the collage art of dadaism and surrealism, and some e-lit experiments can trace their roots back to these very methods of assemblage. Join us for a playful workshop generating mixed-media collages, paper cuts, silhouettes, and testing their use in digital compositing for video projection. Investigate how to combine imagery in layers in Photoshop and video in final cut for rich digital composites and short animations.

(source: ELO 2015 catalog)

In 2001 Henry Jenkins discussed the growing prevalence of ‘transmedia storytelling’. Transmedia storytelling is, simply put, franchises: a movie is followed by a game, then perhaps a comic, website and so on. An example is the Wachowski brother’s Matrix franchise. For Jenkins each media, each channel, communicates different aspects of a storyworld. Since 2003 Jane McGonigal, and others, have been researching the phenomenological and social aspects of ‘alternate reality gaming’. Alternate reality gaming requires players to traverse websites, games, public play, SMS and so on. Microsoft’s The Beast was the first of such ‘games’ that required participation with websites, posters, faxes, hacking, chatbots and Spielberg’s film AI (McGonigal, 2003).
Academic research into multi-channel storytelling is at present approached from the media studies and phenomenological perspective. As yet no poetics to address transmedia, alternate reality gaming, cross- or multi-platform and cross-media of content have been proposed in academia; in addition no poetics has been invented for multi- channel single-story creation (that is: one story told over multiple media).
This paper provides an overview of the poetics being developed for multi-channel storytelling. It is a narrative schema intended for instructional use in story creation and literary criticism.