The Patch: An artful syn(aes)thetic mapping of linguistic data through collaborative digital / analogue literacy processes

KEDRICK JAMES, RACHEL HORST, YUYA TAKEDA & ESTEBAN MORALES University of British Columbia

The Patch is an experimental workshop approach to creative / critical text analysis, a collaborative w/reading¹ (Torres, 2005) of linguistic data in which an input text undergoes multiple states and stages of transformation before reaching the output as multimedia audiovisual artifacts (see Figure 1). It emphasizes the role of teachers as creative pedagogues who collaborate with students in the realization of meanings that can be mapped simultaneously across three different ecologies of significance: mental, social and physical. In taking up Guattari’s notion of the three ecologies, we place a critical yet playful lens on the notion of creative economies, a concept that attempts to territorialize, indeed quantify, the explicit value of human creativity as a force and function within national economies (Schlesinger, 2016). As Guattari (2000) states,

by their very essence analytic cartographies extend beyond the existential Territories to which they are assigned. As in painting or literature, the concrete performance of these cartographies requires that they evolve and innovate, that they open up new futures, without their authors [auteurs] having prior recourse to assured theoretical principles or to the authority of a group, a school or an academy ... Work in progress! (p. 40)

The Patch’s textual procedures are recursive and recombinant; inputs and outputs can be re-patched to render alternative w/readings that privilege neither the technological nor the human, neither the digital nor the analogue, in collaborative cultural production.

FIGURE 1. The Patch schematic. Using the metaphor of an electrical circuit, we chart the stage and state transformations the input text undergoes throughout the workshop.

This immersion of being in the becoming textual field of creative analysis manifests a polysingularity — an experiential context — in the face of analytic fixity and dogmatic abstraction, forging instead creative interagential analytics that undermine regimes of knowledge control and capital flow in cultural (re)production. The output of the Patch remains indistinguishable from the context and circumstance of analysis, in line with Bateson’s remonstration that “it is important to see the particular utterance or action as part of the ecological subsystem called context and not as the product or effect of what remains of the context after the piece which we want to explain has been cut out from it” (Bateson, 1972, p. 338). This reneging of data analytic theories and practices disrupts discursive conventions by which analysis and criticism might be reduced to the parroting of tropes; as a praxis it repels the alienating transubstantiation of meaning as intellectual property from the material, personal and political contexts in which it occurs. Such a rethinking of creative work in the digital age — an age marked by its uptake of hypersignifying media, informational manipulation and profound social divisions of wealth and power — responds to divisive capitalization. We see the notion of creative pedagogues introduced in this special issue as stimulating the development of underlying values of collective enterprise and creative solidarities in the production of dissensual cartographies of meaning; we thus have sought to map collectively relevant topoi of active semiosis in a linguistic text and thereby account for imagination’s role in stimulating a rhizomatic growth of knowledge according to the three ecologies as outlined in Guattari’s anticapitalist critique.

Instantiated as a workshop, the Patch embodies economic value as a site of cultural production, teaching participants to work in creative and collaborative production teams that are integrated with contemporary digital technologies of reading, writing and media production, thereby promoting the participants’ future economic potential. Our work systematically challenges the central tenets of creative economies, which emphasize the capital side of cultural capital through its targeting “wealth and job creation through the generation and exploitation of intellectual property” (Department for Culture, Media, and Sport [DCMS], 1998, p. 3), and its attempt to instrumentalize and quantify what Yúdice (2003) calls the “transcendent” (p. 11) aspects of culture that have fallen into the domain of the “managerial professionals” (p. 12). By positioning analysis within the material circumstance of interpretation as creative play and performance with and through technologies, we discard authoritative summaries in favour of imaginative recreations that prioritize and map hard-to-measure outcomes and impacts on mental, social and material ecologies that vitalize culture. It is these same aspects of culture that the latest update to the seminal Creative Industries Mapping Document of the UK Department of Culture, Media, and Sport (DCMS) admitted, “For personal wellbeing, educational attainment, life chances and soft power, more work is needed to refine how we measure the specific impact that culture makes” (DCMS, 2016, p. 58). Originating in Britain, and now having spread through international policies, the economics of creativity have translated the material and social impact of cultural products and practices into actuarial registers (Schlesinger, 2016) but have missed personal investments such as those made by creative pedagogues who inspire and expand the creative consciousnesses and solidarities of their students.

We approach this work as “ecosophical” textual analysis using arts-based practices in concordance with Guattari’s (2000) postulation that:

in mapping out the cartographic reference points of the three ecologies, it is important to dispense with pseudoscientific paradigms. This is not simply due to the complexity of the entities under consideration but more fundamentally to the fact that the three ecologies are governed by a different logic to that of ordinary communication between speakers and listeners which has nothing to do with the intelligibility of discursive sets, or the indeterminate interlocking of fields of signification (p. 40).

As creative pedagogues working in the context of a centre for digital literacy, we are equally interested in the potential to teach and incorporate digital literacies into our arts-based practices, and as researchers our work is focused on the attempt to look at a symbiosis of the organic and the automated, the interagencies of the human and non-human at work today in the generation of meaning (Adema & Hall, 2016). In this way, the Patch proceduralizes multiple stages in which modal transformations occur between the analogue and the digital, the individual and the social, and the unit and the whole in contextual networks that leave textual traces of intention, pleasure and symptomatic autoreferentiality. At each of these key stages in modal transformation we exploit notions of glitch and error (Peña & James, 2016) as a means of revealing otherwise hidden processes in the technical automation and cognitive automaticity of language-meaning generation. Guattari (2000) states “the crucial objective is to grasp the a-signifying points of rupture — the rupture of denotation, connotation and signification — from which a certain number of semiotic chains are put to work in the service of an existential autoreferential effect” (p. 56). As a shorthand for the processes we undertake in analyzing linguistic data through the Patch, we treat the words digital and analogical as acronyms of two enactments of literacy (see Figure 2).

FIGURE 2. Digital and analogical literacies as procedural gestalts.

These terms are not polarized in opposition; instead, different surplus values of signs are given off by the filtering effect of these modalities. Both modes are generative of new meaning-potentials within critical / creative literacy praxis; therefore, we position both modes as significant in authoring meaning through eco-logical processes that are grounded in the autopoesis of living language.

Theories espoused by Deleuze and Guattari (1983; 1987) are rooted in a rethinking of the abstract machine of capitalism and its assemblages of utterance into territories of meaning that can be owned and thus co-opted for the extraction of capital. The Patch sets up a sequence of recombinant transformations that break with the forms of territorialized cultural production that seldom benefit artists, while removing intellectual goods from the creative commons (Lessig, 2004) and depleting shared human productions and experiences that inhere to collective subjectivities and are, ipso facto, our inheritance. Discourse operates as a flow, and the analytical desiring machine we call the Patch (see Figure 15) engages this flow through processes that interrupt, collapse and expand the text, cause it to breathe, shout, be lulled by melody and sink silently into code. Like desiring-machines (Deleuze & Guattari, 1983), the Patch disrupts all continuities and arranges them in chains of signification that reflect not a summary of origins and expectations, but “the surplus value of a code” (p. 39). Deleuze and Guattari (1983) continue:

It is an entire system of shunting along certain tracks, and of selections by lot, that bring about partially dependent, aleatory phenomena bearing close resemblance to a Markov chain. The recordings and transmissions that have come from the internal codes, from the outside world, from one region to another of the organism, all intersect, following the endlessly ramified paths of the great disjunctive synthesis. If this constitutes a system of writing, it is a writing inscribed on the very surface of the Real: a strangely polyvocal kind of writing, never a biunivocalized linearized one; a transcursive system of writing never a discursive one; a writing that constitutes the entire domain of ‘real inorganization’ of the passive syntheses, where we would search in vain for something that might be labelled the Signifier–writing that ceaselessly composes and decomposes the chains into signs that have nothing that impels them to become signifying (p. 39).

The Patch is this kind of writing — iteratively atomizing, tentatively reconstituting, embodying and recoding chains of significance in the synaesthetic surplus of polysingular potentiality and possible meanings.

A practice of deterritorialization and recombinant cultural recycling is central to these procedural techniques (James, 2017a), which together make up the version of the Patch workshop described below. The stages and states of recursive transformation through which a text flows (each Patch is a sequence of operations including close and distant reading, automatic writing and oral performance, algorithmic interventions and transmediations) create human and computational artifacts from which creative analyses and analytic creativities emerge energetically, as surplus values and excess data, fostering new assemblages of meaning that undercut regimes of predictive syntax and discourse. Figure 1 provides a full schematic of the Patch processes examined in this paper. To date, the Patch workshop has been conducted on ten occasions.² This paper presents data from three workshops for which the same textual input was used. Two workshops were presented with Bachelor of Education students (n = 75) in a course called Teaching Writing. The third workshop was an in-house session at the Digital Literacy Centre (DLC), in which the authors and a visiting scholar engaged as participants using the same textual input. To help guide the reader through this series of data-analytic writing processes, we provide a variety of figures and examples. Readers are encouraged to view the multimedia artifacts-as-outputs that are linked to online resources accompanying this journal.

FIGURE 3. Patch schematic, Stage I: Digital processes.

Input data

The first step of the Patch, Stage I state i, is to read and discuss the textual input. For these workshops, we chose a short story by the Argentinian author Jorge Luis Borges (1975), “On Exactitude in Science”:

In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was Useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other Relic of the Disciplines of Geography. (p. 131)

This story, originally written in Spanish and then translated / transformed into English, contains 145 words and describes Borges’ answer to a question that arose in Lewis Carroll’s Sylvie and Bruno: “What do you consider to be the largest map that would be really useful?” (Carroll, 1893, para. 40, emphasis in the original) to which another character replies, “[W]e use the country itself, as its own map, and I assure you it does nearly as well” (para. 42). The idea of the largest useful map is transformed and resingularized a century later by Borges in “On Exactitude in Science”. “There is no individual enunciation” (Deleuze & Guattari, 1987, p. 80), explains the crowd of authors that is Guattari and Deleuze, there are only “collective assemblages” (p. 80). With his short story, Borges joins Caroll and “all the voices present within a single voice” (Deleuze & Guattari, 1987, p. 80), or, as Thomas King (2003) quotes Silyx Okanagan author and educator Jeanette Armstrong,

Through my language I understand I am being spoken to, I’m not the one speaking. The words are coming from many tongues and mouths of Okanagan people and the land around them. I am a listener to the language’s stories, and when my words form I am merely retelling the same stories in different patterns. (p. 2)

In Borges, the cartographers, in their zeal for exactitude, create the largest of maps, a map as large as the Empire itself. The map lays over the land inch by inch and mile by mile, and due to this exquisite exactitude becomes utterly useless. Among the many reasons we chose this story as the source text / input data for our inquiry is the central philosophical problematization of accuracy or exactness in scale when mapping space, geography, text and meaning.

Mapping is a kind of colonization of geography; like language, it is a way of laying claim through naming, delineating and defining boundaries. If Borges’ story is a map, like the people of the Empire we deliver it up to the “Inclemencies of the Sun and Winter” and create of the “tattered ruins” new meanings and geographies (Borges, 1975, p. 131). In the Patch, we invite the creative pedagogue / cartographer to re-engage with the geography of a text and creatively remap, remix and reinvent new complexities. In this way, we problematize the notion of definitiveness and finality in text; Borges’ story is one moment, one terminal, for ideas that radiate throughout time, minds and assemblages of subjectivities. Throughout the Patch process, we explore the idea from information theory that “an infusion of noise into a system can cause it to reorganize at a higher level of complexity” (Hayles, 1999, p. 70). Each state along the circuit of this process systematically introduces noise and excess into the system / geography of the text, purposefully inviting innovative and generative complexities to occur.

Text network analysis

After Borges’ text is shared with the participants, the metaphor of the map is literally configured in InfraNodus, a textual network analysis tool. InfraNodus transforms the text into a visual network, plotting word / nodes and their co-occurrences in lexico-syntactic relationships. The resulting graph is a map, a two-dimensional representation of the text that breaks the linearity of its narrative, allowing the visualization of its polysingularity or its “multiplicity of possibilities … offering a more holistic approach” to the text (Paranyushkin, 2012, p. 3). InfraNodus visualizes Borges' story as pathways of meaning and identifies structural gaps between nodes inside the graph where ideas and meanings can be explored by creating new connections or betweennesses (Paranyushkin, 2011).

When the textual network is shared with participants, a new form of reading is encouraged. The units-to-be-read are transformed and the focus is now on the words mapped as a system, which allows what literary historian Alfredo Moretti (2000) called distant reading, where the intention is to unveil meaning without getting involved in linear reading. This is in opposition to close reading, where the goal is to pay attention to the linearity of the story by attempting to capture all the features of the text (Baldick, 2015). Distant reading seeks to capture the nature of the story by aggregating data and then analyzing it (Schulz, 2011), creating a map of the linguistic data to be apprehended by participants.

Other features of InfraNodus are focused not on the visualization of new pathways of meaning but on tools to better read the text network graph. For example, the network structure is analyzed and categorized from dispersed (where ideas are completely spread out) to diversified (where several topics are central), focused (where communities of meaning are centred around a few topics) and biased (where ideas are centred around one single topic). InfraNodus also identifies words that are most influential (those that serve as anchoring terms for connecting other words) and the main topical groups (communities of words) in the input text. Borges’ story, for example, is categorized as focused with the most influential words being map, empire, there and tattered and shows several communities of meaning in the network, identified by different colours (see Figure 4).

FIGURE 4. “On Exactitude in Science” as a textual network in InfraNodus.

The decomposition of the text into its constituent elements leads not only to a visualization of the specific text’s polysingularity, but it also configures the network with relevant information to begin assembling new structures of meaning beyond the boundaries of the text under consideration. In the Patch, “the relevance of such models to [mapping] mental ecology … [are]: (1) their capacity to recognize discursive chains at the point when they break with meaning; (2) the use they make of concepts that allow for a theoretical and practical auto-constructability” (Guattari, 2000, p. 56).

Algorithmic augmentation

The third step in the Patch is to take the initial 145-word text and begin to complicate and elaborate upon the embedded meaning by further algorithmic intervention via the linguistic algorithms of Andrew Klobucar and David Ayre’s GTR Language Workbench. Language workbench is a project that “explore[s] how creative writing (and language use in general) might take advantage of digital processing applications to create new and innovative forms of literary art, electronic or otherwise” (Klobucar & Ayre, n.d.). Figure 5 illustrates one of several processors we used to elaborate upon the kernel story. This one is designed to find and replace nouns, adjectives and verbs in the original text with antonyms, hyponyms and synonyms.

FIGURE 5. Algorithmic text transformation.

The initial 145 words become augmented with iterations that extend the map outwards to include families of words and related meanings. For example, the first sentence of the short story:

In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province.

Becomes the following:

In that conglomerate, the art of fashioning attained such imperfections that the mapping of a single Kansu occupied the wholes shebang of a written communication, and the single-valued function of the imperium, the whole works of a state.

We ran the story through a series of these processes, adding each permutation to the initial text. In this way we augmented our initial data set of 145 words, to 1,088 words.

Unmapping the text

The next algorithmic intervention is a word scrambling tool called Word Shuffler (Online Random Tools, n.d.). By removing the syntactic mapping of the words, we invite participant / poet / cartographers to engage with the unformed geography at the level of the words themselves. Our minds naturally find meaning and coherence in worlds and words as we encounter them, and once this coherence is encountered, it becomes difficult to unsee it to find new patterns and geographies. With this disruptive process, we are initiating a practice of creative reading in which we intentionally unmap then remap words through ecological praxes, and

seek something that runs counter to the ‘normal’ order of things, a counter-repetition, an intensive given which invokes other intensities to form new existential configurations. These dissident vectors have become relatively detached from their denotative and significative functions and operate as decorporealized existential materials. (Guattari, 2000, p. 45)

FIGURE 6. The Patch schematic, Stage II: Engaging the analogue.

In this stage, the words become deterritorialized both from the kernel story / singularization as well as the basic principle of authorship / ownership of that story. The particle-signs of individual words become available for reabsorption into new discursive ecologies and geographies of meaning.

i read slow

The randomized words are now projected upon a screen using a program called iReadFast (Del Maschio, n.d.). This program allows the reader to control the rate of words as they appear sequentially upon the screen. Intended to help readers increase their reading speed, we use this program instead to facilitate a creative apprehension of text. Rather than attempt to grasp and comprehend each word, we invite increased noise into the reading process by displaying the words at improbable rates of up to 1,200 words per minute. Unlike a computer, which can process linguistic data at fantastical speeds, the human mind works more slowly and imaginatively. We intentionally invite this human slowness, along with emotion, desire, memory and intuition, into the creative abduction of reading / encountering words. This is the first state transformation in the Patch, as we cross the digital / analogue divide (see Figure 1). In so doing, we invite a generative opportunity for glitch and error in our human processing.

The instructions are to allow the flickering words on the screen to wash over the eyes. Rather than seek meaning among the words or attempt to write (map) each word by word, inch by inch — an impossible task for human beings at rates this fast — we encourage participants to follow their poetic instinct when selecting words. We record only those words that appear louder and more stimulating to our eyes than the others. When watching words flash by in this way, it becomes apparent that some words emerge in our minds with a kind of resonating clarity, while others simply pass by unheard. (This sonic analogy will be taken up in the transmediation stage of the Patch in which we transform text into MIDI³ code that is then sonified as music). This idea of the generative togetherness / collaboration between the digital and the human is explored and amplified throughout the Patch: Our slowness and limitation in processing speed becomes the root of our creative agency in the act of reading, filtering and mapping linguistic data.

Upon the large white paper / space spread across the tables before us, the participants gather around with black felt pens in hand, encountering the projected words, scrambled and deterritorialized and flickering at impossible speeds. Our arms cross over each other, reaching into that white space, drawing the words by hand. The handwritten words sometimes misspelled, sometimes illegible, are singularized gestures of meaning. Here the words flashing from the digital space are plucked, transformed and resingularized into flesh and gesture (see Figure 5). Here, “[w]e are dealing ... with the very important occidental difference between the visual and the haptical (a seeing by touching, as when we say that blind [people] see with their hands) … We are also dealing here with a narrative of touch, a narrative of membranes as revived” (Wetzel, 2006, p. 54). This materialization from digital image into tangible cursive is a kind of “scarring, stitching, or the ‘suture’ mending the gaping hole between the original and the image” (Wetzel, 2006, p. 55).

FIGURE 7. Analogue filtering of digital textual data.

We draw the words, populating the space with a new geography as of yet unmapped. To use the language of network analysis, each word is a node without edges — there is no betweenness among them. Next, we physically embody InfraNodus’s digital text network generation, but here poetry and improvised meaning will be our guiding principles for generating analogue lexical networks. We have deterritorialized the text, losing

not only … the pagination that makes reading possible … [but] also fill[ing] it with information derived from elsewhere, and, in doing so, [we] assemble that information into a new structure. Rather than reduce the sophistication of the text, a semantic model embeds that text in a web of others (Gavin, 2019, p. 19).

Here, Gavin is referring to digital network analysis in which each word is in relation to the entire digital corpus of the language. In physical networking, the de/re-composition of text invites the participants’ human algorithmic processing, which incorporates not the entire corpus of the language but the entire personal corpus of each participant. The words resonate not with their limited meanings embedded in the specific Borges’ map / story, but within the participants’ own life maps, thus combining physical, mental and social geographies.

In Stage II state ii, we begin to create poetic networks of meaning by encircling nodes with multicoloured pens and connecting them into small assemblages of poetic utterance or expression (see Figure 6). The empty space is haptically mapped, and networks emerge, overlap, and commingle in a “logic of intensities, or eco-logic, [which] is concerned only with the movement and intensity of evolutive processes” (Guattari, 2000, p. 44). We do not yet seek a coherent narrative between these poetic utterances. This “[p]rocess, … oppose[d] here to system or to structure, strives to capture existence in the very act of its constitution, definition and deterritorialization” (Guattari, 2000, p. 44).

FIGURE 8. Analogue textual network.

Assemblage of utterance and collaborative remix

Once we have delineated our analogue network among the isolated nodes, we begin narrowing the meaning through poetic association, creating betweenness and relationships among the nodes. Each participant chooses at least five of these networks to begin altering, augmenting and assembling into a discrete poetic utterance. Now we begin to divine and invent contextual relationships between the words, allowing for new words and concepts to enter into the poetic geography. Each participant works on their own collection of poetic utterances without seeking cohesion between them. Once the lines have been drafted, we then cut them up and collaboratively reassemble them into a poem (see Figure 9). As Swiss and Burgess (2012) suggest, “[c]ollaboration allows writers and artists … to reconsider both our work and our identities, to literally see them anew, as we move from individual to composite subjectivity” (p. 75). We continue here to problematize the concept of authorship / ownership / identity by exemplifying how the lineage of each poetic utterance has been de/re/territorialized numerous times within the Patch, breathing through subjectivities, both digital and analogue, and momentarily cohering within a poem that is neither final nor expository but merely (and abundantly) a singularity within this process of constant transformation.

In the case of new media poetry, the work sometimes remediates procedural writing, gestural abstraction, and conceptual art, while contributing to an emergent poetics. New forms of digital poetry, especially collaborative digital poetry, challenge already contested terms such as ‘poetry’ and ‘literature,’ and further complicated boundaries between literary genres. (Swiss & Burgess, 2012, p. 74).

FIGURE 9. Collaborative, recombinant, cut-up poetry.

FIGURE 10. The Patch schematic, Stage IIIa: Collaborative performance.

Collaborative oral performance

We have now mapped the social ecology of our collective aesthetic resonation with the words, infused them with personal poetic sensibilities, and collectively regenerated the text by recombining the composite linguistic elements within the deterritiorialized data. At this point we call upon participants to collaboratively perform their cut-up poems to enunciate and produce the language as physical stimulus. This envelops us proprioceptively. The text becomes environmental and a function of hollows in bones, muscle mass and spaces in architectural structures, reverberating off flat surfaces and thereby mapping words temporally in space. This emphasis on oral language convinced Guattari (2000) “that the question of subjective enunciation will pose itself ever more forcefully as machines producing signs, images, syntax and artificial intelligence continue to develop. Here we are talking about a reconstruction of social and individual practices” (p. 41). Thus far, we have allowed the visual to dominate our mapping of textual process, and vocal performance introduces the first transmutation that departs from alphabetic or binary code and embodies the materialities of the body, the voice and the environment in which it is situated. In this mode, variations in pronunciation and accent, mis/readings, as well as the composite sound of multiple voices — sometimes in unison or taking turns — puts both organic and artificial listeners in a creative role. While the performances occur, the speech-to-text processor packaged with InfraNodus attempts to assimilate a logical text from the performed poems using probabilistic algorithms trained on everyday speech. In other words, the syntactic networks which InfraNodus produces while recodifying these live, collaborative performances are very loosely structured. The software describes them as evolving / developing / diffuse, networks of meaning opened up by logical disjunctures (and inherent ambiguities) associated with poetic expression wherein listeners or readers are expected to contribute more fully to the transaction of meaning. These poetic gaps in predictive speech patterns help us map fertile sites (topoi) of semiosis where fresh meanings can emerge through disruption of functional, efferent transactions of meaning (Rosenblatt, 1994). As a result, the subsumption of the voicing of the poems into code is an exquisite partnership of the human voice and automated interpreter, wherein the algorithmic listener creatively approximates the actual text being performed. The voice recognition software produces a new literal text that is simultaneously visualized, the mapping of data that has undergone individual, social and environmental transformations. In this state, the data of recorded voices is harvested by our DLC team as acoustic and literal information to be incorporated in the culminating artifact / video output.

FIGURE 11. The Patch schematic, Stage VIa: Voice to text.

The generative digital glitch

As the participants read their co-authored poems out loud, their voices become text through the speech-recognition algorithm, and the graphemic representation of these evolving new textual networks is projected upon a screen as they speak. The audience receives multiple synesthetic enactments of the recycled poem: 1) a live polyvocal performance of the poem, 2) an input linear text that is being shown as the reading progresses, and 3) a developing visual network of words and terms that is projected on a large screen. The poem that was born of cut-up pieces of paper is resingularized as a digital poem, embedded with openness, interactivity, hypermediality and analogue-digital collaboration (Torres, 2005), where “the incorporeal transformation is recognizable by its instantaneousness, its immediacy, by the simultaneity of the statement expressing the transformation and the effect the transformation produces” (Deleuze & Guattari, 1987, p. 81). The poems’ multiple layers of representation do not act in opposition to each other but transform themselves as a whole among the collaboration between participants, the enactment of the poem and the technology that is used to expand its pathways of meaning.

FIGURE 12. Mapping one poem through a series of transformations.

One valuable outcome of turning our voices into text for InfraNodus to map is the glitch⁴ — the unforeseen behaviour of the computer when it interprets the voice input (Peña & James, 2016) — caused by technical limitations in the software’s capability of processing atypical vocal sounds. These glitches promote creative and pedagogical engagement with the text by allowing participants to explore the inter-relationships between the sound of their voices and the resulting visualization — an instantaneous transmediation of the poem (Peña & James, 2020). Figure 12 maps one poem through this morphological process.

This glitching, in the form of missing words and misrecognized terms, affords a learning and a creative opportunity in which the software’s agency is visible in the assembly of the textual network; this inquiry promotes an interrogation of the “relationship between served and server, between used and user, dialogically and creatively” (Peña & James, 2016, p. 123).

FIGURE 13. The author’s poetic performances as text networks in InfraNodus.

The resulting graphical text networks allow participants to compare the original story (in Figure 4) and their poetic performances (in Figure 13) by visualizing the multiple potentialities of text, the surplus of signs transfigured through the Patch. The variations among the textual networks, more than a mere quantification of Borges’ story and the poetic performances of the participants, show the creative and pedagogical value of the Patch. The deterritorialization of the text, filtered and recombined through several states and stages of transformation and transmediation, now makes visible entire geographies of interagentic expressions of meaning. Networks of meaning are generative and situated rather than conclusive and transcendent of the content, where no two poems are the same, and the chaos of the Real is apparent. Even if the lexicon is derived from the same fiction, no single interpretation is extracted to form dominant trajectories of meaning.

FIGURE 14. The Patch schematic, Stage IIIb states i and ii: Typed poems and MIDI transmediation.

Redigitization

The stage change from the physicalized poetry — handwritten, cut-up and taped back together into its typed, digitized and infinitely reproducible file format — is a significant and meaningful transformation. The digital poem becomes visually distinct from its physicalized iteration (see Figure 12). Not only do participants continue to make edits in the typing process, but there is a homogenization of form that takes place as well. As Hayles writes,“[d]ifferent technologies of text production suggest different models of signification; changes in signification are linked with shifts in consumption; shifting patterns of consumption initiate new experiences of embodiment; and embodied experience interacts with codes of representation to generate new kinds of textual worlds” (Hayles, 1993, p. 69). In Figure 15 we visualize the transformations one word / node makes, through three different workshops, as it moves from the source input text through the processes and subjectivities of the Patch; it visualizes the relative frequency of each algorithmic variation of map / mapping, as it is taken up or discarded along an ideational transfiguration through this semiotic desiring machine. The radiating lines of poetry are poly-resingularizations of the initial “map” but do not stop there in meaning; rather, they continue on in a journey of transformation.

FIGURE 15. A visualization of the Patch as desiring machine, following the word map through its flow of transformations in three different Patch events.

What happens when the map of the empire is deterritorialized, when the paperweight snow globe of a story is smashed in the collider of collective imaginations? This not only reveals the adaptability of logics involved in apprehending the linguistic data, but it also maps an underlying relational network between human and automated agents processing the data according to the related ecologies of mind, society, and the physical environment. As demonstrated in Figure 15 which details the transformations of a single seme, map — both noun and verb, an indexical supersign coding the infinitely complex singularities of place but also the act of exploration that produced it — evolves as it occupies different ecological niches according to the aesthetic needs and desires that nourish it. This is no mere roll call of cognates buzzing around a Signifier, but instead the sign’s evolving situatedness within the minds of the participants and their collective imaginations that draw incessantly on experience of physical and virtual worlds. Map becomes a central figure in a relational network that draws together texts, readers, writers, listeners, speakers, and viewers, shunting phrases together and setting up emotional topographies:

Try to map this vast emptiness —

its effects a tattered mathematical function,

a single-valued aesthetic of the Metropolis.

The chains of signification, in which map takes up residence, reinvent the Signifier each time while holding traces that leave a filigreed residue of context and intention. It is impossible, in fact, to isolate these ecologies of meaning from each other, to say that the map in Borges’ narrative has a subjective meaning apart from its social and political meaning, or what it means to the physical space and relational networks in which it was conjured. It is metaphor only insofar as it initiates a cascade or avalanche of emergencies; in its mutations it escapes closure, and is unredeemable for legitimate tender.

Data sonification

The next step in the Patch is a gesture towards a world of inquiry that we cannot possibly explore very deeply within the confines of either this paper or the Patch workshop itself; instead, this step is an invitation to further explorations and experimentations in the sonification of linguistic data. The transmediatory process of sonification is being used across the disciplines, from astronomy to medicine to geography, in order to provide new pathways for recognizing pattern and meaning within data.

Data sonification offers the chance of analyzing and communicating processes in very different ways. Firstly, sonfication artefacts are events … [they] retain temporal and performative dynamics within themselves as they play in time. Secondly, human affective processing works very differently between the visual and the aural (Palmer & Jones, 2014, p. 222-223).

In the Patch, we take up sonification as a way to encounter poetic data whose semiotic properties have been transformed into frequencies of vibration.

For this transmediation, we developed a text-to-midi program based upon jFugue, software that translates textual features into musical sequencing commands. This program, now called Singling⁵, allows us to make specific decisions about how we will hear the data. Letters, spaces and other textual features correspond to such melodic information as tempo, note and octave range. The program then both translates the text file into MIDI format (a file that can be edited in a variety of different software programs) and also plays / performs the MIDI file with user-selected digital instruments. Participants then listen to their poetic data performed musically by the computer (James, 2017b). Here temporality becomes a significant feature, given that “current, dominant visualizations or processes struggle to appreciate this as we freeze the world, cut it up into objects for study, risking that the essence of life is lost” (Palmer & Jones, 2014, p. 223). Unlike InfraNodus, in which we encounter the data all at once with its linearity removed, polysingularity in sound would be one instantaneous blast in the speakers. Instead, we can begin to layer sonified data upon itself, listening to a polylinearity of meaning, in which each layer of data becomes a voicing within the orchestration.

FIGURE 16. The Patch schematic, Stage V: Video, the synthesized output.

Throughout the Patch workshop, each process of text production, collaboration, performance and amplification are documented with video recording. The video and audio files are edited on-site during the workshop by one of the facilitators. The images are delineated in chronological order and sped up for the entire video to fit under two to three minutes. Fragments of audio files of poetry performances are selected and inserted into the video, and the sonified MIDI sounds are also incorporated in their entirety. This synthetic video, which blends diverse and divergent texts produced by the participants in the workshop, is presented at the end as the cumulative output of the Patch. Here, we wish to highlight two interrelated aspects of each video: (1) as a polysemiotic ensemble, and (2) as an expression of creative solidarity (Gaztambide-Fernández, 2009).

OUTPUT: Video as a Polysemiotic Ensemble

Throughout the Patch, Borges’ short story is processed through multiple digital tools and the participants’ creative interactions. Texts diverge through these heterogenerative processes, and their meanings are deterritorialized. The written and remixed texts, vocal performances of poetry, glitched voice-to-text visual network, text-to-MIDI sonification, and images of participants’ movements produce what Guattari (2000) calls dissensus. While such a state of rupture is something to be celebrated, as pedagogues, we also look for a terminal where differences meet in order “to resingularize serialized ensembles” (Guattari, 2000, p. 46). In the Patch, the video is the terminal for this resingularization.

Because of the nature of video, texts of different modalities are mapped onto a linear timeline (Wildfeuer, 2014). However, in order to produce a short video, temporalities are differentially manipulated for the images and sounds. The images of the participants’ movements in writing activities and InfraNodus network production are sped up by 800%-1,000%, while the speed of audio files remain unchanged. Also, vocal performances of poetry are fragmented and re-mixed to fit the length of the video. As discussed above, the speed of the sonification of the MIDI file is controlled. However, speed of the recorded sound is not manipulated at this stage. Hence, despite the linearity, three expressions of temporalities are operating in the video.

In addition to the movements in the video being sped up, the production is also done in haste. As the editing of the images and audio happens in parallel to the recording of them, editorial decisions such as the selection of the fragments of poetry performance depend largely on intuition rather than deliberation. This resembles the iReadFast writing activity in the Patch and destabilizes the exactitude of the mapping. Compressed and converged, the video / output lets the participants experience three layers of texts (images, oral performance, and sonification) in ensemble, catalyzing reflection and stimulating inspiration (an example Patch ouput video is available at the following link: https://youtu.be/EO202QiYEYA).

Video as an Expression of Creative Solidarity

In addition to the poetic texts, the video reminds us that participants generate another important kind of text during the Patch: their movements. The movements of writing-hands express a form of solidarity that consists not of uniformity and solidity, but of difference and fluidity. Here, we borrow Gaztambide-Fernández’s (2009) idea of “creative solidarity” in a pluralized form. While it is clear that Gaztambide-Fernández (2009) used the word “solidarity” with an awareness that it takes multiple forms depending on how people gather together, the idea of solidarity is indefatigably associated with unions — therefore, oneness. In its plural form, creative solidarities, we wish to foreground the multiple, transitory potentialities of the collective. According to Gaztambide-Fernández (2009), creative solidarities are spaces “where discourse is unstable and language is highly polysemic, where meanings are negotiated, and discursive practice is contested” (p. 89). Inside the creative solidarities, there is heterogeneity as opposed to conformity.

While embracing transitivity as a form of solidarity, creative solidarities do not deny the importance of discourses that organize meaning. Guattari (2000) was certainly aware of the danger of radical destabilization of meaning: “There is the possibility of violent deterritorialization which would destroy the assemblage of subjectification” (p. 45). Without others, we cannot even exist as subjects. To be in solidarity, Gaztambide-Fernández (2009) says, “means to be in relationships that recognize interdependence and the realization that our lives and our work cannot carry on without others” (p. 90).

As we argued above, the Patch catalyzes heterogenesis of texts. It highlights the polysemy of texts and provokes them into dissensus. However, it also demonstrates a trace of solidarities in the form of audiovisual artifacts. The video not only synthesizes the different modalities of texts into a polyphony (or a cacophony, depending on one’s aesthetic taste), but also synthesizes creative solidarities. Instead of the image of fists going up in the sky, hands of people are scribbling words, meshing networks, and collectively composing poetry. Such interweaving of words is also expressed in re-mixed fragments of voices together with the sonified MIDI sounds in ensemble. The video, therefore, is a polysemic signifier that denotes dissensus and connotes solidarities. This output becomes a new input for the participants and lets the diverged chain of signification continue to evolve.

FIGURE 17. The Patch schematic, Stage V, state i: reflection and praxis.

Conclusion

As artists, scholars, teachers and researchers, we see creativity as arising from the intentional enfolding of chance, the autopoiesis of the text as a living (eco)system capable of responding to interagential, polydimensional, and polysemiotic feedback collapsing into any given instance of singularization. We treat this notion of unpredictability and discovery through the process of synthesizing information as central to our aesthetic and academic activities as creative pedagogues and arts-based researchers. Rather than seeking predictable outcomes, we believe that the Patch can be used to channel these chaotic variables of instantiation and present complex data-analytic gestalts in entertaining and imaginative ways. The synergetic, audiovisual result of the Patch suggests intrinsic value functioning within many strata and scales of relational networks. As creators, we are witness to the results of our creations, and if we continue in our practices, we will recognize patterns in the outcomes over time. We are ourselves deterritorialized in this country of every twenty minutes, in this empire of an hour. We are thus positioned as learners, mapping out the probable outcomes of a given analytic process, monitoring the desiring-machine that we make manifest: We could very well be describing what it is to be a teacher. As creative pedagogues, we observe the results of our magical interventions in normative dynamics and witness the perturbations and effects of chance operations that result in patterned outcomes. As Richardson (2000) states, “any dinosaurian beliefs that ‘creative’ and ‘analytical’ are contradictory and incompatible modes are standing in the path of a meteor, they are doomed for extinction” (p. 962). In the case of the Patch, the textual gestalt captures not only the surplus signification of the input text but also its modulations by the variables of the material circumstance during which the data was processed, analyzed and utilized aesthetically, collectively, to produce poems, soundscapes and videos.

Notes

1.       W/reading combines writing and reading as a way to recognize the creative role of the reader in digital contexts where interaction and choice are fundamental to navigating through a text. For more information, see Torres, R. (2005). Digital poetry and collaborative wreadings of literary texts. Telepoesis. http://www.telepoesis.net/papers/dpoetry.pdf

2.       For examples of the Patch workshop processes and products, see https://dlsn.lled.educ.ubc.ca/wordpress/the_patch/

3.       Musical Instrument Digital Interface

4.       Glitch, as Jim Vespe writes in the October 2019 issue of Air & Space Magazine, https://www.airspacemag.com/airspacemag/just-right-word-180973113/ has Yiddish origins meaning a slippery place, to slide or glide, which connects to the notion of slippages in the normative flows of discourse discussed here. This term was taken up by engineers to mean a spike in voltage or an error in systems. Although glitch has negative connotations for engineering, we understand glitch as a positive slippage from normative operations, a place where human and machinic creativity are collateral and where hidden, underlying processes are made apparent through disruption of normative or expected outcomes.

5.       To explore the affordances of Singling for text sonification, please see https://dlsn.lled.educ.ubc.ca/wordpress/singling/

References

Adema, J., & Hall, G. (2016). Posthumanities: The dark side of 'The Dark Side of the Digital'. Journal of Electronic Publishing, 19(2), n.p. https://doi.org/10.3998/3336451.0019.201

Baldick, C. (2015). Close reading. In The Oxford Dictionary of Literary Terms (4th ed.). Oxford University Press.

Bateson, G. (1972). Steps to an ecology of mind. Ballantine Books.

Borges, J. L. (1975). On exactitude in science. In J. L. Borges (Ed.), A universal history of infamy (N. T. di Giovanni, Trans.) (p. 131). Penguin Books Ltd.

Carroll, L. (1893). Sylvie and Bruno concluded. Lit2Go. https://etc.usf.edu/lit2go/211/sylvie-and-bruno-concluded/4652/chapter-11-the-man-in-the-moon/

Del Maschio, G. (n.d.). iReadFast. Gengisweb. http://www.gengis.net/prodotti/iReadFast_Mac/en/index.php

Department for Culture, Media, and Sport [DCMS]. (1998). Creative industries mapping document. London: Department for Culture Media and Sport. https://www.gov.uk/government/publications/creative-industries-mapping-documents-1998

Department for Culture, Media, and Sport [DCMS]. (2016). The Culture White Paper. London: Department for Culture, Media & Sport. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/510798/DCMS_The_Culture_White_Paper__3_.pdf

Deleuze, G., & Guattari, F. (1983). Anti-Oedipus: Capitalism and schizophrenia. University of Minnesota Press.

Deleuze, G., & Guattari, F. (1987). A thousand plateaus. University of Minnesota Press.

Gavin, M. (2019). Is there a text in my data? (Part 1): On counting words. Journal of Cultural Analytics, 1(1), 1–24. https://doi.org/10.22148/001c.11830

Gaztambide-Fernández, R. A. (2009). Toward creative solidarity in the “next” moment of curriculum work. In E. Malewski (Ed.), Curriculum studies handbook: The next moment (pp. 78-94). Routledge.

Guattari, F. (2000) The three ecologies. The Althlone Press.

Hayles, N. K. (1993). Virtual bodies and flickering signifiers. October, 66, 69-91. https://doi.org/10.2307/778755

Hayles, N. K. (1999). How we became posthuman: Virtual bodies in cybernetics, literature, and informatics. The University of Chicago Press.

James, K. (2017a). Antics of a cultural recycler: Visualizing data poetry. In L. Butler-Kisber, J. J. Guiney Yallop, M. Stewart, & S. Wiebe (Eds.), Poetic inquiry: Reflections and renewal. (pp. 244-259). MacIntyre Purcell Publishing.

James, K. (2017b). The piano plays a poem. In P. Sameshima, A. Fidyk, K. James, & C. Leggo (Eds.), Poetic inquiry: Enchantment of place. (pp. 49-58). Vernon Press. https://doi.org/10.1080/00332925.2019.1568122

King, T. (2003). The truth about stories. House of Anansi Press.

Klobucar, A., & Ayre, D. (n.d.) GTR Language Workbench. https://web.njit.edu/~newrev/3.0/workbench/Workbench.html

Lessig, L. (2004). Free culture: How big media uses technology and the law to lock down culture and control creativity. Penguin. https://www.loc.gov/item/2019666613/

Moretti, F. (2000). Conjectures on world literature. New Left Review, (1), 54–67. https://newleftreview.org/issues/ii1/articles/franco-moretti-conjectures-on-world-literature

Online Random Tools (n.d.). Word Shuffler. Browserling. https://onlinerandomtools.com/shuffle-words

Palmer (née Reiser), M., & Jones, O. (2014). On breathing and geography: Explorations of data sonifications of timespace processes with illustrating examples from a tidally dynamic landscape. Environment and Planning A, 46(1), 222-240. https://doi.org/10.1068/a45264

Paranyushkin, D. (2011). Identifying the pathways for meaning circulation using text network analysis. Nodus Labs. https://noduslabs.com/research/pathways-meaning-circulation-text-network-analysis/

Paranyushkin, D. (2012). Visualization of texts polysingularity using network analysis. Nodus Labs. https://noduslabs.com/research/visualization-text-polysingularity-network-analysis/

Peña, E., & James, K. (2016). A glitch pedagogy: Exquisite error and the appeal of the accidental. Journal of the Canadian Association for Curriculum Studies, 14(1), 108-127. https://jcacs.journals.yorku.ca/index.php/jcacs/article/view/40306/36169

Peña, E., & James, K. (2020). Raw harmonies: Transmediation through raw data. Leonardo, 53(2), 183-188. https://doi.org/10.1162/leon_a_01635

Richardson, L. (2000). Writing: A method of inquiry. In N. K. Denzin & Y. S. Lincoln, (Eds.), Handbook of qualitative research (2nd ed.). Sage Publications.

Rosenblatt, L. M. (1994). The reader, the text, the poem: The transactional theory of the literary work (Paperback ed.). Southern Illinois University Press.

Schlesinger, P. (2016). The creative economy: Invention of a global orthodoxy. Innovation: The European Journal of Social Science Research, 30(1), 73-90. https://doi.org/10.1080/13511610.2016.1201651

Schulz, K. (2011). What is distant reading? The New York Times. https://www.nytimes.com/2011/06/26/books/review/the-mechanic-muse-what-is-distant-reading.html

Swiss, T., & Burgess, H. (2012). Collaborative new media poetry: Mixed and remixed. In A. A. Delwiche, J. J. Henderson, & Ebooks Corporation (Eds.), The participatory cultures handbook. Routledge. https://doi.org/doi:10.4324/9780203117927

Torres, R. (2005). Digital poetry and collaborative wreadings of literary texts. Telepoesis. http://www.telepoesis.net/papers/dpoetry.pdf

Wetzel, M. (2006). The authority of drawing: Hand, authenticity and authorship. In S. Neef, J. V. Dijck, & F. C. J. Ketelaar (Eds.), Sign here: Handwriting in the age of new media (1st ed.) (pp. 50–59). Amsterdam University Press. https://doi.org/10.25969/mediarep/13237

Wildfeuer, J. (2014). Film discourse interpretation: Towards a new paradigm for multimodal film analysis. Routledge.

Yúdice, G. (2003). The expediency of culture. Duke University Press.