What escapes computation in interactive performance?

Browse the growing collection of provocations here!

Adam Russell


Consider the use of gesture recognition in interactive performance as a form of temporal linkage between the shifting intentions and actions of performers across time. Following Bergson, let us think of what is being linked here (past and present) as not static or point-like, but as a multiplicity of unfolding movements.

Formalising such links requires ‘chopping up’ complex interpenetrating movements which are distributed between people in space and time. But what then are these pieces we chop ourselves and others into? Generally we might refer to them as ‘intervals’ which record some part or aspect of a larger movement, both through temporal boundaries and selection of features. But if we think of these intervals as traces or echoes of lived movement, then we can think of their links too as partial, reductive traces of a larger multiplicity of potential or virtual linkage.

In this sense we can think of a concrete graph of recognised similarity relations over concretely defined intervals as a partial subset of a much, MUCH larger abstract graph that relates all possible intervals at all scales with all other possible intervals. So we might begin to rethink Bergson’s claims for the role of his ‘memory image’ in perception as describing a virtual space in which computable functions between time intervals form only a negligible subset.

Adam Russell

Video credits: Tools that Propel by Sarah Levinsky
(first dancer: Maria Evans; other dancers: see youtube description)

Scott Rubin

Can live artistic interactive performance be formalized? Can an unimaginable dynamic network of interruptions, subversions, coy heterophony, recontextualized mistakes, instantaneous flourishes of communal joy, self-sabotage, and narcissism be formalized? Considering which rules to break, and when; considering when to invent, re-invent, lead, and follow; does computation experience and react with us? What does computation perceive?

-Scott Rubin

Sha Xin Wei

  • The unsolved hard problem of consciousness: the gulf between “internal consciousness” and physical observables, implies that no computational techniques based on physical sensing can yield certificate of intention.   
  • Any choice of measure, or instrument of measurement, makes some aspect of experience legible, and others illegible.
  • Any sensor makes some feature of activity visible and all other features invisible.
  • Movement, as a temporal phenomenon, escapes synchronic representation.
  • Experience > perception > data.  Examples abound from psychoacoustics and psychovision, and sensorimotor studies, gestalt, phenomenology.
  • Felt experience (Eugene Gendlin), and tacit dimension (Michael Polanyi, Satinder Gill).
  • All algorithms on Turing-equivalent machines (i.e. on any digital computer) suffer from absolute limits:
  • The space of computable functions is, in a strict sense, negligible as a subset of the space of all functions on the same domain and range.  In particular, the space of functions mapping between finite discrete subsets of the reals is of measure zero in the space of all continuous (measurable) functions on the reals.
  • Undecidability puts a strong (fatal) bound on what can or cannot be amenable to any algorithm.

-Sha Xin Wei

Jean-Marc Matos

3 distinct interactive and participatory dance projects: Narcissus Reflected, RCO and BodyFail, in which various complex elements escape computation. On the level of the variability of audience involvement and participation, statistical evaluation of audience choices and decisions via their mobile phones, and their physical experiential actions and reactions. Finally, like in the case of BodyFail, it is precisely what escapes proper computation that constitutes the thematic itself of the piece: with non-habitual movements audience can make the computer system reach a crash.

-Jean-Marc Matos, Compagnie K. Danse

Sarah Fdili Alaoui

What escapes computation in artistic performance is almost everything… From rich experiences, intentions, movement in continuous space and time, computation makes numerical representations, and these are incomplete snapshots of a larger story. Rather than the attempt to capture faithfully the phenomena in stake, computation should be a constrained frame that generates few possibilities and de-familiarizes the body.

What escapes computation in artistic performance is the whole conversation on politics of inclusion. Computation doesn’t like outliers nor special cases. It normalizes based on given data. Computation doesn’t engage in social conversation nor in what our future should look like. What escapes computation is the body, with skin, flesh and bones.

My body that needs to sleep, to eat and to have sex. My body that ages. My body that gets tired. My body that can only flex up to its limits. My body that is not isolated. My body that learns and grows. My body that exists in society for the years it has. My body that took time to become and that will take time to end. My body that beholds my experiences, feelings, fears, skills and weaknesses. My body is where artistic performance emerges. My body is so human with all its cells and their connections. That is what escapes computation.

– Sarah Fdili Alaoui

David Coll

You weren’t there, so why do i try to tell you about it?

Presence is a funny thing. In a world that consists mainly of missed connections, how do electronic reminders serve to make one ‘miss’ anything? I feel bad I couldn’t make it to your conference, and I am not ashamed of showing up like this. These emotions strike me as much more crucial than any agency i may or may not have. If you’re reading this, we just created a little feedback for a second, lets leave it at that and go our separate ways.

-David Coll

Baptiste Caramiaux

As a provocation more than a personal statement, I would argue that if by interactive performance we mean technology-mediated performance involving interaction between human and digital technology of some kind, nothing escapes computation. In other words, interactive performance is, by definition, the act of performing computations.

-Baptiste Caramiaux

Lucy Fandel

I am a dance artist and my sister is a hydrogeologist, developing a software that maps tens of thousands of iterations of how groundwater can move through a karst landscape. We are embarking on a project which relates movement on a human scale to the movement of groundwater, with an understanding that these are intimately linked by two essential qualities: ephemerality and physical presence. Yet, dance, geology, and mapping softwares are concerned with movement on vastly different scales and rhythms. Might our choreographic and digital landscapes provoke a renewed sensitivity and conscientious towards the future of the ecological system which we cohabit? Our questions and answers in this project are wholly dependent on the emotional and intellectual investment of the spectators who embark in our project. They are meaningful only in so far as they speak to someone’s lived reality in their landscape and ecosystem. This meeting between human, performance, and landscape might be vividly enabled by maps and models, but it exists outside of them, in a tender and ephemeral place.

-Lucy Fandel

Christine Jackson

I am thinking about audience/spectators/participants/viewers/listeners. The experience of processing interactive performance can never be fully tracked and monitored. There is a deep personal and private aspect to the process of encountering art. I safeguard that actually; I wish to protect it from computation.

-Christine Jackson

Garrett Laroy Johnson

To articulate more-than-human insights, we have to identify what limitations we have already let creep in about how thought can move, what a body can do, and how intentionality can emerge in technical ensembles.

We cannot capture intention, at least so far as intention is commonly construed. What is captured by intention in systems of categorizations like scores or code represent intention as a proxy — that which we authorize to act on behalf of processual intentionality. Systems of categorization impose an *image of intention*. Schematizing both from top-down but also from the past into the present, categories reach forward into the moment of intention’s emergence to retrofit to the already past; they mask the processual with the static. These masks have many names: poses, states, vocabularies. . .

It is in the mode of the proxy that we can capture intention, we may however wish not to; these proxies are real in the sense that their effects are real (e.g. our intentional potentialities are tokenized by the so-called Big Four to open advertising portals made for users *just like us*). It is trivial to create such a system today; so much so that we may come to regard them as a core technique of a computationally engaged research-creation practice.

Plenty of works explore the free play of categorization, what 50-years ago we called moments of Cybernetic Serendipity. They ask what the moving thinking becoming cyborg body can be, about the more-than-human, but these projects are human-all-too-human. Their questions will go unanswered by their interlocutors: expensive productions of Siri, Eliza, or Tay which differ from their antecedents in appearance, not in kind. To ask new questions and construct new problems, we’ll need to lift our gaze from the lurid lure of Narcissus’s lake, lest we drown in the echo of old Humanism’s hubris.

-Garrett Laroy Johnson

Johannes Birringer

Although the notion of choreography hasn’t disappeared in the context of interactive performance and virtual art, it undergoes a re-evaluation in terms of how bodily movement produces data, and how a performer/immersant engages with an interface environment which is dynamic, programmable and networked – thus open to unpredictable and emergent states. Unpredictable states and “affective computing”? What kind of data is handled by computation, and how? For dancers working with wearable sensors or camera-motion-sensing environments, real-time processing means they will surrender most of their kinesthetic and expressive qualities of experiencing movement in-space-over-time to (limited) ways in which programming parameters can “read” or map incoming data back to visual or audio output. What are the mappings of somatic affect? It of course depends on what someone wants to get out of interactive constellations, what programmers are satisfied with. From the perspective of dancers, it’s not illuminating to rely on system-feedback as it cannot be easily re-internalized in the same way in which physical processes work in reality. Thus VR seems a more pertinent context? The articulation of sound and light (images) in augmented reality, through “kinaesonic” gestures, can create tactile feedback in the projected environment outside of the performer’s body. Yet there’s a disjuncture insofar as the data acquired from bodies (making bodies instruments or biophysical objects to be played/extracted from) drive other temporal objects in the environment, they’re not subjectively felt or “known” by the dancer. The machine has not learnt much about gender, race, age, and different abilities either.

-Johannes Birringer

Erin Christine Bell

Sensational description. Within the enactive approach, primarily propelled as a research proposal by Francisco Varela, is the importance of combining felt experience with 3rd person (computational) methods. One solo ‘performance’ of this sort of collaboration was conducted with Dr. Stephen Fulder and the Multidisciplinary Brain Research Center, Bar-Ilan University, Israel – in which Fulder spoke of his experience during meditation as his brain activity was being monitored. In my own research of the felt experience of improvisational dance interaction I have found consistent intensity of sensation located in regions of the chest. Is it possible to use such descriptions as a map, as a guide for regions of the dancers’ bodies to monitor? How might you propose to combine computational technology of the felt experience of interacting performers (especially in real time, during live performance)?

-erin christine bell

Ian Heisters

Is a computer or an algorithm a tool like a paintbrush, or is it something more like a gun? Computers were born in war as means of targeting weapons and gathering intelligence. Have they ever really escaped that history? Our machine learning algorithms are developed by the most powerful corporations in the world for purposes of mining surveillance intelligence from our home videos. The batteries in our laptops, sensors, and gizmos are made from cobalt mined by children under horrific conditions. All this to prop up an economy of short-term resource extraction and long term climate change. If we concern ourselves with how the aesthetic of a piece of software might influence our art making, can we ignore how the morality of the software’s maker might inform our work?

How did you, the reader, the audience member, the professor, the dancer, the technologist, the musician, come to be here today? I mean to ask, what systems of privilege educated you to appreciate art or use a computer? How do you afford to make art? Who’s paying your bills or buying your work? Especially for art that requires all these expensive and problematic electronics?

Can your art escape the history of your tools? Would you perform a duet with an AR-15 and look at me funny if I asked you why? If there’s a difference between an algorithm and a gun, it’s that the former is invisible. Or at least ignored. Our history escapes computation, evades consideration. Very conveniently.

-Ian Heisters

Evelyn Ficarra

Escapes: is not fully explained by computation, is ambiguous on some level, its origin is multivalent or unclear

Computation: calculation, usually to do with mathematics, often with use of computers, but not necessarily, as in colloquial phrase ‘that does not compute’

Interactive: interaction can happen on many levels – chemical, electronic, emotional, physical… implies communication of some kind, possibly following a set of rules, algorithms or expectations. Audience presence at a traditional performance is arguably already interactive, even if all audience does is watch / listen and applaud, human performers will feel different if audience is there compared to performing to no one.

Performance: implies audience, even audience of one, even if performer and audience are the same person. We can perform ourselves in every day life – according to J Butler we ‘perform’ our genders. There are also more formal performances where we buy tickets and watch / experience art involving live performers. Can a machine perform? Yes if there is an audience. Can the machine be its own audience? Yes. Is a machine always ‘performing’? Probably.

If I watch a machine performing, and I’m the only person in the audience, is that interactive, even if nothing I do alters the performance? Yes, the quality of my attention, and my projections of meanings on to the machine, makes it interactive.

What escapes computation in interactive performance? Human presence. Subjectivity of human presence. Emotional interaction / bonding between performers / performers & audience / audience members. The gaze escapes computation. Listening escapes computation. Moving in the fullness of the experience of moving, escapes computation, though elements of it can be broken down and understood in terms of math and physics.

What escapes computation in interactive performance? Thinking it through, I’d have to say – almost everything.

-Evelyn Ficarra

Christopher Knowlton

How can movement computing capture social movements, particularly as civic action begins to take place in digital space? How does internet activism translate to the physical world? Chicago-based Erica Mott Productions’ “Mycelial: Street Parliament” is an immersive performance installation that examines social movements, civic participation and interconnectedness in the digital age. This project is a multi-year interdisciplinary exchange between Egyptian and American artists and scientists. Sentiment analysis of relevant social media feeds during the 2011 Egyptian Revolution and Occupy Wall Street movement decomposed both uprisings into 18 days of five basic emotions: joy, fear, anger, sadness and surprise. This data was then recomposed into five emotive soundscapes, emerging from our composers’ deep exchanges about each culture’s associations to sound. Dancers and audience create the sound score in real-time as they trigger infrared cameras mapped with these soundscapes. A custom-designed mobile app prompts and guides the audience, democratizing the performance and gamifying viewers into their own choreography. As part of our outreach, exhibition and performances, an interactive booth allows people to create a “movement tweet”. This five-second silhouetted movement response can overlay and interact with a library of other tweets, weaving together these individual threads into a larger digital choreographic conversation. These movement tweets are also projected into the space overlaying the live bodies and choreography. Ultimately, “Mycelial” strives to use technology to create an embodied experience that conflates corporeal movements with social movements, one that returns us to kinesthetic empathy where the artistic practice can serve as cultural diplomacy.

-Christopher Knowlton, Erica Mott Productions

Francisca Morand and Javier Jaimovich

1) During the process of movement-sound-biosignal interaction, a movement language emerges that is particular to the technological and creative process. When there is emphasis on movement or dance, the processes of choreographic composition require a particular creation methodology, or can they use non-interactive choreographic composition methods? What are the differences?

2) While composing and designing interactions for a sonic interactive body with physiological signals, regardless of having a unifying theme that drives the artistic work, what are the implications of the order in which the components of the piece are composed? e.g. start with movement? with sound composition? with interaction design?

-Francisca Morand and Javier Jaimovich

Victor Gonzalez-Sanchez

Can we better understand the complex process of human music perception through a standardisation of the quantification of involuntary correspondences between motion and music?

Exploring the frequency-domain and time-domain links between sound and motion signals in a systematic manner might encourage international and interdisciplinary collaboration, boosting developments in the field and knowledge transferability.

-Victor Gonzalez-Sanchez

Jessica Rajko

As I began to write, Ananya Chatterjea’s [1] work immediately came to mind. Chatterjea questions the dominance of somatically informed, postmodern methods of ‘listening to the body’ as having ownership over embodied discourse within dance. In this she does not discredit or devalue such work, but instead articulates that to assume these practices and affiliated research universally speak to the whole of embodied experience both flattens such inquiry and omits others perspectives – particularly those that consider sociocultural, political, and economic ideologies to be inextricably entangled within our multilayered notions of embodiment. Considering Chatterjea’s arguments within the context of this discussion, I ask the following questions:

  • Does the somatic, postmodern practice of deconstructing embodied experience along spatial, temporal, kinesthetic, anatomical parameters neatly align with the limitations, goals, and aims of computational practices?
  • If so, are we aware that this neat coupling both flattens our work and excludes other embodied perspectives?
  • Is our flattening of embodiment akin to what Norman defines as the making of ideal lab spaces; a “clean-room type boxes with level floors, materially bounded to facilitate rigging and equipment maintenance, and the marking up of well delineated test areas?” [2, p.5] If so, what happens if we “diversify and extend the limits of our analytical apparatus?” [2, p.5]

My point here is not do devalue or dramatically change our practices, but to question how the repetition of a specific disciplinary coupling (namely somatically informed dance from postmodern lineages and movement sensing for interaction design) cultivates a flattened definition of ‘meaningful’ research and ‘pleasing’ aesthetics within the field. My aim is to better understand the gaps, limits, and assumptions we make about our practices – not necessarily to change them, but to better articulate their value and limitations beyond the scope of the field.

[1] Chatterjea, A., 2004. Butting out: Reading resistive choreographies through works by Jawole Willa Jo Zollar and Chandralekha. Wesleyan University Press.

[2] Sally-Jane Norman. 2015. Grappling With Movement Models: Performing Arts And Slippery Contexts. In Proceedings of the 2nd International Workshop on Movement and Computing. ACM, New York, NY, USA, 136-141.

-Jessica Rajko

Harald De Bondt

In an interactive performance, the viewer becomes the creator. But because the interaction is limited in time the creation is reduced to “tweaking”. This could be partially resolved by facilitating different ways to prepare and expand the creation outside of the performance.

-Harald De Bondt

Amy LaViers

Can machine bodies express the same ideas as human bodies? This question has been explored in my lab through embodied practice, study of theoretical underpinnings of computer science, and experimental quantification of robotic system capacity. The figure below, from an in progress work (https://arxiv.org/abs/1707.05365), shows the dearth in external complexity of extant robotic systems over the past 15 years (x-axis: internal complexity as measured by number of transistors in on-board CPU; y-axis: external complexity as measured by possible poses).  As a result of this work, I do not think that machines can express the same breadth of behavior as natural systems.

-Amy LaViers

Jaime del Val

A fundamental and underestimated aspect of any embodied experience, including performance, is proprioception, the internal sense of movement of the body, which actually is always in relation to a larger environment, it has a molecular, diffuse and swarm-like movement, connecting to all sensing modalities, for which I call it an alloceptive swarm accross a transmodal sensory continuum. We are always inside this field-swarm, cannot measure it from an outside, it doesn’t move in a pregiven space-time but generates non-linear spacetimes, it doesn’t displace but creates fields of endless internal torsions and tensions, fields of elasticity and vibrancy, of rhythmic indeterminacy. Such Movement is irreducible to algorithms, if we understand algorithms as discrete movement segments that can be codified and recodified. It’s the substrate for a Body Intelligence (BI) that can defy reductive paradigms of disembodied, mental, algorithmic AI. Performance can be a field and process of experimentation that mobilises BI. Creating conditions for the flourising of such perceptual fields, without attempting to reduce them, would be an essential challenge for the design of non-reductive, non-control oriented digital ecologies, and for a neurodiverse and plural culture, for a perceptual democracy to come.

-Jaime del Val

Linda Lewett

Film: Linda Lewett
Multi-media performance: Maida Withers
Performers: Audry Chen, Anthony Gongora, Nikolai Shchetnev, Maida Withers
Music on Film: Solovtida Samosad Band
Photography: Dianne Falk

Nature cycles through destruction and creation just as ‘good’ and ‘evil’ coexist in human nature. Apparent opposites (or reverse sides of a mirror) coexist to fan the flames of life.  Is it foolish to waste energy fighting back? or does the struggle create the energy required for movement?  Perhaps Good and Evil are outmoded concepts in light of new understanding of materialism. Attached media was created to promote solo performance by provocateur dancer choreographer Maida Withers; based on themes in Solzhenitsyn, materiality of memory, and the absurd relationship between the leaders of Russia and the USA.

-Linda Lewett, ARTtvLLc.com

Dawn Stoppiello

What seems to be escaping is an artistic, metaphoric and content-based consideration of the thing being computized within a performance work. Matching the choice of sensing tool/system/instrument to the content of any given work that wants a sensing tool/system/instrument in it. Giving deep thought to the kind of sensing tool/system/instrument and how it will play on the content AND meaning AND experience AND its necessity in the making. BECAUSE each sensing tool/system/instrument will impose a kind of choreography on the maker. BECAUSE the sensing tool/system/instrument is the conduit between the input (the kind of sensing that is measured from a body) and the output (the media result in sound/image/etc). Why does an artist choose to express a work with a guitar over a piano, a paint brush over clay, a Kinect camera over a costume measuring flexion/extension…the method of the sensing SHOULD be chosen to match the nature/quality of the work that wants to be made with/for it. Is that happening? How are movement artists choosing the systems they are working with/for? If you are the body/dancer/choreographer are you choosing your sensing tool/system/instrument or are you being given an “opportunity” by a computer-based artist to make for the tool/system/instrument they devised? Are you being asked to explore a “special sensing machine” that you are unfamiliar with? Are movement artists being handed guitars and asked to make a work with/for it even though they have never played a guitar before? And if so, how well-crafted can the resulting works be? Or are these tool/system/instruments simply controllers with mutable outputs and therefore contain no intrinsic poetic meaning or require any special skill to play – just a joystick used for any game. Discuss.

-Dawn Stoppiello

Christopher Knowlton (2)

Many dancers understand that their medium – dance, and more broadly, movement – is not spontaneously generated. The accumulated amalgamation of movement patterns learned from family, culture, teachers, training and observations form a veritable genealogy that finds expression through the moving body. Dance writers, historians and even audience members often classify these movements into styles based on similarities to other forms they have observed. With enough expertise in viewing, one can even identify an individual dancer’s region or mentor, similar to how one might identify regional accents while speaking. In parallel, biomechanists understand that movement contains intrinsic information about the body, and that information can be made readable through motion capture technology. Like writing, an early technology that makes a record of language, motion capture can create a record of the embodied communication that is dance. However, many approaches in movement computing tend to lose, ignore or simplify the cultural context in which movement occurs. This is significant, as dance, like language, is a cultural product. Using motion analysis, can we quantify and analyze the relationships between dance forms in the way linguistics analyzes language? How might that challenge pre-constituted systems of recognition, classification and representation in dance? What might we discover about identity and embodied cultural knowledge? As a professional dancer and biomechanist in Chicago, I propose a biomechanical mapping of dance that employs machine learning techniques to motion capture data. In doing so, I hope to better understand the complex interactions of movement practices, culture and identity in dance.

-Christopher Knowlton

Zhi XU

The variable body.

-Zhi XU

Lauren Hayes

What importance can established practices such as improvisation in the fields of dance, somatics, and electronic music offer in thinking about the relationships between performance, media, and space?

LLEAPP was started by creative music practitioners and early career researchers at the University of Edinburgh, UK, in 2009. It provides a forum for self-directed creative practice research and has since been hosted at the University of East Anglia (UK), De Montfort University (UK), and Newcastle University/Culture Lab (UK). PARIESA were proud to host its 6th edition within the School of Arts, Media + Engineering, Herberger Institute for Design and the Arts, Arizona State University. www.lleapp.org

-Lauren Hayes

Jean-Philippe Rivière

In my opinion, if I had to choose from the vast panel of what escapes computation in artistic performance, I’d say experience. Experience of the performer but also the experience of the audience. Through experience, the performer will develop a proper style and will create a performance different from that of someone else. In the same way, someone who looks at the performance will look at this performance through the prism of his own experience. Even a self learning machine may itself have a different experience from ours. Paradoxically, the fact of learning, will prevent the machine to quantify or compute experience. Because that would be subject to his own interpretation.

-Jean-Philippe Rivière