Exploring Cinematic Mixed Realities

Exploring Cinematic Mixed Realities: Deformative Methods for Augmented and Virtual Film and Media Studies

Arguably, all cinema, with its projection of three-dimensional spaces onto a two-dimensional screen, is a form of mixed reality. But some forms of cinema are more emphatically interested in mixing realities—like Hale’s Tours (dating back to 1904), which staged its kinesthetic, rollercoaster-like spectacles of railway travel inside of a train car that rocked back and forth but otherwise remained stationary. Here the audience of fellow “passengers” experienced thrills that depended not so much on believing as on corporeally feelingthe effects of the simulation, an embodied experience that was at once an experience of simulated travel and of the technology of simulation. Evoking what Neil Harris has called an “operational aesthetic,” attention here was split, as it is in so many of our contemporary augmented and virtual reality experiences, between the spectacle itself and its means of production. That is, audiences are asked both to marvel at the fictional scenario’s spectacular images and, as in the case of the “bullet time” popularized a century later by The Matrix, to wonder in amazement at the achievement of the spectacle by its underlying technical apparatus. The popularity of “making of” videos and VFX reels attests to a continuity across cinematic and computational (or post-cinematic) forms of mixed reality, despite very important technological differences—including most centrally the emergence of digital media operating at scales and speeds that by far exceed human perception. Seen from this angle, part of the appeal—and also the effectiveness—of contemporary AR, VR, and other mixed reality technologies lies in this outstripping of perception, whereby the spectacle mediates to us an embodied aesthetic experience of the altogether nonhuman dimensionality of computational processing. But how, beyond theorizing historical precursors and aesthetic forms, can this insight be harnessed practically for the study of film and moving-image media?

Taking a cue from Kevin L. Ferguson’s volumetric explorations of cinematic spaces with the biomedical and scientific imaging software ImageJ, I have been experimenting with mixed-reality methods of analysis and thinking about the feedback loops they initiate between embodied experience and computational processes that are at once the object and the medium of analysis. Here, for example, I have taken the famous bullet-time sequence and imported it as a stack of images into ImageJ, using the 3D Viewer plugin to transform what Gilles Deleuze called cinema’s presentation of a “bloc of space-time” into a literal block of bullet-time. This emphatically post-cinematic deformation uses transparency settings to gain computational insight into the virtual construction of a space that can be explored further in VR and AR settings as abstract traces of informational processing. Turned into a kind of monument that mixes human and computational spatiotemporal forms, this is a self-reflexive mixed reality that provides aesthetic experience of low-level human-computational interfacing—or, more pointedly, that re-constitutes aesthesis itself as mixed reality.

Clearly, this is an experimental approach that is not interested in positivistic ideas of leveraging digital media to capture and reconstruct reality, but instead approaches AR and VR technologies as an opportunity to transform and re-mix reality through self-reflexively recursive technoaesthetic operations. Here, for example, I have taken the bullet-time sequence, produced with the help of photogrammetric processes along with digital smoothing and chromakeying or green-screen replacement, and fed it back into photogrammetry software in order to distill a spatial environment and figural forms that can be explored further in virtual and augmented scenarios. Doing so does not, of course, present to us a “truth” understood as a faithful reconstruction of pro-filmic reality. On the contrary, the abstraction and incoherence of these objects foreground the collision of human and informatic realities and incompatible relations to time and space. If such processes have analytical or theoretical value, it resides not in a positivistic but rather a deformative relation to data, both computational and experiential. Indeed, the payoff, as I see it, of interacting with these objects is in the emergence of a new operational aesthetic, one that transforms the original operational aesthetic of the scenario—its splitting of attention between spectacle and apparatus—and redirects it to a second-order awareness of our involvement in mixed reality as itself a volatile mixture of technoaesthetic forms. Ultimately, this approach questions the boundaries between art and technology and reimagines the “doing” of digital media theory as a form of embodied, operational, and aesthetic practice.

Virtual and Augmented Reality Digital (and/or Deformative?) Humanities Institute at Duke

I am excited to be participating in the the NEH-funded Virtual and Augmented Reality Digital Humanities Institute — or V/AR-DHI — next month (July 23 – August 3, 2018) at Duke University. I am hoping to adapt “deformative” methods (as described by Mark Sample following a provocation from Lisa Samuels and Jerome McGann) as a means of transformatively interrogating audiovisual media such as film and digital video in the spaces opened up by virtual and augmented reality technologies. In preparation, I have been experimenting with photogrammetric methods to reconstruct the three-dimensional spaces depicted on two-dimensional screens. The results, so far, have been … modest — nothing yet in comparison to artist Claire Hentschker’s excellent Shining360 (2016) or Gregory Chatonsky’s The Kiss (2015). There is something interesting, though, about the dispersal of the character Neo’s body into an amorphous blob and the disappearance of bullet time’s eponymous bullet in this scene from The Matrix, and there’s something incredibly eerie about the hidden image behind the image in this famous scene from Frankenstein, where the monster’s face is first revealed and his head made virtually to protrude from the screen through a series of jump cuts. Certainly, these tests stand in an intriguing (if uncertain) deformative relation to these iconic moments. In any case, I look forward to seeing where (if anywhere) this leads, and to experimenting further at the Institute next month.

The Video Essay: Writing with Video About Film and Media

2017-08-01 09.45.19 am

This fall, I am excited to teach a new course, “The Video Essay: Writing with Video About Film and Media,” as a part of Stanford’s Introductory Seminars program. Geared towards sophomores from any major, this small class will combine practical instruction in video editing, analysis and discussion of exemplary video essays, hands-on lab sessions, and group critique of student work.

The course draws essential inspiration from the NEH-funded “Scholarship in Sound & Image” workshop, organized by Christian Keathley and Jason Mittell at Middlebury College, which I participated in back in 2015.

More info about the course can be found on Stanford’s Introductory Seminars website.

#SCMS17 Conference Program

16864700_1218972544844963_3352301399512773470_n

The conference program for the 2017 SCMS conference in Chicago is now available online (opens as PDF). As I mentioned recently, I will be participating in panel K3 (Friday, March 24, 2017, 9:00-10:45am) — a workshop dedicated to “Deformative Criticism and Digital Experimentations in Film & Media Studies.”

Speculative Data: Full Text, MLA 2016 #WeirdDH

SpeculativeData-jpg.001

Below you’ll find the full text of my talk from the Weird DH panel organized by Mark Sample at the 2016 MLA conference in Austin Texas. Other speakers on the panel included Jeremy Justus, Micki Kaufman, and Kim Knight.

***

Speculative Data: Post-Empirical Approaches to the “Datafication” of Affect and Activity

Shane Denson, Duke University

A common critique of the digital humanities questions the relevance (or propriety) of quantitative, data-based methods for the study of literature and culture; in its most extreme form, this type of criticism insinuates a complicity between DH and the neoliberal techno-culture that turns all human activity, if not all of life itself, into “big data” to be mined for profit. Now, it may sound from this description that I am simply setting up a strawman to knock down, so I should admit up front that I am not wholly unsympathetic to the critique of datafication. But I do want to complicate things a bit. Specifically, I want to draw on recent reconceptions of DH as “deformed humanities” – as an aesthetically and politically invested field of “deformance”-based practice – and describe some ways in which a decidedly “weird” DH can avail itself of data collection in order to interrogate and critique “datafication” itself.

SpeculativeData-jpg.002

My focus is on work conducted in and around Duke University’s S-1: Speculative Sensation Lab, where literary scholars, media theorists, artists, and “makers” of all sorts collaborate on projects that blur the boundaries between art and digital scholarship. The S-1 Lab, co-directed by Mark Hansen and Mark Olson, experiments with biometric and environmental sensing technologies to expand our access to sensory experience beyond the five senses. Much of our work involves making “things to think with,” i.e. experimental “set-ups” designed to generate theoretical and aesthetic insight and to focus our mediated sensory apparatus on the conditions of mediation itself. Harnessing digital technologies for the work of media theory, this experimentation can rightly be classed, alongside such practices as “critical making,” in the broad space of the digital humanities. But due to their emphatically self-reflexive nature, these experiments challenge borders between theory and practice, scholarship and art, and must therefore be qualified, following Mark Sample, as decidedly “weird DH.”

SpeculativeData-jpg.003.jpeg

One such project, Manifest Data, uses a piece of “benevolent spyware” that collects and parses data about personal Internet usage in such a way as to produce 3D-printable sculptural objects, thus giving form to data and reclaiming its personal value from corporate cooptation. In a way that is both symbolic and material, this project counters the invisibility and “naturalness” of mechanisms by which companies like Google and Facebook expropriate value from the data we produce. Through a series of translations between the digital and the physical—through a multi-stage process of collecting, sculpting, resculpting, and manifesting data in virtual, physical, and augmented spaces—the project highlights the materiality of the interface between human and nonhuman agencies in an increasingly datafied field of activity. (If you’re interested in this project, which involves “data portraits” based on users’ online activity and even some weird data-driven garden gnomes designed to dispel the bad spirits of digital capital, you can read more about it in the latest issue of Hyperrhiz.)

SpeculativeData-jpg.004

Another ongoing project, about which I will say more in a moment, uses data collected through (scientifically questionable) biofeedback devices to perform realtime collective transformations of audiovisual materials, opening theoretical notions of what Steven Shaviro calls “post-cinematic affect” to robustly material, media-archaeological, and aesthetic investigations.

SpeculativeData-jpg.005

These and other projects, I contend, point the way towards a truly “weird DH” that is reflexive enough to suspect its own data-driven methods but not paralyzed into inactivity.

Weird DH and/as Digital Critical (Media) Studies:

So I’m trying to position these projects as a form of weird digital critical (media) studies, designed to enact and reflect (in increasingly self-reflexive ways) on the use of digital tools and processes for the interrogation of the material, cultural, and medial parameters of life in digital environments.

SpeculativeData-jpg.006

Using digital techniques to reflect on the affordances and limitations of digital media and interfaces, these projects are close in spirit to new media art, but they are also apposite with practices and theories of “digital rhetoric,” as described by Doug Eyman, with Gregory Ulman’s “electracy,” or with Casey Boyle’s posthuman rhetoric of multistability, which celebrates the rhetorical affordances of digital glitches in exposing the affordances and limitations of computational media in the broader realm of an interagential relational field that includes both humans and nonhumans. In short, these projects enact what we might call, following Stanley Cavell, the “automatisms” of digital media – the generative affordances and limitations that are constantly produced, reproduced, and potentially transformed or “deformed” in creative engagements with media. Digital tools are used in such a way as to problematize their very instrumentality, hence moving towards a post-empirical or post-positivistic form of datafication as much as towards a post-instrumental digitality.

SpeculativeData-jpg.007

Algorithmic Nickelodeon / Datafied Attention:

My key example is a project tentatively called the “algorithmic nickelodeon.” Here we use consumer-grade EEG headsets to interrogate the media-technical construction and capture of human attention, and thus to complicate datafication by subjecting it to self-reflexive, speculative, and media-archaeological operations. The devices in question cost about $100 and are marketed as tools for improving concentration, attention, and memory. The headset measures a variety of brainwave activity and, by means of a proprietary algorithm, computes values for “attention” and “meditation” that can be tracked and, with the help of software applications, trained and supposedly optimized. In the S-1 Lab, we have sought to tap into these processes in order not just to criticize the scientifically dubious nature of these claims but rather to probe and better understand the nature of the automatisms and interfaces taking place here and in media of attention more generally. Specifically, we have designed a film- and media-theoretical application of the apparatus, which allows us to think early and contemporary moving images together, to conceive pre- and post-cinema in terms of their common deviations from the attention economy of classical cinema, and to reflect more broadly on the technological-material reorganizations of attention involved in media change. This is an emphatically experimental (that is, speculative, post-positivistic) application, and it involves a sort of post-cinematic reenactment of early film’s viewing situations in the context of traveling shows, vaudeville theaters, and nickelodeons. With the help of a Python script written by lab member Luke Caldwell, a group of viewers wearing the Neurosky EEG devices influence the playback of video clips in real time, for example changing the speed of a video or the size of the projected image in response to changes in attention as registered through brain-wave activity.

At the center of the experimentation is the fact of “time-axis manipulation,” which Friedrich Kittler highlights as one of the truly novel affordances of technical media, like the phonograph and cinema, that arose around 1900 and marked, for him, a radical departure from the symbolic realms of pre-technical arts and literature. Now it became possible to inscribe “reality itself,” or to record a spectrum of frequencies (like sound and light) directly, unfiltered through alphabetic writing; and it became possible as well to manipulate the speed or even playback direction of this reality.

SpeculativeData-jpg.009

Recall that the cinema’s standard of 24 fps only solidified and became obligatory with the introduction of sound, as a solution to a concrete problem introduced by the addition of a sonic register to filmic images. Before the late 1920s, and especially in the first two decades of film, there was a great deal of variability in projection speed, and this was “a feature, not a bug” of the early cinematic setup. Kittler writes: “standardization is always upper management’s escape from technological possibilities. In serious matters such as test procedures or mass entertainment, TAM [time-axis manipulation] remains triumphant. [….] frequency modulation is indeed the technological correlative of attention” (Gramophone Film Typewriter 34-35). Kittler’s pomp aside, his statement highlights a significant fact about the early film experience: Early projectionists, who were simultaneously film editors and entertainers in their own right, would modulate the speed of their hand-cranked apparatuses in response to their audience’s interest and attention. If the audience was bored by a plodding bit of exposition, the projectionist could speed it up to get to a more exciting part of the movie, for example. Crucially, though: the early projectionist could only respond to the outward signs of the audience’s interest, excitement, or attention – as embodied, for example, in a yawn, a boo, or a cheer.

SpeculativeData-jpg.010

But with the help of an EEG, we can read human attention – or some construction of “attention” – directly, even in cases where there is no outward or voluntary expression of it, and even without its conscious registration. By correlating the speed of projection to these inward and involuntary movements of the audience’s neurological apparatus, such that low attention levels cause the images to speed up or slow down, attention is rendered visible and, to a certain extent, opened to conscious and collective efforts to manipulate it and the frequency of images now indexed to it.

According to Hugo Münsterberg, who wrote one of the first book-length works of film theory in 1916, cinema’s images anyway embody, externalize, and make visible the faculties of human psychology; “attention,” for example, is said to be embodied by the close-up. With our EEG setup, we can literalize Münsterberg’s claim by correlating higher attention levels with a greater zoom factor applied to the projected image. If the audience pays attention, the image grows; if attention flags, the image shrinks. But this literalization raises more questions than it answers, it would seem. On the one hand, it participates in a process of “datafication,” turning brain wave patterns into a stream of data called “attention,” but whose relation to attention in ordinary senses is altogether unclear. But this datafication simultaneously opens up a space of affective or aesthetic experience in which the problematic nature of the experimental “set-up” announces itself to us in a self-reflexive doubling: we realize suddenly that “it’s a setup”; “we’ve been framed” – first by the cinema’s construction of attentive spectators and now by this digital apparatus that treats attention as an algorithmically computed value.

So in a way, the apparatus is a pedagogical/didactic tool: it not only allows us to reenact (in a highly transformed manner) the experience of early cinema, but it also helps us to think about the construction of “attention” itself in technical apparatuses both then and now. In addition to this function, it also generates a lot of data that can indeed be subjected to statistical analysis, correlation, and visualization, and that might be marshaled in arguments about the comparative medial impacts or effects of various media regimes. Our point, however, remains more critical, and highly dubious of any positivistic understanding of this data. The technocrats of the advertising industry, the true inheritors of Münsterberg the industrial psychologist, are anyway much more effective at instrumentalizing attention and reducing it to a psychotechnical variable. With a sufficiently “weird” DH approach, we hope to stimulate a more speculative, non-positivistic, and hence post-empirical relation to such datafication. Remitting contemporary attention procedures to the early establishment of what Kittler refers to as the “link between physiology and technology” (73) upon which modern entertainment media are built, this weird DH aims not only to explore the current transformations of affect, attention, and agency – that is, to study their reconfigurations – but also potentially to empower media users to influence such configuration, if only on a small scale, rather than leave it completely up to the technocrats.

Speculative Data #WeirdDH #MLA16 #S107

storify-mla16

The “Weird DH” panel at MLA 2016 in Austin, chaired by Mark Sample, was great fun, and it generated quite a bit of discussion, both online and off. Luckily, Eileen Clancy preserved the twitter discussion in a Storify (https://storify.com/clancynewyork/weird-dh), so in case you couldn’t make it out last week, you can still catch up on some of the topics and reactions to the talks by Jeremy Justus, Micki Kaufman, Kim Knight, and myself.

(If I get around to it, I will also be posting the full text of my talk very soon.)

Conversations in the Digital Humanities at Duke

page-events-dhduke

Today, Oct. 2, 2015, the Franklin Humanities Institute, the Wired! Lab, the PhD Lab in Digital Knowledge, and HASTAC@Duke will be presenting “Conversations in the Digital Humanities,” the inaugural event of the new Digital Humanities Initiative at Duke University. More information about the event, in which I will be participating alongside colleagues from the S-1: Speculative Sensation Lab, can be found on the FHI website.

Also, all of the 10-minute “lightning talks” will be live-streamed. The first block of sessions, from 2:15-3:45pm EST, will be streamed here, and the second block, from 4:00-5:40pm, will be viewable here. (Apparently, the videos will be archived and available after the fact as well.)

Here is the complete schedule:

2:00 – 2:15
Welcome and Introduction to Digital Humanities Initiative

2:15 – 3:45 
Session 1 (10 minutes per talk)

  1. Project Vox (Andrew Janiak, and Liz Milewicz)
  2. NC Jukebox (Trudi Abel, Victoria Szabo)
  3. Visualizing Cultures: The Shiseido Project (Gennifer Weisenfeld)
  4. Going Global in Mughal India (Sumathi Ramaswamy)
  5. Israel’s Occupation in the Digital Age (Rebecca Stein)
  6. Digital Athens: Archaeology meets ArcGIS (Tim Shea, Sheila Dillon)
  7. Early Medieval Networks (J. Clare Woods)

3:45 – 4:00
Coffee Break

4:00 – 5:40 
Session 2 (10 minutes per talk)

  1. Painting the Apostles – A Case Study in “The Lives of Things” (Mark Olson, Mariano Tepper, and Caroline Bruzelius)
  2. Digital Archaeology: From the Field to Virtual Reality (Maurizio Forte)
  3. The Memory Project (Luo Zhou)
  4. Veoveo, children at play (Raquel Salvatella de Prada)
  5. “Things to Think With”: Weird DH, Data, and Experimental Media Theory (S-1 Lab)
  6. s_traits, Generative Authorship and the Emergence Lab (Bill Seaman and John Supko)
  7. Found Objects and Fireflies (Scott Lindroth)
  8. Project Provoke (Mary Caton Lingold and others)

5:40 – 6:00 
Reception

Things to Think With

mindwave2

As a late addition to the program, the Duke S-1 Speculative Sensation Lab will be participating in “Conversations in the Digital Humanities” this coming Friday, October 2, 2015, at the Franklin Humanities Institute at Duke. The event, which will consist of a series of brief “lightning talks” on a range of topics that run the gamut of current DH work, will take place from 2:00-6:00pm in the FHI Garage in Smith Warehouse, Bay 4. More info here: Conversations in the Digital Humanities.

Here is the abstract for the S-1 Lab’s presentation, which I will be participating in along with Lab co-director Mark Olson and our resident programmer Luke Caldwell:

“Things to Think With”: Weird DH, Data, and Experimental Media Theory

S-1 Speculative Sensation Lab

The S-1 Speculative Sensation Lab, co-directed by Mark Hansen and Mark Olson, experiments with biometric and environmental sensing technologies to expand our access to sensory experience beyond the five senses. Much of our work involves making “things to think with,” i.e. experimental “set-ups” designed to generate theoretical and aesthetic insight and to focus our mediated sensory apparatus on the conditions of mediation itself. Harnessing digital technologies for the work of media theory, this experimentation can rightly be classed, alongside such practices as “critical making,” in the broad space of the digital humanities. But due to their emphatically self-reflexive nature, these experiments challenge borders between theory and practice, scholarship and art, and must therefore be qualified, following Mark Sample, as decidedly “weird DH.”

In this presentation, we discuss a current project that utilizes consumer-grade EEG headsets, in conjunction with a custom Python script by lab member Luke Caldwell, to reflect on the contemporary shape of “attention,” as it is constructed and addressed in individual and networked forms across media ranging from early cinema to “post-cinema.”

Sculpting Data (and Painting Networks)

data-gnome-animation

On March 28, 2015, members of the Duke S-1 Speculative Sensation Lab will take over a panel at the 2015 AEGS Conference <how do you do Digital Humanities?>. (See here for the conference website, which includes the full program.) General conference info:

The conference will be held in Tompkins Hall on the NC State University campus in Raleigh, NC, on Friday, March 27th and Saturday, March 28th.  Friday evening we will host a keynote panel of Digital Humanities scholars. These scholars will discuss how they “do” Digital Humanities in their research and pedagogy. On Saturday, participants will present their research in 15 minutes presentations.
Again, the final panel of the conference, Session IV (1:55 – 3:10pm on Saturday, March 28), will be devoted to the S-1 Lab’s recent work, especially the Manifest Data project that I have been posting about here. Titled “Digital Metabolisms: Manifesting Data as a Collaborative Research Process,” the panel consists of the following presentations:

Amanda Starling Gould, Duke University, “Digital Metabolism: Using Digital Tools to Hack Humanities Research”

Luke Caldwell, Duke University, “Leveraging Benevolent Spyware for Humanities Research”

Libi Striegl, Duke University, “3D Printing as Artistic Research Intervention”

Karin & Shane Denson, Duke University, “Sculpting Data”

David Rambo, Duke University, “Manifest Data as Digital Manifest Destiny”

(Observant readers of this blog will notice that I am to give two presentations on March 28: both at NC State and at the SCMS conference in Montreal. In fact, Karin will be representing the two of us in Raleigh, but we’re putting together some presentation materials that we’re quite proud of — and that we think will creatively solve the logistical problems of being in two places at once! More soon!)

Mario Modding Madness

2015-02-03 09.24.30 pm

In case you missed it: you can watch a split-screen video presentation of my digital humanities-oriented talk, “Visualizing Digital Seriality,” which I gave last Friday, January 30, 2015, at Duke University — here (or click the image above).

More about the project can be found here.