Category: Things

Exploring Cinematic Mixed Realities: Deformative Methods for Augmented and Virtual Film and Media Studies

Arguably, all cinema, with its projection of three-dimensional spaces onto a two-dimensional screen, is a form of mixed reality. But some forms of cinema are more emphatically interested in mixing realities—like Hale’s Tours (dating back to 1904), which staged its kinesthetic, rollercoaster-like spectacles of railway travel inside of a train car that rocked back and forth but otherwise remained stationary. Here the audience of fellow “passengers” experienced thrills that depended not so much on believing as on corporeally feelingthe effects of the simulation, an embodied experience that was at once an experience of simulated travel and of the technology of simulation. Evoking what Neil Harris has called an “operational aesthetic,” attention here was split, as it is in so many of our contemporary augmented and virtual reality experiences, between the spectacle itself and its means of production. That is, audiences are asked both to marvel at the fictional scenario’s spectacular images and, as in the case of the “bullet time” popularized a century later by The Matrix, to wonder in amazement at the achievement of the spectacle by its underlying technical apparatus. The popularity of “making of” videos and VFX reels attests to a continuity across cinematic and computational (or post-cinematic) forms of mixed reality, despite very important technological differences—including most centrally the emergence of digital media operating at scales and speeds that by far exceed human perception. Seen from this angle, part of the appeal—and also the effectiveness—of contemporary AR, VR, and other mixed reality technologies lies in this outstripping of perception, whereby the spectacle mediates to us an embodied aesthetic experience of the altogether nonhuman dimensionality of computational processing. But how, beyond theorizing historical precursors and aesthetic forms, can this insight be harnessed practically for the study of film and moving-image media?

Taking a cue from Kevin L. Ferguson’s volumetric explorations of cinematic spaces with the biomedical and scientific imaging software ImageJ, I have been experimenting with mixed-reality methods of analysis and thinking about the feedback loops they initiate between embodied experience and computational processes that are at once the object and the medium of analysis. Here, for example, I have taken the famous bullet-time sequence and imported it as a stack of images into ImageJ, using the 3D Viewer plugin to transform what Gilles Deleuze called cinema’s presentation of a “bloc of space-time” into a literal block of bullet-time. This emphatically post-cinematic deformation uses transparency settings to gain computational insight into the virtual construction of a space that can be explored further in VR and AR settings as abstract traces of informational processing. Turned into a kind of monument that mixes human and computational spatiotemporal forms, this is a self-reflexive mixed reality that provides aesthetic experience of low-level human-computational interfacing—or, more pointedly, that re-constitutes aesthesis itself as mixed reality.

Clearly, this is an experimental approach that is not interested in positivistic ideas of leveraging digital media to capture and reconstruct reality, but instead approaches AR and VR technologies as an opportunity to transform and re-mix reality through self-reflexively recursive technoaesthetic operations. Here, for example, I have taken the bullet-time sequence, produced with the help of photogrammetric processes along with digital smoothing and chromakeying or green-screen replacement, and fed it back into photogrammetry software in order to distill a spatial environment and figural forms that can be explored further in virtual and augmented scenarios. Doing so does not, of course, present to us a “truth” understood as a faithful reconstruction of pro-filmic reality. On the contrary, the abstraction and incoherence of these objects foreground the collision of human and informatic realities and incompatible relations to time and space. If such processes have analytical or theoretical value, it resides not in a positivistic but rather a deformative relation to data, both computational and experiential. Indeed, the payoff, as I see it, of interacting with these objects is in the emergence of a new operational aesthetic, one that transforms the original operational aesthetic of the scenario—its splitting of attention between spectacle and apparatus—and redirects it to a second-order awareness of our involvement in mixed reality as itself a volatile mixture of technoaesthetic forms. Ultimately, this approach questions the boundaries between art and technology and reimagines the “doing” of digital media theory as a form of embodied, operational, and aesthetic practice.