From GUI to NUI: Microsoft’s Kinect and the Politics of the (Body as) Interface

Article PDF

Introduction: Selfe and Selfe’s Impact on (Graphical-User) Interface Studies

In computers and writing (C&W) scholarship, interface studies has been a minor but persistent scholarly topic in numerous publications. Beginning with Patricia Sullivan’s article from the late 1980s, in which she argues that scholars associated with technical communication should turn their attention to the subject, “interface” has been explored by scholars in a wide range of fields and sub-fields associated with C&W, including the following: rhetoric theory (Carnegie; Carpenter; Brooke), composition theory and pedagogy (Buck; Haas and Gardner; Mardsjo; Rosinski and Squire; Selfe and Selfe), multimodal composition (Fagerjord; Skulstad and Morrison), and technical and networked communication (Grabill; Spinuzzi).¹ One approach to interface studies that has predominated in the scholarship is related to socio-political issues of equity, identity, and access, and since its publication almost twenty years ago, Cynthia Selfe and Richard Selfe’s article, “The Politics of the Interface,” has been a touchstone for this critical approach.

Selfe and Selfe’s article is an oft-cited reference in articles about the socio-political impact of technological interfaces for good reason. The article is an impassioned call for a “critical and reflective stance” toward the adoption of computers in the classroom (482). It was written at a time when a growing number of administrators and faculty viewed computers as a relatively benign but powerful way to update our classrooms and methods of teaching. Countering that early-adopter fever, Selfe and Selfe challenged us to scrutinize the ways in which software interfaces reintroduce forms of oppression and marginalization that we had been working to extricate from our classrooms. Like a Trojan Horse, the socio-cultural biases against which we’d fought for so long risked being inadvertently reinscribed into the newly established forms of human-computer interaction. After all, technologies are not neutral, and interfaces in particular are expressive of a wide range of socio-cultural and epistemological biases that can impede our pedagogical objectives.

Selfe and Selfe’s article is divided in two parts. The first comprises their critiques of interface technologies. It is the longer of the two sections. Three of the arguments they develop are the following. First, computers do not necessarily serve democratic ends: “Computer interfaces . . . are also sites within which the ideological and material legacies of racism, sexism, and colonialism are continuously written and rewritten” (484). Second, the “desktop metaphor” expresses the values of corporate culture, which reinforces the values of white middle- and upper-class users (487). Third, the structure and logic of computer interfaces are aligned with the values of rationality, hierarchy, and logocentrism characteristic of Western patriarchal cultures (491). In the shorter second part, they offer three tactics or suggestions for addressing the problems with interfaces that they had just critiqued. First, they encourage a general level of critical awareness about technology issues on the part of both pre-service and in-service teachers (496). Second, they encourage faculty to develop their own interfaces, their own software (497). Finally, they invite students to create mock-ups of their preferred interfaces—interfaces expressive of an alternative (499).

While Selfe and Selfe’s article has had a lasting effect on our field because it helped us recognize the socio-politics of the graphical-user interface (GUI), twenty years is a ‘long time ago’ in the history of popular computing. In that time, the shift from GUI to NUI (natural-user interface) technologies has transformed what a computing environment is. Today, computing environments are increasingly immersive experiences in which it is difficult to distinguish between a user and a system, which means that the ways in which we achieve a critical and reflective stance is changing.

Epitomizing the difficulties NUI technologies introduce is one of the tag lines for Microsoft’s Kinect, “You are the controller!” (see Figure 3 above). If we are the controller/interface, from where do we stand in order to act critically and reflectively? If, in an immersive, NUI environment it’s no longer possible to have a standpoint, a point of perspective, how do we critique the technology?

In order to extend the legacy of Selfe and Selfe’s contribution to interface studies, I offer two interrelated propositions. The first, related to the exigence of the user-as-interface, is that we redirect our critical attention from the interface on the screen to what Adrienne Rich once called the “geography closest in” (i.e., our bodies)—and specifically to tactical or critical explorations of a body’s potential. The power of NUI technologies is derived from the use of our voice and gestures to engage with a system. At this point, most NUI interfaces with which we interact look like GUIs, but, as NUIs mature, the ways in which we control a computational environment will have less to do with on-screen metaphors, icons, and buttons and more to do with an ethos that we perform. Anticipating this radical shift in computing is an opportunity to redefine the focal point of our critical concerns.

The second proposition is in response to the mainstreaming of computational thinking—in part due to the availability of easy-to-learn programming languages. Since Selfe and Selfe’s article was published, programming is no longer something that wizards do exclusively on the other side of the two-cultures split. At this point in the history of popular computing, humanists can engage directly with computational media. Numerous high-level languages have become available that have lowered the learning curve, and several of those languages, such as Processing, offer novice programmers powerful ways to design immersive, interactive environments. Related to this point, computers and networks have become fast enough to make programming an expressive art because the historical limits on memory, bandwidth, and processor speed do not constrain programmers to the equivalent of a current-traditional emphasis on conciseness and clarity in writing. Today, alongside computer science, an experimental, ad hoc culture of open (i.e., free and collaborative), experimental work has blossomed.

For these reasons, it’s time to focus on the production of novel interfaces that embody our critical values and concerns, rather than limiting ourselves to the analysis and critique of someone else’s technologies. In Selfe and Selfe’s article, the call to invent new interfaces is overshadowed by a focus on representational critique. As I’ll explain shortly, based in part on my experiences working with Microsoft’s Kinect, socio-cultural critique in immersive NUI environments leads to the conclusion that representational critique must be transformed into an inventional art, a post-representational, tactical pursuit.

NUI on the Rise

In a presentation titled “Predicting the Past,” August de los Reyes explains that NUIs comprise a suite of human-computer technologies including voice, touch, gesture, and stylus or pen. When you interact with a computer system by speaking to it, tapping or swiping across it, gesturing toward it in the air, or using a stylus to write on a screen, you are using a NUI technology. According to De los Reyes, NUI will be the third paradigm shift in computing. In Figure 4 above, which is a graphic from his PowerPoint presentation, De Los Reyes depicts an evolution in popular computing that begins with the command-line interface, and after the present era of GUI, shifts to NUI. It’s also worthwhile emphasizing that the historical movement toward NUI coincides with a movement toward the increasingly intuitive and directly accessed, which means that the implied line separating self and technology is blurred.

It is important to note that some NUI technologies have been around for several decades, but it’s the relatively recent explosion of interest in and profitability from products including Apple’s iPhone and iPad and Microsoft’s Kinect that has led to the uptrend in positive discourse and outright evangelism about them—especially from Microsoft.

In the following three quotes, which are paradigmatic of the optimism and excitement around NUIs, a new age of computing seems apparent. In their book, Brave NUI World, Daniel Wigdor and Dennis Wixon write, “Now we stand at the brink of another potential evolution in computing. Natural-user interfaces (NUIs) seem to be in a position similar to the GUI in the early 1980s” (2). Chief Research and Strategy Officer at Microsoft Craig Mundie claims, “The transition to a natural user interface will change everything from the way students write term papers and play computer games to how scientists study global population growth and its impact on our natural resources.” Even Bill Gates has responded to the promise of NUI. In an essay titled “The Power of the Natural User Interface,” in which he focuses on the Kinect, Gates writes, “One of the most important current trends in digital technology is the emergence of natural user interface, or NUI. We’ve had things like touch screen and voice recognition for a while now. But with Kinect, we are seeing the impact when people can interact with technology in the same ways that they interact with each other.”

Only time will tell if NUI technologies will be the next computing paradigm, but based on my own experiments with the Kinect, I see considerable opportunities for radically new approaches to theory and practice associated with both rhetoric and writing. And if NUI does eclipse GUI, leading to new conventions for human-computer interaction, some of us will turn to Selfe and Selfe’s article because of its long-valued contribution to a socio-political approach to interface studies.

Exploring the Kinect’s Potential

My own work with the Kinect began in early 2011 with two colleagues in the College of Design at NC State. We developed an interactive art project that enabled users to “write” graffiti on a screen projected on the side of a building. The “Interactive Graffiti Project” was based on the skeletal or joint data streaming from the system. Arguably, the most impressive part of the Kinect is that it can identify human figures from everything else in its viewing frame, break each of them down into a set of twenty joints, and track all twenty joints continuously at 30 frames-per-second. This capability had not been so easily available for research before the Kinect was released.

For a programmer, access to the skeletal data means that you have access to twenty three-dimensional data points representative of a user’s position and movements around and in front of the sensor, in “real-time.” This is the data stream on which most of the commercial games for Kinect are based: the ability to make an on-screen avatar kick balls, jump around, or dance is based on your redefinition as a surface comprising twenty points.

Although the Kinect defines a jointed body in conventional terms, and many of the on-screen interfaces reinforce that perspective to-date, there’s no limit to the ways in which those data points can be redefined computationally. For example, you could redeploy skeletal data from a user as points (or folds) comprising a novel, topological surface. Once we deterritorialize the origin of those points, we can experiment, developing new types of bodily gesture and movement contributing to a new canon of digital delivery. And once a user’s movements and position are redefined radically, the environmental feedback from the projected movements has the potential to transform how that user experiences herself, which can lead to new, counter-hegemonic experiences of self. In Materializing New Media, Anna Munster emphasizes the potential for developing new experiences of our bodies in our “encounters” with code when she writes,

Computers offer us multiplications and extensions of our bodily actions… These multiplications by no means provide seamless matches between body and code; the mismatch characteristics of divergent series triggers the extension of our corporeality out toward our informatic counterparts… It is this extensive vector that draws embodiment away from its historical capture within a notion that the body is a bounded interiority. (33)

Munster is not writing explicitly about politics in this excerpt, but her argument extends easily to it. If we equate her description of a bounded interiority with the socio-cultural constraints constitutive of person’s sense of self, her point is that mismatches between body and code, which might emerge from a novel redefinition of the skeletal data points, can lead to new experiences of our embodied selves. And if these mismatches are designed tactically, they can change the limits of the bounded interiorities interfacing with the code. Related to the trope of the ‘geography closest in,’ we might characterize this kind of NUI-based approach to tactical invention as a novel form of geopolitics born of an extensive vector that points beyond the centripetalism of hegemonic relations.

Depth is Flat

One of the other data streams on which creative work is now based is the depth data. At thirty frames-per-second, the Kinect sends a 640×480 frame or image of depth values representative of the depth of every object in the sensor’s viewing range. In other words, it streams a 640×480 matrix of depth values representative of an approximately 8×14’ space in front of the sensor.

In a project titled emBody(text) {, Kevin Brock and I used the Kinect’s depth data stream to develop an embodied, hypertextual environment. Our goal was to create an interactive experience that would be intuitive and compelling for an audience largely unfamiliar with the Kinect’s technical capabilities. In a way, emBody(text) { was our “Hello, World” program. Our digital writing project debuted in a group exhibit titled ID:ENTITY at Raleigh’s contemporary art museum, CAM Raleigh (see Figure 6 above), and then at the CHAT Festival at Duke University. A few months later, in collaboration with Rachel Bagby, a Twitter-based version of the project titled emBody(dekaaz) { was on display at CCCC 2012. For the version at CCCC 2012, the interface was based on the following characteristics:

• We imagined that the space in front of the Kinect was divided among ten distinct zones.
• We created ten equivalent zones or layers along the z-axis in the virtual, 3D space of the projected screen.
• We distributed each of the ten most recent tweets to #dekaaz in one of the ten virtual depth planes.
• We made it possible to “mash up” or combine parts of different tweets when participants interacted with two or more zones in front of the Kinect sensor.
• We developed a method of identifying which of the ten zones had the most activity at a given moment in order to shift perspective on the 3D space.

The effect of all of this was something like an embodied hypertext. On the second floor of the convention center at CCCC 2012, passive or active participation with emBody(dekaaz) { led to mash-ups of the three-stanza poems sent to #dekaaz.

As I reflect on my experiences with the Kinect’s depth data, it occurs to me that it is a “degree zero” for experimental work because the data is (in Deleuzian terms) an intensive form, pure potential. ²Whereas you have to “deconstruct” the implied (human) territory or ground of the skeletal data points in order to reach an ontologically flat plane, the depth data is an already-posthumanized space. Ontologically, it’s already flat. Working with the depth data puts one in the kind of uncountry that Victor Vitanza once characterized as wild and savage (53), and which, due to its savage nature, we might then value as a space in which the bricoleur practices a (computational) ‘science of the concrete.’³ The opportunities for developing new forms of interactions are considerable.

Conclusion

As Selfe and Selfe explained in their article, politics reside in the interfaces of our techno-mediated worlds, and since the interface of a NUI environment has shifted, so should the focus of our attention. In Samantha Swartz’s “hack” of the Kinect (see Figure 7 above), which is based on the depth data, we are presented with a complex, ever-changing series of curving, pointilized surfaces. During the film, we realize that the surfaces are two participants; at one point, we can see them holding hands. But the identifiable hands and arms soon melt away into an intensive swirl of movement. The depiction of those bodies—the projected feedback—is based on the interface that Swartz designed. Swartz’s hack is an example of the ways in which the mismatch about which Munster wrote can lead to new relations between bodies and code.

As a field, we have an opportunity to explore new forms of digital writing and rhetoric by critically and creatively engaging with the data streams that represent who we are and what we can be. In immersive NUI computing environments, the politics of the interface are derived from the ways in which we experience the feedback of our selves. In fact, the interface itself is a derivative of a dynamic equation comprising bodies and code; politics associated with interface design is a kind of integral calculus, as we can see in Swartz’s experiment. If we contribute to the development of new interfaces, thereby changing some of the ways in which we experience the data streams that would otherwise serve convention, we can accomplish two important things simultaneously. We can extend a critical legacy toward interface studies inaugurated by Selfe and Selfe, and, perhaps most importantly, we can elevate the role and value of invention serving those ends in today’s brave new NUI worlds.

Endnotes

1. In his article, “On Divides and Interfaces,” Jeffrey Grabill hazards that Patricia Sullivan “provides perhaps the first mention of interface design in Computers and Composition” (469). I hazard to agree with him. return
2. In Writing Degree Zero, Roland Barthes characterizes poetic language unrestricted by convention and acculturation as “reduced to a sort of zero degree, pregnant with all past and future specifications” (Barthes 48). My Barthean allusion is meant to characterize the depth data as similarly pregnant with potential. The allusion to Deleuze’s concept of intensive qualities parallels the reason for the Barthean one. return
3. A “science of the concrete” and bricolage are allusions to Claude Levi-Strauss’ opening chapter, “The Science of the Concrete,” in Savage Mind. return

Works Cited

• Bagby, Rachel. “Dekaaz: Word Jazz for the Mind.” Rachel Bagby: Bless Your Voice. Be the Song. 2012. Web.
• Barthes, Roland. Writing Degree Zero. Trans. Annette Lavers and Colin Smith. New York: Farrar, Straus & Giroux, 1977. Print.
• Brooke, Collin Gifford. Lingua Fracta: Toward a Rhetoric of New Media. Cresskill, NJ: Hampton, 2009. Print.
• Buck, Amber M. “The Invisible Interface: MS Word in the Writing Center.” Computers and Composition 25.4 (2008): 396-415. Print.
• Carnegie, Teena A. M. “Interface as Exordium: The Rhetoric of Interactivity.” Computers and Composition 26.3 (2009): 164-173. Print.
• Carpenter, Rick. “Boundary Negotiations: Electronic Environments as Interface.” Computers and Composition 26.3 (2009): 138-148. Print.
• De los Reyes, August. “Predicting the Past.” Web Directions South. Sydney Convention Center. 25 Sept. 2008. Web.
• Fagerjord, Anders. “Between Place and Interface: Designing Situated Sound for the iPhone.” Computers and Composition 28.3 (2011): 255-263. Print.
• Gates, Bill. “The Power of the Natural User Interface.” The Gates Notes. 28 Oct. 2011. Web.
• Grabill, Jeffrey T. “On Divides and Interfaces: Access, Class, and Computers.” Computers and Composition 20.4 (2003): 455-472. Print.
• Haas, Mark, and Clinton Gardner. “MOO in Your Face: Researching, Designing, and Programming a User-Friendly Interface.” Computers and Composition 16.3 (1999): 341-358. Print.
• Hayles, N. Katherine. “Deeper into the Machine: Learning to Speak Digital.” Computers and Composition 19.4 (2002): 371-386. Print.
• Levi-Strauss, Claude. The Savage Mind. Chicago: U of Chicago P, 1968. Print.
• Mardsjo, Karin. “Interfacing Technology.” Computers and Composition 13.3 (1996): 303-316. Print.
• Mundie, Craig. “Computing Naturally.” Microsoft News Center. 3 Mar. 2010. Web.
• Munster, Anna. Materializing New Media: Embodiment in Information Aesthetics. Hanover, NH: Dartmouth College P, 2006. Print.
• Ramsay, Stephen. “On Building.” Stephen Ramsey Blog. N.d. Web.
• Rieder, David M., and Kevin Brock. “emBody(text) {.” ID:ENTITY. Self: Perception + Reality. 3rd Emerging Artists Series. CAM Raleigh. 18 Nov. 2011-13 Feb. 2012. Digital Interactive.
• Rieder, David M., Kevin Brock, and Rachel Bagby. “emBody(dekaaz) {.” Conference on College Composition and Communication. St. Louis Convention Center, St. Louis. 22 Mar. 2012. Digital Interactive.
• Rosinski, Paula, and Megan Squire. “Strange Bedfellows: Human-Computer Interaction, Interface Design, and Composition Pedagogy.” Computers and Composition 26.3 (2009): 149-163. Print.
• Selfe, Cynthia, and Richard Selfe. “The Politics of the Interface: Power and Its Exercise in Electronic Contact Zones.” College Composition and Communication 45.4 (1994): 480-504. Print.
• Skjulstad, Synne, and Andrew Morrison. “Movement in the Interface.” Computers and Composition 22.4 (2005): 413-433. Print.
• Spinuzzi, Clay. “Light Green Doesn’t Mean Hydrology: Toward a Visual-Rhetorical Framework for Interface Design.” Computers and Composition 18.1 (2001): 39-53. Print.
• Sullivan, Patricia. “Human-Computer Interaction Perspectives on Word-Processing Issues.” Computers and Composition 6.3 (1989): 11-33. Print.
• Vitanza, Victor J. Negation, Subjectivity, and the History of Rhetoric. Albany: SUNY P, 1997. Print.
• Wigdor, Daniel, and Dennis Wixon. Brave NUI World: Designing Natural User Interfaces for Touch and Gesture. New York: Morgan Kaufman, 2011. Print.