Augmented Reality

Securing Augmented Reality Output

A year ago, Pokemon Go became immensely popular as players explored their surroundings for Pokemon in the smartphone-based augmented reality (AR) app. This hyper-popular game, which barely scratched the surface of AR’s potential, led to increased interest in the technology. The AR industry is expected to grow to $100 billion by 2020, and with increasing interest in AR automotive windshields and head-mounted displays (HMDs), we could soon be able to experience immersive AR environments like the one depicted by designer and film-maker Keiichi Matsuda in Hyper Reality.

Hyper Reality Screenshot
But what would happen if a pop-up ad covers your game, causing you to lose? Or if, while driving, an AR object obscures a pedestrian?

These are the types of situations researchers consider in a new paper, Securing Augmented Reality Output. In the paper, Lab student Kiron Lebeck, along with CSE undergraduate Kimberly Ruth, Lab Affiliate Faculty Franzi Roesner, and Lab Co-Director Yoshi Kohno address how to defend against buggy or malicious AR software that may unintentionally or inadvertently augment a user’s view of the world in undesirable or harmful ways. They ask, how can we enable the operating system of an AR platform to play a role in mitigating these kinds of risks? To address this issue, the team designed Arya, an AR platform that controls output through a designated policy framework, drawing policy conditions from a range of sources including the Microsoft HoloLens development guidelines and the National Highway Traffic and Safety Administration (NHTSA)’s driver distraction guidelines.

Arya Driving Scenario
By identifying specific “if-then” policy statements, this policy framework allows the Arya platform to apply a specific mechanism, or action, to virtual objects that violate a condition. In a simulated driving experience, for example, Arya makes transparent pop-up ads and notifications that could distract the driver by applying a specified action, in this case transparency, to objects that violate the specific policies:
• Don’t obscure pedestrians,
• Only allow ads to appear on billboards, and
• Don’t distract the user while driving.

By implementing Arya in a prototype AR operating system, the team was able to prevent undesirable behavior in case studies of three environments, including a simulated driving scenario. Additionally, performance overhead of policy enforcement is acceptable even in the un-optimized prototype. The team, among the first to raise AR output security issues, demonstrated the feasibility of implementing a policy framework to address AR output security risks, while also surfacing lessons and directions for future efforts in the AR security space.

To read more, see Securing Augmented Reality Output.

Director Calo Testifies on Augmented Reality before U.S. Senate

Lab Faculty Co-Director, Ryan Calo, testified before the U.S. Senate Committee on Commerce, Science, and Transportation at a hearing exploring augmented reality. Watch the hearing here and read his testimony below.

“Chairman Thune, Ranking Member Nelson, and Members of the Committee, thank you for the opportunity to discuss the promise and perils of augmented reality.

Augmented reality (AR) refers to a mobile or embedded technology that senses, processes, and outputs data in real time, recognizes and tracks real-world objects, and provides contextual information by supplementing—or in some cases, replacing—human senses. AR differs from so-called virtual reality in that AR users continue to experience most of their physical environment. AR has many positive applications, from training tomorrow’s workforce, to empowering people with disabilities. But the technology also raises novel or acute policy concerns that companies and policymakers must address if AR is to be widely adopted and positively affect American society.

The UW Tech Policy Lab is a unique, interdisciplinary research unit at the University of Washington that aims to help policymakers develop wise and inclusive technology policy. We have studied AR and its impact on diverse populations and discuss our findings in detail in the appended whitepaper Augmented Reality: A Technology and Policy Primer.

Our research suggests that AR raises a variety of question of law and policy, including around privacy, free speech, and novel forms of distraction and discrimination. For example: Will the constant recording of a user’s environment give hackers, companies, and government unparalleled access to the bedroom, the boardroom, and other private spaces? Could the superimposition of information over reality render the AR user vulnerable or unsafe? And are there situations—such as job interviews—where knowing everything about an individual could result in discrimination or subject the AR user to legal liability? Industry must design AR products with these and many other questions in mind.

Thank you again for the interest in our research and the opportunity to appear before the Committee. I look forward to your questions.”

PokemonGO and Policy for Augmented Reality Applications

With widespread adoption, PokemonGO has brought the novel policy considerations of augmented reality to a wide audience. Over the last week, members of the Lab have highlighted some of these issues. Co-Director Calo,  noted the novel nature of a game that requires players to physically travel and potentially actionable nuisance created by the developers (Verge). In an article in New Scientist, Emily McReynolds highlighted the benefits of including a diverse set of stakeholders in the design of these applications.

In the Tech Policy Lab’s Augmented Reality Law and Policy Primer we provide a preview of the policy implications of this developing technology and make conditional  recommendations. Our key findings included:

1. AR exists in a variety of configurations, but in general, AR is a mobile or embedded technology that senses, processes, and outputs data in real-time, recognizes and tracks real-world objects, and provides contextual information by supplementing or replacing human senses.
2. AR systems will raise legal and policy issues in roughly two categories: collection and display. Issues tend to include privacy, free speech, and intellectual property as well as novel forms of distraction and discrimination.
3. We recommend that policymakers—broadly defined—engage in diverse stakeholder analysis, threat modeling, and risk assessment processes. We recommend that they pay particular attention to: a) the fact that adversaries succeed when systems fail to anticipate behaviors; and that, b) not all stakeholders experience AR the same way.
4. Architectural/design decisions—such as whether AR systems are open or closed, whether data is ephemeral or stored, where data is processed, and so on—will each have policy consequences that vary by stakeholder.

Augmented Reality Primer

This whitepaper—which grows out of research conducted across three units through the University of Washington’s interdisciplinary Tech Policy Lab—is aimed at identifying some of the major legal and policy issues AR may present as a novel technology, and outlines some conditional recommendations to help address those issues. Our key findings include:

1. AR exists in a variety of configurations, but in general, AR is a mobile or embedded technology that senses, processes, and outputs data in real-time, recognizes and tracks real-world objects, and provides contextual information by supplementing or replacing human senses.

2. AR systems will raise legal and policy issues in roughly two categories: collection and display. Issues tend to include privacy, free speech, and intellectual property as well as novel forms of distraction and discrimination.

3. We recommend that policymakers—broadly defined—engage in diverse stakeholder analysis, threat modeling, and risk assessment processes. We recommend that they pay particular attention to: a) the fact that adversaries succeed when systems fail to anticipate behaviors; and that, b) not all stakeholders experience AR the same way.

4. Architectural/design decisions—such as whether AR systems are open or closed, whether data is ephemeral or stored, where data is processed, and so on—will each have policy consequences that vary by stakeholder.

Augmented Reality – Technology & Policy Primer

This whitepaper is aimed at identifying some of the major legal and policy issues augmented reality (AR) may present as a novel technology, and outlines some conditional recommendations to help address those issues. Our key findings include:

1. AR exists in a variety of configurations, but in general, AR is a mobile or embedded technology that senses, processes, and outputs data in real-time, recognizes and tracks real-world objects, and provides contextual information by supplementing or replacing human senses.

2. AR systems will raise legal and policy issues in roughly two categories: collection and display. Issues tend to include privacy, free speech, and intellectual property as well as novel forms of distraction and discrimination.

3. We recommend that policymakers—broadly defined—engage in diverse stakeholder analysis, threat modeling, and risk assessment processes. We recommend that they pay particular attention to: a) the fact that adversaries succeed when systems fail to anticipate behaviors; and that, b) not all stakeholders experience AR the same way.

4. Architectural/design decisions—such as whether AR systems are open or closed, whether data is ephemeral or stored, where data is processed, and so on—will each have policy consequences that vary by stakeholder.