Artificial Intelligence

New Research in Adversarial Machine Learning

Last fall, a team of researchers with the Lab’s Ivan Evtimov, Earlence Fernandes, and Co-Director Yoshi Kohno shared research on ArXiv showing that malicious alterations to real world objects could cause devices to “misread” the image. Specifically, the team tricked an object classifier, like those present in self-driving cars, into misidentifying a stop sign as a speed limit sign.

Now, the team of researchers from Samsung Research America, Stanford University, Stony Brook University, University of California Berkeley, University of Michigan, and University of Washington have presented two papers updating this research – one at Computer Vision and Pattern Recognition (CVPR) 2018 and another at the 12th USENIX Workshop on Offensive Technologies (WOOT) 2018.

Robust Physical-World Attacks on Deep Learning Visual Classification
At CVPR 2018, the team presented an updated version of the paper we shared last fall in “Robust Physical-World Attacks on Deep Learning Visual Classification”:

Recent studies show that the state-of-the-art deep neural networks (DNNs) are vulnerable to adversarial examples, resulting from small-magnitude perturbations added to the input. Given that that emerging physical systems are using DNNs in safety-critical situations, adversarial examples could mislead these systems and cause dangerous situations. Therefore, understanding adversarial examples in the physical world is an important step towards developing resilient learning algorithms. We propose a general attack algorithm, Robust Physical Perturbations (RP2), to generate robust visual adversarial perturbations under different physical conditions. Using the real-world case of road sign classification, we show that adversarial examples generated using RP2 achieve high targeted misclassification rates against standard-architecture road sign classifiers in the physical world under various environmental conditions, including viewpoints. Due to the current lack of a standardized testing method, we propose a two-stage evaluation methodology for robust physical adversarial examples consisting of lab and field tests. Using this methodology, we evaluate the efficacy of physical adversarial manipulations on real objects. With a perturbation in the form of only black and white stickers, we attack a real stop sign, causing targeted misclassification in 100% of the images obtained in lab settings, and in 84.8% of the captured video frames obtained on a moving vehicle (field test) for the target classifier.

Robust Physical-World Attacks on Deep Learning Visual Classification. Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, Dawn Song. Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT (supersedes arXiv:1707.08945), June 2018.

Physical Adversarial Examples for Object Detectors
At WOOT 18, the researchers presented new research in their paper “Physical Adversarial Examples for Object Detectors.” In this paper, they expand on their previous work by examining the possibility of fooling object detection modules by real-world alterations to objects, a broader class of algorithms than their previous work examined.

Deep neural networks (DNNs) are vulnerable to adversarial examples—maliciously crafted inputs that cause DNNs to make incorrect predictions. Recent work has shown that these attacks generalize to the physical domain, to create perturbations on physical objects that fool image classifiers under a variety of real-world conditions. Such attacks pose a risk to deep learning models used in safety-critical cyber-physical systems.

In this work, we extend physical attacks to more challenging object detection models, a broader class of deep learning algorithms widely used to detect and label multiple objects within a scene. Improving upon a previous physical attack on image classifiers, we create perturbed physical objects that are either ignored or mislabeled by object detection models. We implement a Disappearance Attack, in which we cause a Stop sign to “disappear” according to the detector—either by covering the sign with an adversarial Stop sign poster, or by adding adversarial stickers onto the sign. In a video recorded in a controlled lab environment, the state-of-the-art YOLO v2 detector failed to recognize these adversarial Stop signs in over 85% of the video frames. In an outdoor experiment, YOLO was fooled by the poster and sticker attacks in 72.5% and 63.5% of the video frames respectively. We also use Faster R-CNN, a different object detection model, to demonstrate the transferability of our adversarial perturbations. The created poster perturbation is able to fool Faster R-CNN in 85.9% of the video frames in a controlled lab environment, and 40.2% of the video frames in an outdoor environment. Finally, we present preliminary results with a new Creation Attack, wherein innocuous physical stickers fool a model into detecting nonexistent objects.

Physical Adversarial Examples for Object Detectors. Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramer, Atul Prakash, Tadayoshi Kohno, Dawn Song. 12th USENIX Workshop on Offensive Technologies (WOOT 2018), Baltimore, MD (arXiv:1807.07769) (supersedes arXiv:1712.08062), August 2018

Robust Physical-World Attacks on Machine Learning Modules

Adv ML Website Carousel

Could graffiti convey a hidden message to your car? Or cause a robot to do something unexpected? Cars and robots, as well as other devices, are more frequently relying on images of their surroundings to make decisions. New research explores the possibility that malicious alterations to real world objects, like the road sign above, could cause these devices to “misread” the image and take a certain adverse action. The paper Robust Physical-World Attacks on Deep Learning Modules is by a research team spanning the University of Washington, including Ph.D. student Ivan Etimov, Tech Policy Lab postdoc Earlence Fernandes, and Co-Director Yoshi Kohno, along with Kevin Eykholt and Atul Prakash from the University of Michigan, Amir Rahmati from Stony Brook University, and from the University of California Berkeley Bo Li and Dawn Song.

To address this question, the researchers created an algorithm that could generate these alterations, a methodology to evaluate their effectiveness in fooling machine learning, and then applied both to the real world example of autonomous vehicles. They experimented to see whether physically altering an object, in this case a road sign, could cause the computer of an autonomous vehicle to classify it incorrectly. Autonomous vehicles learn to classify objects using machine learning, where the car’s computer “learns” what objects such as road signs, pedestrians, and other cars look like by being shown thousands of photos of each object. If you’re not familiar with machine learning, check out the Lab’s fun primer video “What is Machine Learning?” here. Current self driving car systems can include this type of camera sensor, as well as a variety of others such as lidar, radar, and GPS.

The researchers wanted to explore whether it’s possible to fool these machine learning “brains” by slightly altering images shown to the classifier, which identifies, in the case of autonomous vehicles, the different road signs seen by the car’s camera sensors. While previous research has focused on altering an image digitally and then feeding that digital image into a classifier, the research team wanted to see if it was possible to physically, rather than digitally, alter the content of the image to maliciously fool the classifier.

In order to generate ways to alter these road signs, the researchers applied their new algorithm that looked at what the trained classifier “knew” about road signs, and generated ways to alter the signs that would fool the classifier when used in the real world. The research focuses on two types of alterations generated by the algorithm:
• poster-printing attacks, where an attacker prints an actual-sized poster of a road sign that has subtle variations and pastes it over the real sign, and
• sticker attacks, where an attacker prints the generated sticker design and places it onto the existing road sign.

Poster Printing Attack                                                                    Sticker Attack

Following their proposed methodology, the researchers took photos of the signs from a range of physical conditions that mimic different positions under which a sensor might encounter the object, and then fed those images into a machine learning application, in this case a road sign classifier. When photos of the above stop signs taken from different angles and distances were fed into the researchers’ road sign classifier in lab testing, the classifier misread them as speed limit signs 100% of the time for the poster-printing attack, and 66% of the time for the sticker attack. Because these attacks mimic vandalism or street art, it can be difficult for a casual observer to identify the risk they could pose.

The researchers show that it is possible to generate real world alterations to objects that fool machine learning under a variety of conditions. They propose a new methodology for evaluating the effectiveness of these alterations under a range of diverse physical conditions that mimic those a sensor may encounter the object under in the real world. The researchers’ aim is to help improve the security of technology like autonomous vehicles in the future, by identifying security risks now. To read more, see the paper Robust Physical-World Attacks on Deep Learning Models as well as the FAQ .

Tech Policy Lab Joins Partnership on Artificial Intelligence

The Tech Policy Lab is delighted to be joining the Partnership on AI to Benefit People and Society, a non-profit organization charged with exploring and developing best practices for AI. The Lab, which aims to position policymakers, broadly defined, to make wiser and more inclusive tech policy, joins a diverse range of voices from academia, industry and non-profit organizations committed to collaboration and open dialogue on the opportunities and rising challenges around AI.

PAI Logo_Large_2

The Lab has worked to advance AI in the public interest since our inception, through conferences, workshops, and research, among other initiatives.  In 2015, we organized the fourth annual robotics law and policy conference, WeRobot. And in 2016, we co-organized the Obama White House’s inaugural public workshop on AI, focusing on legal and governance implications of AI.  Our research focuses on the policy implications of AI and includes studying AI-connected devices in the home.

We are planning many more research initiatives around AI, including AI-assisted decision-making, AI and cybersecurity, and AI and diversity. We will bring to our AI research our commitment to the inclusion of diverse perspectives in tech policy research and outcomes, including our Diverse Voices method (made available earlier this year through our How-To Guide) which engages diverse panels of “experiential” experts in short, targeted conversations around a technology to improve inclusivity in tech policy outcomes.

The Partnership on AI will be a great network and resource as we undertake this work. We look forward to collaborating with a diverse group of stakeholders from industry, academia, and policy around the Partnership on AI’s goals: to develop and share best practices, advance public understanding of AI, create a diverse network of experts around AI, and examine AI’s impact on people and society.

About the Tech Policy Lab

The Tech Policy Lab is a unique, interdisciplinary research unit at the University of Washington. The Lab’s mission is to position policymakers, broadly defined, to make wiser and more inclusive tech policy.  Situated within a globally renowned research university, the Tech Policy Lab is committed to advancing artificial intelligence in the public interest through research, analysis, and education and outreach. To learn more about the Lab’s cutting edge research, thought leadership, and education initiatives, go to

About the Partnership on AI

The Partnership on AI to Benefit People and Society (Partnership on AI) is a not-for-profit organization, founded by Amazon, Apple, Google/DeepMind, Facebook, IBM and Microsoft.  Our goals are to study and formulate best practices on the development, testing, and fielding of AI technologies, advancing the public’s understanding of AI, to serve as an open platform for discussion and engagement about AI and its influences on people and society and identify and foster aspirational efforts in AI for socially beneficial purposes. We actively designed the Partnership on AI to bring together a diverse range of voices from for-profit and non-profit, all of whom share our belief in the tenets and are committed to collaboration and open dialogue on the many opportunities and rising challenges around AI. For the full list of founding members and partners, go to