In 2017, Savannah Thais attended the NeurIPS machine learning conference in Long Beach, California, hoping to learn more about techniques she could use in her doctoral work on electronic identification . Instead, she returned home to Yale with a transformed worldview.
At NeurIPS, she had listened to a talk by artificial intelligence researcher Kate Crawford, who discussed bias in machine learning algorithms. She mentioned a new study showing that facial recognition technology, which uses machine learning, detected gender and racial bias in its data set: women of color were 32% more likely to be misclassified by the technology than white men.
The study, published in the form of The master’s thesis by Joy Adowaa Buolamwini, has become a benchmark in the world of machine learning, revealing how seemingly objective algorithms can make mistakes based on incomplete data sets. And for Thais, who had been introduced to machine learning through physics, it was a watershed moment.
“I didn’t even know about it before,” says Thais, now a research associate at Columbia University’s Data Science Institute. “I didn’t know these were technology issues, that these things were happening.”
After completing her doctorate, Thais turned to studying the ethical implications of artificial intelligence in science and society. Such work often focuses on direct impacts on people, which may seem entirely distinct from algorithms designed, for example, to identify the signature of a Higgs boson when particles collide against a mountain of noise.
But these questions are also linked to physics research. Algorithmic biases can influence physics results, particularly when machine learning methods are used inappropriately.
And work done for the purpose of physics is unlikely to remain in the field of physics. By advancing machine learning technology in the service of science, physicists are also helping to improve it in other areas. “When you’re in a more scientific context and you’re thinking, ‘Oh, we’re building these models to help us do better research in physics,’ that’s quite removed from any societal implications,” Thais says. “But it’s all really part of the same ecosystem.”
Trust your models
In traditional computer models, a human tells the program each parameter it needs to make a decision. For example, the information that a proton is lighter than a neutron can help a program distinguish between the two types of particles.
Machine learning algorithms, on the other hand, are programmed to learn their own parameters from the data fed to them. An algorithm can offer millions of parameters, each with its own “phase space”, the set of all possible iterations of that parameter.
Algorithms do not treat each phase space the same. They weight them differently based on their usefulness for the task the algorithm is trying to accomplish. Since this weighting is not decided directly by humans, it is easy to imagine that making decisions by algorithm could be a way to eliminate human bias. But humans always add their input to the system, in the form of the dataset they give the algorithm to train on.
In his thesis, Buolamwini analyzed an algorithm creating facial recognition parameters based on a dataset consisting largely of photos of white people, mostly men. Because the algorithm had a variety of examples of white men, it was able to come up with a good rubric to differentiate them. Because there were fewer examples of people of other ethnicities and genders, he did a worse job of differentiating between them.
Facial recognition technology can be used in a variety of ways. For example, facial recognition technology can be used to verify a person’s identity; many people use it daily to unlock their smart phone. Buolamwini gives other examples in his thesis, including “the development of more empathetic human-machine interactions, health monitoring, and locating missing people or dangerous criminals.”
When facial recognition technology is used in these contexts, its inability to work equally well for everyone can have a range of consequences, from the frustration of being denied access to a convenience service, to the danger of being misdiagnosed in a medical setting, to the threat of being falsely identified and arrested. “Characterizing how your model works in phase space is both a scientific and ethical question,” says Thais.
Cosmologist Brian Nord has been thinking about this for years. He began using machine learning in his work in 2016, when he and his colleagues realized that machine learning models could classify objects observed by telescopes. Nord was particularly interested in algorithms that could decode the strangeness of light that bends around celestial bodies, a phenomenon known as gravitational lensing. Because these models excel at classifying elements based on existing data, they can identify stars and galaxies in images much better than a human.
But other uses of machine learning for physics are much less reliable, says Nord, a scientist in Fermilab’s AI Projects Office and Cosmic Physics Center. While a traditional program has a limited number of parameters that physicists can manually modify to obtain correct results, a machine learning algorithm uses millions of parameters that often do not correspond to actual physical characteristics, making it impossible to physicists to correct them. “There is no robust way to interpret errors resulting from an AI method that we can look at in terms of how we think about statistics in physics,” says Nord. “It’s not something that still exists.”
If physicists are not aware of these problems, they may use models for purposes beyond their capabilities, which could compromise their results.
Nord is working to develop machine learning capabilities to facilitate all stages of the scientific process, from identifying testable hypotheses and improving telescope design to simulating data. He envisions a not-too-distant future where the physics community can conceive, design and execute large-scale projects in much less time than the decades such experiments currently take.
But Nord is also acutely aware of the potential pitfalls of advancing machine learning technology. The image recognition algorithm that allows a cosmologist to distinguish a cluster of galaxies from a black hole, Nord points out, is the same technology that can be used to identify a face in a crowd.
“If I’m using these tools to do science and I want to fundamentally improve them to do my science, it’s very likely that I’ll improve them in other places where they’re applied,” Nord says. “I’m basically building technology to monitor myself.”
Responsibilities and Opportunities
Physics is the source of one of the most famous scientific ethical dilemmas: the creation of nuclear weapons. Since the days of the Manhattan Project – the government research program aimed at producing the first atomic bomb – scientists have debated the extent to which their involvement in the science behind these weapons amounts to responsibility for their use.
In his 1995 Nobel Peace Prize acceptance conference, physicist Joseph Rotblat, who left the Manhattan Project, appealed directly to the ethical sensitivity of scientists. “At a time when science plays such a powerful role in the life of society, when the fate of all humanity may depend on the results of scientific research, it is incumbent on all scientists to be fully aware of this role and to carry out actions themselves accordingly,” Rotblat said.
He noted that “doing fundamental work, pushing the frontiers of knowledge… often you do it without thinking much about the impact of your work on society.”
Thais says she sees the same pattern repeating itself today among physicists working on artificial intelligence. There is rarely a moment in the scientific process when a physicist can pause to consider their work in a broader context.
As physicists learn more about machine learning alongside physics, they should also be exposed to ethical frameworks, Thais says. This can happen at conferences and workshops as well as in online training materials.
Last summer, SLAC National Accelerator Laboratory researcher Kazuhiro Terao organized the 51st SLAC Summer Institute, focused on the theme “Artificial Intelligence in Fundamental Physics.” He has invited speakers on topics such as computer vision, anomaly detection and symmetries. He also asked Thais to address ethics.
“It’s important for us to understand not only the hype around AI, but also the types of things it can do and what types of things it can be biased toward,” says Terao.
Ethical research into artificial intelligence can teach physicists to think in ways that are also useful for physics, Terao says. For example, learning about the biases of machine learning systems can encourage healthy scientific skepticism about what these systems can actually do.
Ethics research also offers physicists the opportunity to improve the use of machine learning in society as a whole. Physicists can use their technical expertise to inform citizens and policymakers about the technology, its uses and its implications, says Nord.
And physicists have a unique opportunity to improve the science of machine learning itself, says Thais. This is because physical data, unlike facial recognition data, is highly controlled – and there is a lot of it. Physicists know what types of biases exist in their experiments and how to quantify them. This makes physics as a field a perfect “sandbox” for learning how to build models that avoid bias, Thais says.
But this can only happen if physicists integrate ethics into their thinking. “We need to think about these questions,” says Thais. “We can’t escape the conversation.”
