In AI We Trust — Too Much?

I’ve been thinking about technology and trust for much of my career. Here’s one example: Back in 2011, my research focused on the intersection of human trust, robots, and highly critical, time-sensitive scenarios. The research team I was on centered on emergency evacuations and would run scenarios where people were in a room when the fire alarms went off. As people spilled into hallways filled with smoke, we had a robot guide them to the exit. For some of the participants, this same robot had led them to a wrong room earlier.

And we intentionally made the robot head away from the exit.

People could see the exit signs. They could see the smoke. And what we found over and over again was that, even when the robot exhibited bad behavior and led them away from the exit, people would follow the robot.

People over-trust technology because they believe it works most of the time, we found.

When technology doesn’t work, opinions can sometimes swing wildly in the opposite direction toward distrust or under-trusting. People can overreact. But that’s rare — people don’t generally under-trust. For instance, after a plane crash, nobody says, “We need a ban to make sure no one ever flies again.” The bigger challenge in thinking about technology and trust is that we over-trust and leave ourselves vulnerable to technology’s mistakes.

I see several needs for this moment in technology’s evolution. The first — and this probably requires regulation — is that technology companies, particularly those in artificial intelligence and generative AI, need to figure out ways to blend human emotional quotient (EQ) with technology to give people cues on when to second-guess such tools. This will help ensure that customer trust in the technology is justified. Second, users of these technologies have to train themselves to be continually on the alert.

Surface the Risks

In January, I traveled to Davos, Switzerland, for the 54th Annual Meeting of the World Economic Forum. I participated in two panel discussions about jobs of the future — one called “About Robo-Allies” and the other “How to Trust Technology.” During the second, which was presented as a town hall discussion and Q&A, I asked the audience members how many of them had used ChatGPT or some equivalent generative AI technology. Every hand went up. I asked how many had used it to actually do a job or do some kind of work, and it was almost (but not quite) 100%.

By now, many people, especially those in the corporate setting, have played with ChatGPT since it debuted in late 2022 to experiment with how it might write a marketing piece, research a topic, or develop code.

People use it even though the tool delivers mistakes. One lawyer was slammed by a judge after he submitted a brief to the court that contained legal citations ChatGPT had completely fabricated. Students who have turned in ChatGPT-generated essays have been caught because the papers were “really well-written wrong.” We know that generative AI tools are not perfect in their current iterations. More people are beginning to understand the risks.

What we haven’t yet figured out is how to address this as a society. Because generative AI is so useful, it is also valuable: When it’s right, it really does make our work lives better. But when it’s wrong and we aren’t using our human EQ to correct it, things can go badly quickly.

Build Trust Using Regulation and Skepticism

My philosophy on this topic is driven by the question of how to best blend human EQ into our AI tools. In some ways, the rush of so many players into the AI space does worry me. It’s similar to the days when electricity was born, when a lot of inventors were experimenting with incandescent lamps. We ended up with light bulbs literally exploding in people’s houses. They had indoor lighting, but there was a danger. And it took a while before the regulations and standards by UL (Underwriters Laboratory) and CE (Conformité Européenne) came about. Those rules said, “OK, if you are an inventor in this space, there are some rules you have to follow in order for you to release your inventions for public consumption. You have to have some certification. You have to have some validation.”

We don’t have that in AI. Basically anyone can participate and create a product. We have inventors who don’t know what they’re doing who are selling to companies and consumers who are too trusting: People see that the product has venture capital backing and say, “Yeah, let’s bring it in.” That’s a concern.

As a technology researcher and college dean, I also dabble a bit in policy with respect to AI and regulations by serving on the National AI Advisory Committee. I think policy will be critical to building trust. (I wrote about the possibility of regulations around AI for MIT SMR back in 2019.) Policies and regulations allow for equal footing by establishing expectations and ramifications if companies or other governments violate them. Now, some companies will disregard the policies and just pay the fines — but there still is some concept of a consequence.

Right now, there’s a lot of activity around regulations. There’s the proposed EU AI Act, draft AI guidelines released by the Japanese government, and slightly different proposals in the United States, including President Biden’s AI executive order. There’s state-specific activity, too: Last fall, California’s governor called for a study on the development, use, and risks of AI within the state, with the goal of establishing “a deliberate and responsible process for evaluation and deployment of AI within state government.”

At the same time, human users of AI need to be more attentive to the fact that it can deliver wildly incorrect results at almost any time. At the Davos town hall on trust and technology, I shared the stage with Mustafa Suleyman, then the CEO of consumer software company Inflection AI and a cofounder of DeepMind, an AI company Google acquired in 2014. (He has since become CEO of Microsoft AI.) As a part of Google, Suleyman was responsible for integrating the company’s technology across a range of its products.

“Our past mental model of default trusting technology doesn’t really apply” with large language models (LLMs) like ChatGPT, he acknowledged. “I think, at this moment, we have to be incredibly critical, skeptical, doubtful, and ask tough questions of our technology.”

Suleyman and I talked about specific ideas to help build trust with consumer-facing applications of AI. One was the most basic: for developers to use benchmarks that, as Suleyman put it, “evaluate the freshness and the factuality of models” — in other words, to make the models more truthful. Suleyman said that LLMs will get better fairly quickly and predicted that factual accuracies are going to go from “today, when they’re hovering around sort of 80%, 85%, all the way up to 99% to 99.9%” in the next three years.

Although I do believe that these models will continue to become more accurate, I also believe that factual correctness will continue to be a moving target as we challenge the technology to do more with more information.

Another approach to building trust: Get the models to acknowledge when they don’t know and to communicate that to users — to say, “I’m not quite sure on this,” or, “I can’t answer this right.” Suleyman speculated that if a model “was consistently accurate with respect to its own prediction of its own accuracy, that’s a kind of different way of solving the hallucinations problem.” Specifically, “if it says, ‘Well, no, I can’t write you that email or generate an image because I can only do X,’ that’s increasing your trust that it knows what it doesn’t know,” he said.

In fact, I recommend going a step further: forcing the AI to reduce its level of service if the user ignores the AI’s acknowledgment of its limitations.

I think we need to remember that fire evacuation experiment in 2011. Just as we don’t want people in a smoke-filled hallway following a robot away from the exit door, we don’t want users to have blind trust in what AI is presenting to them.

With some kinds of technologies, like network devices, data systems, and cloud services, there is a move toward zero trust because people assume that they’re absolutely going to get hacked. They assume that there are bad actors, so they design processes and frameworks to deal with that.

In AI, there’s really no standard for designing our interactions with these systems under the assumption that the AI is bad. We, therefore, must think about how we design our systems so that if we assume malicious intent, we can figure out what to do on the human side or on the hardware side to counter that.

Technologists aren’t trained to be social scientists or historians. We’re in this field because we love it, and we’re typically positive about technology because it’s our field. That’s a problem: We’re not good at building bridges with others who can translate what we see as positives and what we know are some of the negatives as well.

There is much room for improvement in making sure that people not only understand technology and the opportunities it provides but also the risks it creates. With new regulations, more accurate systems, more honesty about whether an answer is a guess, and increased diligence by users, this can happen.

“The MIT Sloan Management Review is a research-based magazine and digital platform for business executives published at the MIT Sloan School of Management.”

Post details > link to site