To Discover Breakthrough Ideas, Look to the Outsiders

Investors and business leaders are always on the lookout for the next big thing, the paradigm shift that will upend industries and change the world. The hope is to get in early and ride the wave — or at least avoid getting flattened by it. Yet where should they be looking?

“There’s a huge literature on innovation,” says Amir Goldberg, an associate professor of organizational behavior at Stanford Graduate School of Business. “One of the perennial debates is whether new ideas are more likely to come from established players or new entrants in a field — those in the core or on the periphery, so to speak.”

There are plenty of examples of each. “In American culture, we celebrate the plucky outsider, the maverick who bucks conventional thinking,” Goldberg says. But many game-changing innovations come from dominant firms, like the Apple iPhone or the invention of transformer architecture at Google, which revolutionized artificial intelligence.

The problem with such anecdotes, he says, is that they focus on the moment of creation, not conception. “We see the tangible outcome, but who knows where the original spark came from?” Anyway, he adds, innovation is often more of an organic process without any specific eureka moment or single inventor.

But a few years ago, Goldberg and Paul Vicinanza, PhD ’23, along with Sameer Srivastava, a professor at UC Berkeley’s Haas School of Business, had a breakthrough idea of their own. Seeing how emerging AI programs could digest vast amounts of documents, they wondered if they could deploy these tools to detect early signs of new thinking in language.

One advantage of this approach was that they could study innovations that went beyond the usual focus on new technology. “Some of the most impactful innovations are intangible things like business strategies,” says Vicinanza, now a postdoctoral researcher at the University of California, Berkeley. “Think of how Amazon revolutionized commerce.”

Since the method was quite general, they broadened their study to look not just at business but also law and politics. They built a dataset of 5 million congressional floor speeches starting in 1961, collected 4.2 million court decisions over 50 years, and tapped the transcripts of 108,000 quarterly earnings calls beginning in 2006.

Then, they tasked a deep learning model known as BERT with assessing every sentence in those millions of texts and rating the speakers in relative degrees of prescience. They chose the term “prescience” with care. “Novelty isn’t enough,” Vicinanza says. “Anyone can say something new — I could start talking gibberish right now, and that would be novel. But it would only be prescient if it became common in the future.”

The results were striking: In all three fields, they found that prescient ideas were much more likely to emerge from the periphery than the core. “In studies of creativity, people tend to focus on brilliant individuals,” Goldberg says. “But unless you think there are more geniuses on the margins, this suggests that where you sit matters at least as much as who you are.”

Predicting Prescience

Tracing ideas to their origin has long been the province of historians working subjectively from their reading of old texts. With this project, Goldberg, Vicinanza, and Srivastava have developed a way of quantifying that attribution process — and doing it at scale.

Like other language models (think ChatGPT), BERT is a “prediction machine,” Goldberg says. It generates text by predicting missing words in a document, based on the corpus of texts it’s trained on. The researchers realized they could run it in reverse, taking a real historical sentence and seeing how predictable it would have been at the time.

If a statement reflected new thinking, the model would be “perplexed.” If the thought would be less perplexing in a future context — meaning it had gained currency — it was marked as prescient. “The assumption is that the model represents the general imagination of the time,” Goldberg says. “So if a statement perplexes the model, it would have perplexed your peers.”

They didn’t know what the results would show. “There are a lot of reasons to expect innovation to come from the center,” Goldberg says. “A dominant firm has more resources to invest in blue-sky thinking, it can hire the best people, and if all else fails, it has the power to shape the world to fit its ideas” — a sort of brute-force way of accomplishing prescience. (Up to a point — Mark Zuckerberg has yet to convince many that the metaverse is the future.)

On the other hand, outsiders haven’t been socialized to think in the same way that entrenched elites do. Plus, it’s in their interest to disrupt the status quo. Thomas Kuhn’s classic 1962 book, The Structure of Scientific Revolutions, argues that the establishment resists paradigm shifts because they undermine the in-group’s status.

Indeed, Goldberg’s team consistently found that prescient thinking was much more likely to emerge from the periphery of a field, and that held for business, politics, and law. In the latter realm, for instance, highly prescient decisions were 22 times more likely to come from state appeals courts than the U.S. Supreme Court.

“When I showed the results to legal scholars, they were shocked,” Goldberg says. “People tend to think of the Supreme Court as a sort of revolutionary force, with the most original thinkers, but we find the exact opposite: They’re not initiating change in their rulings; they’re codifying change that’s already emerged. The real creativity is happening in the lower courts.”

Vicinanza notes that we shouldn’t conflate prescience with progress. The model’s review of the congressional record tagged former representative and House Speaker Newt Gingrich for his prescience. In the late 1980s and early ’90s, his extreme partisanship and obstructionist tactics were unusual, but they would set the tone for today’s politics.

Another was John Stennis, a Democratic senator from Mississippi who opposed every civil rights bill of the 1960s. Instead of publicly defending segregation, he justified his stance with calls for limited government and individual freedom. His argument that policy should not be based on race anticipated campaigns against affirmative action on the principle of “color-blindness.”

Look to the Little Guys

“We’re not saying innovation always comes from the periphery, only that it’s more likely,” Goldberg notes. This idea, already widely accepted in business, is backed up by the team’s finding that prescient thinking was eight times more likely to emerge from the smallest companies than the largest.

“Startups don’t have a lot to lose,” Vicinanza says. “They can try something wild and crazy, and if it doesn’t work out, no big deal. The cost of pivoting and going after a different customer segment is low — whereas if a big established company tries to radically reshape its business model, there’s probably going to be a lot of pain.”

This is why it’s common for big companies to acquire startups. “What they’re essentially doing is buying new ideas,” Vicinanza says.

However, Vicinanza also says that big tech companies have done a better job of maintaining a startup ethos than traditional manufacturing companies. Sometimes they’ve done this by maintaining separate organizations for research — he points to the example of Google Brain, now part of Google DeepMind, which produced many important AI innovations.

Putting this study in a larger context, it’s intriguing to think about what it means for societies as a whole. “One of the enduring questions in human history is why certain time periods or certain societies are more dynamic and others are more stagnant,” Goldberg says. “Our research doesn’t address that question, but it suggests that an openness to outsiders and welcoming a diversity of voices is a good place to start.”