How Amazon’s algorithm allows extremist merchandise to spread

“It’s a radical decision from them,” said Jieun Shin, a professor in the department of telecommunications at the University of Florida. While Facebook and Twitter, she noted, have made some efforts to cut down on potentially dangerous content, including through fact-check labels and — in Facebook’s case — the formation of an advisory board, Amazon had not taken any such steps until now.

Even though Amazon is not purely a content company the way Facebook is, the more active role taken by social platforms — plus Shopify’s decision to suspend Donald Trump’s merch store — likely played into Amazon’s pivot. “All the major social media companies, commerce platforms at this point, are taking these same steps,” said Chelsea Gross, an analyst at Gartner who studies retail. “It’s in their interest to make sure they’re not an outlier.”

Given that Amazon has largely avoided taking down extremist, anti-vaccine and other harmful products, it is tempting to think the company’s very public stand this week might signal that it will take moderation more seriously going forward. But “I hesitate to say this is the start of a new era or the result of some structural change,” said Gross. “To say they’re doing something unique is hardly true.”

One reason for skepticism is that, even if the company is truly shifting its approach, its ability to cut down on disinformation merchandise will be hamstrung by its own algorithm. As long as Amazon’s product recommendation algorithm takes a purely neutral approach — even to anti-vaccine or white nationalist products — it will continue to give those that slip through the cracks prominent slots in search results.

The role of the recommendation algorithm
Undergirding Amazon’s lax approach to content moderation is its recommendation algorithm. Exactly what Amazon’s algorithm factors in is unknowable, but Professor Shin said that some likely culprits include “hundreds of features” like “which keywords you use, the sales, the reviews, what you are purchasing, what’s in your shopping cart, what other consumers similar to you are purchasing right now.”

Based on a pair of studies, Amazon’s algorithm does seem to take a truly neutral approach to disinformation — and in doing so, allows dangerous content to rank highly. One instructive example is the vast trove of vaccine-related books on the Amazon Marketplace. In a paper published in the summer, Shin studied how Amazon’s recommendation algorithm treats anti-vaccine literature. The study concluded that, controlling for all other variables, Amazon’s algorithm doesn’t specifically favor anti-vaccine books — but it also does nothing to de-prioritize them in favor of factual, pro-vaccine literature.

The end result is that anti-vaccine books are typically the top sellers on Amazon. That’s because all of those “objective” factors that Amazon does take into account — number of sales, number of reviews and so on — seem to benefit anti-vaccine content. “The algorithm just picks up the signal, and they adjust their outcome,” Shin said. “I don’t think the algorithm is biased in that sense, but it should be more accountable.”

A paper published this fall confirmed this phenomenon, concluding that “pro misinformation items have higher ratings than anti misinformation items.” A recent ProPublica investigation into the proliferation of white nationalist books on Amazon also wrote that these books, by nature of the audience they pull in, “seem to have better reviews than other kinds of books” and therefore get good placement in search results.

A long history of selling disinformation
Amazon, meanwhile, has maintained that it doesn’t allow “products that promote, incite or glorify hatred, violence, racial, sexual or religious intolerance or promote organizations with such views.” Yet enforcement has tended to be lax until this week. Major media outlets have pointed to the prevalence of QAnon products on Amazon since at least 2018. Merchandise for the extremist boogaloo movement is also widespread. And before that, Amazon faced continued criticism for allowing anti-vax products to flourish.

Aside from a small handful of exceptions — like in March 2019, when, after a CNN report, the company took down a series of anti-vaccine films hosted on Amazon Prime Video — Amazon has only made limited efforts to respond to these investigations. “It’s really not been a prominent piece of their PR, I would say,” said Gross.

Many of these are not just obscure products tucked away in the far corners of the Amazon Marketplace, either. In 2019, a QAnon book — “QAnon: An Invitation to the Great Awakening” — was among the top 50 bestsellers on Amazon. Modern Retail last October found other Q-emblazoned hat given the  “Amazon’s Choice” designation.

Ultimately, Shin said, Amazon is “agnostic to the democratic implication of what they’re recommending.” Amazon’s decision to purge QAnon and white supremacist merchandise from its platform is a healthy start — but a larger sea change, if it does happen, will have to contend with the role that Amazon’s relentless neutrality to the products it recommends plays in surfacing dangerous merchandise.