Online shopping and the Harry Potter effect

22 December 2008 by 
Magazine issue .  and get 4 free issues.
For similar stories, visit the  Topic Guide
AS YOU tear into the wrapping paper this festive season, you might be thinking of those long weekends your loved ones spent trudging through the crowds to find that elusive perfect expression of their affection. Quaint thought, but according to UK sales figures, if last year's festive season is anything to go by, it is likely that at least 1 in 5 of your friends and family shopped online, probably with a nice glass of wine in hand rather than a shopping bag digging into their palm. Elsewhere in the developed world, the picture is much the same.
This seismic shift in shopping habits is down to more than the mere convenience of shopping in your slippers. A new school of thought in business has emerged that claims it is all about choice. On the net, you can buy just about anything you can think of, and a lot of things you never have. Amazon alone lists millions of products, including everything from the latest New Scientist book to maternity bras, plasma TVs and a vibrating ladybird-shaped massager.
The message is simple: no matter what your taste, you will find something to satisfy it online. Gone are the days when shoppers had a limited selection of blockbusters to choose from. The internet is changing not only how we shop, but what we buy.
And yet the big sellers have never been bigger. Take the latest Harry Potter book, which on launch in 2007 sold 11 million copies in its first day, exceeding the record for the fastest-selling book - set consecutively by the previous three Potter tomes. So what's the real deal? Is the internet really broadening our horizons? Or is it all marketing fluff masking the fact that best-sellers still rule?
The idea that the internet is transforming our buying habits was first popularised in an article written by Chris Anderson, editor-in-chief of the technology magazine Wired, in October 2004. The article, titled "The long tail", became a blog, a best-selling book and a marketing mantra.
Anderson's premise was simple. Before the internet, even the largest retailer had physical constraints on the variety of stuff like books, CDs and DVDs it could profitably sell. Savvy store owners had to tailor their stock to maximise returns from limited shelf space, and consumers had to make do with what was on offer.
In cyberspace, by contrast, shelf space is practically unlimited and other overheads at rock bottom. Online retailers can take advantage of that to sell vast catalogues of obscure products at little or no extra cost.
Anderson postulated that this new breadth of choice was leading to a metamorphosis of the classic demand curve. Instead of a steep peak representing comparatively few big-selling items, a gentler curve spread over far more products was emerging - creating the eponymous long tail (see graph).
To Anderson, this migration to the long tail is a journey of self-discovery for all of us. "As a teenager in the 1970s, I listened to one of the 10 radio stations on offer, but I'm not sure I really knew what my taste was." Now, he says, we are all finding out - and discovering that we like quite different things. His conclusion is summarised in the subtitle of his book: "The future of business is selling less of more".
It is an attractive hypothesis that rapidly acquired an evangelical following. But then dissenting voices emerged. The first shot across the bows came in July this year, when Anita Elberse of Harvard Business School published . Her findings suggest that the long tail is far from the revolution Anderson claimed. The tail is indeed getting longer, but isn't, as Anderson thought, growing fat with choice. Instead it is getting both flatter and thinner, filled with ever more products that sell few or no copies.
Low overheads or no overheads, that kind of long tail is not a rich man's world, least of all for producers. Suppliers are always going to be better off concentrating on the mass-market money-spinners, says Elberse.
Her ideas have recently been backed up by another study. Will Page and Gary Eggleton at the MCPS-PRS Alliance - a UK body that collects royalties for musicians when their songs are played on air or downloaded - and Andrew Bud from the cellphone software company mBlox have analysed a year's worth of downloads from a well-known internet music store. They found that of the 13 million tracks available, 52,000 - just 0.4 per cent - accounted for 80 per cent of downloads.
Of the 13 million tracks available, 0.4% of them account for over three-quarters of downloads
The overall pattern of demand showed nothing like the shift towards a long, fat tail postulated by Anderson. In fact, it showed the reverse: it followed what is known as a log-normal curve, characterised by a sharp spike of best-selling blockbusters that rapidly tailed off into nothing. There was an added irony to Page's findings. If an average album holds 12 tracks, 52,000 songs equate to about 4300 CDs - according to Anderson's Long Tail book, about the number of titles stocked by Wal-Mart, the largest bricks-and-mortar retailer in the US.
In other words, says Page, the extra choice available online is of little economic worth to the retailer. "Scarcity in conventional retailers might be a constraint," he says. "But it could also be a discipline, representing an economically optimal inventory."
Results from a study by anthropologist Alex Bentley and economist Paul Ormerod of Durham University, UK, together with anthropologist Mark Madsen of the University of Washington in Seattle, agree with Page and his colleagues' findings (). The researchers found that in a fashion-led market, where consumer preferences are fickle and fast-changing, the inventory size that maximises profits is actually very small - equivalent to the 10 to 20 titles often stocked in an airport bookstore.
Even for an online retailer such as Amazon, where the profit-to-overhead ratio per item is very large, the optimum inventory is likely to be considerably smaller than the millions of titles actually offered, says Ormerod. He counsels caution: "It's a good marketing strategy to say you sell everything," he says, "but it is unlikely to be profitable on its own." Elberse agrees: "The long tail was a great idea," she says, "but as a business model it was too optimistic." The greatest attraction for shoppers, she suggests, is probably the aggressive discounts Amazon offers on its blockbuster products.
Anderson concedes that, for producers, the economics of the long tail might be shaky, but defends his central thesis: that the choices internet retailers offer open our eyes to a rich new world of possibilities - and that canny retailers can profit from that. "For most people, there's no money in the long tail," he says. "But then, most books don't make money. Most films don't make money." Nevertheless, a market like this will benefit retailers like Amazon, and more importantly us consumers. We might still subsist on a staple diet of blockbusters, but we can enrich that with other titbits from the tail in a way we could not before.
So why, with the cornucopia of goodies now available to us, are blockbusters not just still here, but getting bigger? On the face of it, Anderson's idea of a divergence of tastes in the digital era is logical. But if the long tail effect does not exist, or is not as pronounced as was thought, what is really going on?
Elberse says it's a bit like the influence of multichannel television on the economics of sport. In the old days, if you wanted to watch soccer, you went to watch your local team in the flesh. Now, she says, in the UK you are more likely to decide to stay at home and watch Chelsea play Arsenal. This change of allegiance cuts the cash flowing into the ticket office of your local club while boosting advertising revenues for TV, which accrue disproportionately in favour of the already wealthy top clubs.
It is a phenomenon known to economists as the Matthew effect, after a quotation from the gospel of that name: "For unto every one that hath shall be given." Just as for the long tail effect, there is a plausible explanation of why it should be happening in the modern media environment: easy digital replication and efficient communication through cellphones, email and social networking sites encourage fast-moving, fast-changing fads. The result is a homogenisation of tastes that boosts the chances of popular things becoming blockbusters, making the already successful even more successful.
The winner takes it all
Duncan Watts, a sociologist at Columbia University, New York, has found evidence of just such an effect. Together with his colleagues Matthew Salganik and Peter Dodds, he tested the effect of communication and peer approval on the musical tastes of 14,000 teenage volunteers recruited online (). A set of 48 songs was made available to all the volunteers, who could download whichever songs they wanted. The researchers split the volunteers into eight groups; in some, group members could see what their peers were downloading, but in others they had no such knowledge. In the socially connected groups, the winner took all: popular songs became more popular, unpopular songs more unpopular. This effect was much less pronounced in the socially isolated groups.
What's more, there was a sting in the tail for anyone trying to predict blockbusters: in different groups, different songs tended to become the biggest hits. It is a classic butterfly effect: a small preference for a particular object can, in a highly connected community, rapidly amplify and spread its appeal. Increased social connectedness creates bigger blockbusters but makes predicting what they will be all the more difficult.
Which leads to a curious puzzle: why, when we have so much information at our fingertips, are we so concerned with what our peers like? Don't we trust our own judgement? Watts thinks it is partly a cognitive problem. Far from liberating us, the proliferation of choice that modern technology has brought is overwhelming us - making us even more reliant on outside cues to determine what we like . "Google can deliver 100 million songs to you - but your brain hasn't got any faster," he says.
Why, when we have so much information at our fingertips, do we care what our peers like?
Watts thinks there is a fundamental problem with ideas like the long tail: they reduce our likes and dislikes to a set of isolated, stable preferences. "There is a naive assumption that people are making decisions independently, scouring all possible choices and then optimising," he says. "But people aren't rational in that weird economists' sense."
Quite apart from the cognitive overload aspect, Watts thinks that the films we watch or the music we listen to is not entirely, or even mostly, about the thing being consumed. It is about the social context in which we consume it. "If you're dating a girl who likes AC/DC, you might start listening to AC/DC. With another girlfriend, you might be listening to Aerosmith," he says.
Ultimately, he thinks, our love of the blockbuster might just reflect that we humans are constantly looking out for a place to go - one where others are too. "A culture is a set of people who share beliefs, ideas and artefacts," says Watts. "Blockbusters are part of that - they make us feel we belong to something."
Anderson agrees, and says he has never said any different. "Blockbusters are always going to exist - they're a combination of what's genuinely awesome and banalities that appeal to the lowest common denominator. What the long tail does is break the tyranny of the blockbuster." By making it easier for all of us to access the choice available - and also with blogs, self-publishing, YouTube and the like, to produce it ourselves - the internet puts us in a better position than ever to determine ourselves what our shared tastes are.
Anderson sees the web as a gene pool from which worthy blockbusters can emerge by a process akin to natural selection. "The bluefin tuna lays maybe 3 million eggs, and three hatch. But up until now, it has been too expensive to waste life like that in media production," he says. Now, however, there is the chance that a truly commercial blockbuster can arise organically, by public acclaim, from the long tail.
The organic blockbuster? Surely that has got to taste better than the mass-produced, force-fed Christmas turkeys scheduled on TV for the next few days.
I know what you'll buy next summer
Ever had the feeling that someone is reading your mind? If you browse through any of a growing list of online shopping sites, you might be forgiven for doing so. To boost their chances of making a sale, sites are increasingly using a technology called collaborative filtering to make an educated guess at what you might be thinking of buying.
The software beavers away behind your Amazon homepage, for example, and recommends products based on your previous purchases and clicking history. For example, after I looked up a film version of King Lear directed by Soviet film-maker Grigori Kozintsev, my Amazon homepage suggested I buy Shakespeare's King Lear (naturally), but also Dmitri Shostakovich's first cello concerto (because Shostakovich composed the music for several of Kozintsev's films) and Doctor Zhivago (presumably because it is a film about Russians).
It is this sort of software that paradoxically promotes the sales of blockbusters and is responsible for homogenising choice online, say Daniel Fleder and Kartik Hosanagar of the University of Pennsylvania, Philadelphia. They outline this idea in . To make useful recommendations, most collaborative filters must make guesses based on previous sales and ratings. But obscure products do not have much in the way of sales or ratings to go on, so the filters tend to point us towards more mainstream offerings. By following such a widget's recommendations, I might well vastly diversify my own personal cultural experience, but I will also contribute to a decrease in system-wide diversity, adding more sales to already popular objects.
From issue  of New Scientist magazine, page 52-55.  and get 4 free issues.
 of New Scientist magazine
If you would like to reuse any content from New Scientist, either in print or online, please department first for permission. New Scientist does not own rights to photos, but there are a  available for use of articles and graphics we own the copyright to.