Big Data Changed How We Measure Success
By Emmanuel Legrand of Legrand Network. This story was originally published in French by IRMActu.
Who's No.1? How do you assess and define success? Who are the most popular artists and recordings at any given time? Ever since there has been a music industry, these questions have obsessed record company executives and artists alike. And to answer these questions, systems were put in place to measure success and identify which artists and songs were the most popular.
In the early 20th Century, music publishers were benchmarking success to the number of music sheets they sold. Then, the industry started counting physical sales, and monitoring radio airplay. And in this brave new digital world, a whole range of tools have started to track the vast activity taking place online, from streaming and social network action, to illegal torrents and legal downloads.
Having information was one thing, but what mattered was also exposing information. In the US, magazines like Cash Box and Billboard have grown due to their ability to provide exclusive charts that the industry used as benchmarks. Billboard, initially launched to cover the outdoors advertising business (hence the name), evolved into becoming the main source of tools measuring music successes, alongside a booming music industry. Its first chart appeared in 1913 – unsurprising it was a chart listing the best-selling music sheets. Over the years, the magazine added more charts. The first “hit parade” dates back from 1936, and was followed by such features as the Billboard Hit 100 for singles – now in its 56th year – and the Billboard 200 Albums.
Data revolution
In the '50s until the early '90s, the charts were based on stock variations at retail level, a system that was not totally scientifically full-proof! Ditto with radio stations, as the airplay charts were based on tracks radio programmers were declaring they were adding to their playlists, and not on a real airplay measurement. The system was to change for the better in the early '90s when in 1991 Billboard switched to the technology developed by Soundscan for its sales charts. Based on purchases scanned at till level, the system could for the first time measure actual sales.
It transformed the music landscape in the USA by showing that music genres such country, R&B or rap were far more popular than previous charts showed, to the detriment of “heritage” acts whose performances dwindled. For record labels it was a revolution – they could finally measure in real time the impact of marketing and promotional action.
In parallel, BDS rolled out a groundbreaking system which allowed to monitor radio stations' output through digital imprints of songs. Soundscan and BDS were transformative technologies which had a long lasting impact on the music industry. Very quickly, other major music markets embraced these new tools. It is interesting to note that Billboard's parent company VNU eventually acquired measurement company Nielsen, which in turn acquired Soundscan and BDS, to create a company that was based on two main pillars: data and media outlets to expose the data. [Disclosure: I have worked for VNU/Nielsen for some 20 years, first as a correspondent and then as London-based editor for Billboard-owned publication Music & Media, and then as Global Editor for Billboard.]
Since then, its media side was acquired by Prometheus Global Media while Nielsen has gone its own way. But Billboard has never stopped presenting dozens of charts, and adapting to new means of consuming music. In 2013, the magazine announced that it would add YouTube streams to the Billboard Hit 100. More recently, it added Twitter info to its charts. Meanwhile, Nielsen Soundscan continues to track the sales of legal downloads on a global level and physical/digital sales in the US.
New tools
In the radio monitoring area, BDS suddenly had to make do with the arrival of new players, such as MediaBase, owned by Clear Channel. In this field, more changes took place in Europe, Music Control, a German-based company, was the main player for decades. It provided airplay data covering the main European countries. MC was acquired by Nielsen in the early '00s and, some 10 years later, its London-based operations were closed down during the summer of 2013. Subsequently, many national operators saw their position strengthened, like Yacast in France or Radio Monitor in the UK, the latter having also taken over a few markets previously covered Music Control.
The demise of Music Control was the result of the contraction of the music industry (less clients = less revenues) as well the effect of the advent of new tools which, at start of the new Century, were tracking digital consumption and consumers' behaviour. A bunch of new companies, using the latest technologies, entered the data market. In the radio monitoring field, a company like Kollector in Belgium offers a service mainly aimed at rights societies but also individual artists or indie labels, to track songs. In the same field operates Bach Technology from Norway. Out of the blue, Bach won last year the official tender from the German music industry to monitor national stations.
Bach exemplifies the new breed of companies that have blossomed over the past few years, taking advantage of new and innovative technologies. Its system is based on a proprietary technology (MusicDNA) which does digital fingerprinting of songs on a massive scale to capture radio streams on the internet. It operates a database of several million tracks – 12 million at the latest count – which can be used to identify tracks that are played. Each entry in the database is connected to a title, an artist, and when available, its ISRC code, a global identifier.
MusicDNA's algorithms are capable of identifying over 8,000 tracks simultaneously. “There are many radio monitoring systems available on a local scale but we are the only ones that operate internationally, covering stations in 99 countries, and monitoring more than 10,000 channels,” explains Dagfinn Bach (shown at right), founder and president of Bach Technology, who adds that he plans to expand the pot of stations monitored to 20,000 throughout 2014.
What is Bach's core business? “We are monitoring the music that is played on radio,” says Bach. “Every title/artist is stored in a database, when it is played. Clients get a report with the played songs that they have commissioned. Additionally, clients get global charts on any country world-wide.”
More relevance
Bach did not invent a new service, but took an existing service to a brand new different scale. In the field of data, these recent years saw the advent of new players monitoring new usages and providing new data previously unavailable on consumers' behaviours. Among them is BigChampagne – now a division of the world's leading live music company Live Nation, based in Berverly Hills – which has built an expertise on the way creative works are accessed and consumed. With a tag line stating “More relevant data – better tools”, BigChampagne's data is accessible through a dashboard which combines different analytics covering various set of fields. The tool is optimised to help labels or artist managers to make better-informed decisions. BigChampagne has also created the “ultimate chart”, which identifies the most popular tracks based on action on social networks, concert ticket acquisition, sales of recorded music, radio plays, etc.
Producing analytics is also at the heart of British company Semetric, which has developed MusicMetric, a tool which tracks activity on social networks (Facebook, Youtube, Vevo, LastFM, Soundcloud, Instagram, Myspace, Twitter, Tumblr, Wikipedia, etc) and torrents (mostly illegal downloads…), combined to provide a wide picture of what consumers do online with music content.
MusicMetric's staff self-describes itself as a bunch of music lovers, but they are also data specialists that delve in “data mining”. “The company has an unusually high PhD to staff member ratio,” says Jeremy Silver (shown at left), executive chairman of MusicMetric. “A team of twenty world leading engineers, based in London, is taking the technology to the leading edge of what is possible in todays big data mining space.”
The heart of MusicMetric's system is a proprietary platform built on Hadoop big data management technology. In the space of a couple of years, the system has been used by all the majors and a handful of indie labels, but also music distributors and aggregators, music streaming companies, internet giants, national TV and radio broadcasters, venues and live promoters, and brands.
“Our clients use our data to access the complete competitive landscape of music fan engagement,” says Silver. “They very often have their own data but not their competitors'. They often can see part of the picture but not all of it. We fill in the gaps and give them excellent visibility and visualization of what is happening in a high level of detail.”
Effective marketing
Silver adds, “They use us to plan and track effectiveness in marketing campaigns, make record deal signings, geo-locate fan engagement, benchmark levels of piracy, maintain real time monitoring of advertising campaigns, power playlist creation, music discovery and recommendation engines, and more.” As of the beginning of 2014, Semetric began collecting data not just about music, but also about TV, film, books and games.
Data based on consumers' behaviour has a wide reaching effect. Recently, the global streaming service Spotify bought EchoNest, a company which analyses music tastes in order to provide recommendations. Since recommendation has not been Spotify's forte, EchoNest has the potential to provide Spotify with a tool that could give the service a competitive edge and help differentiate its offer from that of other players such as Beats, Rdio or Pandora.
“We are tracking tens of millions of people’s music listening history on our systems. This powers all of our personalization, so when you log on to one of the services that use the Echo Nest Taste Profile, they’ll know about you right away,” said Brian Whitman, co-founder and chief technology officer at Echo Nest, during a recent presentation at Gigaom.
In view of what the NSA does with data, this way of doing things may not go down well with customers, but for Jim Griffin (shown below), managing director of US consulting company OneHouse (http://www.onehouse.com), data has become a key element helping creative industries make better decisions. “Data is highly relevant, both to decision-making for how to optimise the artist's value and for those who would potentially license the artist's work,” says Griffin.
Total granularity
In the past, adds Griffin, measuring success in the pre- and post- Soundscan era, and before the big data era, was based on retail sales or airplay. And in fact, it delivered very little in terms of useful data. “When SoundScan and BDS launched early 90s, they were told no one wanted either of them, but within months everyone was subscribed to everything they could get,” jokes Griffin.
He adds, “Traditionally, this data has been fuzzed to protect specific retailers or outlets, as well as the integrity of the data-gathering process. SoundScan resisted attempts to determine just what music is selling well across the street, but it would isolate on your zip code. Fuzz no more. Total granularity is at hand, becoming still more granular.”
Granularity, which is the process of breaking up and isolating the smallest part in a system, feeds from data, and data feeds from granularity. In the case of music, or entertainment, it allows to have as precise as possible a picture of what consumers do and when. Overall, it is now less about who's No.1 than who gets the most traction and interaction with consumers.
“We once sold music – and we also sold a bit of crowd,” explains Griffin. “Now we sell lots more crowd and lots less music, so data is essential to determining the deliverable – and the best strategy for delivering. In the product to service transition, data represents relationship just as does a fat Filofax or Rolodex represent your gatekeeper status. These services are not only highly relevant, they are also fashionable, so everybody wants some because … well, everyone else has some.”
