Here’s a puzzling situation: An up-and-coming website looks at how people find it online – but things aren’t adding up. A big chunk of Web traffic seemingly appears out of nowhere, befuddling the people trying to build an audience. After all, it’s hard to grow a fan base when you don’t know where your existing viewers come from.
That might seem like an unusual scenario in the age of SEO and constant analytics, but it’s not. The Atlantic is an enormously popular online portal with staff dedicated to poring over traffic, yet one quarter of their Web traffic comes from this unknown source. Buzzfeed also pointed out decreases in traditional traffic sources and an increase in unspecific traffic. And two interconnected culprits are responsible for this analytics-defying traffic: Dark social and dark Google.
But in 2011 things changed, and Google started withholding around 50 percent of this information.
Dark social and dark Google are something that anyone with a website – even a Tumblr or WordPress blog they want to take off – should care about, because it makes it impossible to accurately gauge your audience, and it also makes it harder to figure out why certain content succeeds and other content falls flat. It’s like an analytics night terror – and the problem may continue to grow as more people use mobile devices to access the Web, since iPads and iPhones throw the most data into the dark.
So what are dark Google and dark social?
Google used to report where traffic came from through its search engine very accurately; withheld information was rare. But in 2011 things changed, and Google started withholding around 50 percent of this information from Web publishers. This withheld information is dark Google.
Dark social is a term coined by The Atlantic’s tech writer Alexis Madrigal to explain the Web traffic that comes from unspecified, mysterious social shares. Dark social accounts for all the people who find articles through sharing, but not on traditional sharing sources like Facebook, Twitter, Reddit, and the other big sources for finding links from others. Dark social is not a little portion of social links; in 2012, Madrigal said it accounted for 69 percent of social sharing. That’s a whole lot of link-swapping unaccounted for by Web analysts, and a whole lot of missed opportunities for people trying to measure their audience.
Rosetta SEO director Stephen Pitts says dark social is the offspring of dark Google. “It’s the same root cause as dark Google,” he explains. But while dark Google is hidden search traffic, dark social is hidden traffic that stems from people sharing links.
The shared cause is the traffic chaos they create. Pitts describes dark social and dark Google as a data referral problem at its core. The information websites need to analyze their traffic is simply not getting passed on anymore. “There is a function that is normally handled by browsers that is a meta-referrer for information. The meta-referrer takes the keyword if it’s coming from a paid or organic search in, say, Google, and sends that to an analytics package. In analytics, it would tell you who the referrer was and the keyword. Now, in this circumstance, what happened is last year Google Search decided to stop providing that as a function because it’s handled by most browsers. So now they put the onus on passing that data to the browsers.”
And now that browsers are in charge, things are falling out of whack. “That wasn’t a problem until Apple, with the release of the iPhone 5 and iOS 6, decided they were trimming down all the data being requested in their browsers. Apple stopped supporting meta-referrer data. So if any iOS device is used, whether it be to instant message, text message, to use an app that’s sending you to a website, it won’t pass the meta-referrer information. So the analytics program thinks its direct traffic.” So there’s no simple way to measure how much is dark Google and how much is dark social; both types of traffic just end up unmeasured.
Even though this creates a giant headache for people trying to figure out their audiences, don’t expect a change.
Not every mobile browser is a problem, though. Pitts uses the Facebook app as an example of a service that still passes data along. “The Facebook app is a little different. If I’m on Facebook on my iPhone and I click to an article on the Web, I’m on the Facebook browser, which still passes the meta-referrer data. The browser functionality on the Facebook app is still providing the information.” That’s not surprising, considering Facebook’s vested interest in collecting data on its users. It wouldn’t want to alienate potential ad-buyers by refusing to confer information.
Apple is by and large the biggest culprit when it comes to not passing along the necessary meta-referrers. “If you use Chrome on your iPhone, the data still gets passed. It’s just if you use Safari [it isn’t].” And Pitts hadn’t heard of similar problems on Android. “It’s mostly iOS. When you’re on an Android device you’re almost always defaulting to a Chrome browser – and it’s mostly Safari that’s the issue.”
But Firefox might also cause problems later on. “They’re talking about trimming everything except first-party cookies,which would in essence do the same thing as iOS.”
So Google assumes that browsers will take responsibility for passing along data, but Apple hasn’t taken up that job with Safari. And thus, marketers, webmasters, and other groups concerned with analytics lose valuable information – especially considering the amount of iPhones and other iOS products out there.
Even though this creates a giant headache for people trying to figure out their audiences, don’t expect a change.”Neither Google nor Apple have incentive to make a change,” Pitts says. “The people who are losing this insight are the marketers and the website owners, which is disappointing because we can’t optimize our sites to improve user experience if we can’t use the data.”
“We see it less likely to happen on financial services and B2B sites, while it’s more likely to happen on retail and hospitality sites,” Pitts explains, noting that people tend to online shop on their phones, while they often wait to do things like refinance their homes when they’re on a traditional, more secure desktop. That doesn’t mean it never happens, though. “It’s amazing to me that people go through tedious, 18 page documents on tiny screens. But people do it. It might be their only device that’s connected.”
Unless people analyzing traffic figure out a workaround, as long as people continue to use Safari and other browsers that don’t send meta-referrers along, this problem will persist and continue to stymy those who want to know their audience.
Is there any benefit for users? Well, dark Google and dark social confound attempts by marketers to figure out ways to target ads, so if you really hate seeing targeted ads, you may take solace in the fact that both Google and Apple are withholding information. Then again, if you like using retail sites that customize their services based on your past preferences, both dark Google and dark social make it harder for tailored user experiences. And anyone with a website they want to drive traffic to – which is a considerable number of you out there – will continue in the frustration of never really knowing all your readers.
