Hindsight being 20/20, we should have done more sooner. Seemingly overnight, all keywords were being gobbled up by the dreadful “(not provided)” category in Google Analytics.
Within a matter of two years, every person who had ever valued Google organic traffic had no choice but to deal with what was happening. As search marketers, we receive so many questions about this topic. Let’s explore!
The History of Not Provided
It’s not that the traffic just stopped — not that at all — what happened was organic traffic continued to show up in analytical programs with no reference as to how they found the site on Google search.
In the Beginning
Google announces Secure Search, as a byproduct: (not provided) surfaces.
To make search more secure, users logged into Google services will use search with a secure (httpS) connection.
With the change, website owners will no longer get “referrer” data from SSL users. Instead, Google passes referral data through a custom token called “Not Provided.” Of course, Advertisers are still going to receive keyword referral data as before from the same set of users.
- Official Google announcement published.
- Keyword referral data usage and value becomes a main talking point in industry circles
- Estimated Not Provided: 1%-2%
Keyword referral loss becomes a concern for many in the search industry. The initial Google estimate was site owners would see percentage loss in the single digits. Within two weeks though, Google Not Provided jumped up for many to nearly 10%.
- Heavier SSL adoption spreads across more users
- Privacy debate ramps up
- Optify releases study showing the Not Provided number to be much higher than perceived
- Estimated Google Not Provided: 8%-14%
Several months after the announcement, the number grew beyond 20%. The backlash was extensive at this point, heated debates were happening all over the place.
- Publishers outside of Search and Tech industries begin voicing concerns
- With millions of users, and an estimated 25% of overall searches, Firefox becomes the first major browser to begin using secure search by default
- Estimated Google Not Provided: 16%-17%
A drastic increase in Google Not Provided data happened in September of 2013, when Google announced all searches would go to httpS. BrightEdge released a study showing the average Google Not Provided percentage was around 50% of all keyword referrals. With the announcement, most realize, it’s just a matter of time (most likely a matter of months) until all of Google organic keyword referral data is masked.
- Months earlier, Moz & Distilled disclose they are now seeing 50% not provided
- One month prior, Apple defaults Safari users to secure search for current iOS catching Google off guard.
- Estimated Google Not Provided: 40%
- Keyword Referral Data Goes Dark
- Jan 18th, 2013 - Chrome moves to secure search.
- August 28th, 2013: Report reveals not provided averages to be nearly 50%
- September??: Google quietly flips the switch. All search activity is now encrypted, except for ads.
Assessing the Not Provided Damage
When Not Provided first crept into our data set, the actual value of keyword targeting came into question.
Frequent questions surfaced, “Why should we pay you to optimize pages when we can’t even see if that keyword traffic is converting?” and “Why shouldn’t we just buy the keywords with PPC to see if they are of value?”.
Why would Google take something away that businesses came to rely on? Although it’s a very valid question, the purpose of this post is not to get into the politics, just know, soon, other search engines will follow suit and the majority of organic keyword data will be masked. Now it’s time to prepare.
How Can I Get Not Provided Data Back?
Your organic search referrals in Google Analytics are now in a generic search bucket. Many bloggers try to lure us in with fun alliteration like “How to Unlock Not Provided” and “10 Steps to Find Not Provided,” but it’s not so simple.
You can’t “unlock it” and you can’t “find it,” it’s gone. What we can do is press harder to find data that supports organic keyword value.
Sure, Google Webmaster Tools has search query reports and Adwords still provides keyword referral data, both have their faults.
Workarounds to Google Not Provided Data Loss
1) Adwords Not Provided Options
Luke Alley explores the Paid & Organic Report in Google Adwords.
“I work in PPC, aka 100% provided, and there is a rarely mentioned feature in Adwords called the Paid & Organic Report that provides some insights on organic terms.
After connecting WMT it allows you to see data when only PPC ads were displayed, when only organic listings were displayed, or if both channels were displayed. While not ideal there is opportunity to gain some insights on organic performance of specific terms. The catch? It’s within Adwords, which means you’re spending money with Google. Also, it just happened to be released in August 2013. Coincidence? Probably not.” ~ Luke Alley, AvalaunchMedia.com
Luke raises another interesting point, you have to pay to play to see what’s going on. Adwords will require ad spend and to replicate organic keywords will cost you money. Moreover, the traffic is not truly the same as your those users on the same query. It’s a valuable way to analyze the quality of search terms, but ad performance is not necessarily going to speak on behalf of your organic rankings sharing the SERP.
2) Google Webmaster Tools
Will Scott uncovers some pain-points and tips for using search queries within Google Webmaster Tools.
“Google Webmaster Tools is clunky to use; it’s not terribly easy to look at keyword data and metrics using WMT.
Two of the most disappointing non-features in WMT is you are limited to 90 days of data, so every day that passes you lose valuable reference material. And, date comparison capability is non-existent. There are ways to catalog the Webmaster Tools data. WMT is a great resource, however, to discover how your site is ranking and what people are seeing and clicking on.
Impressions vs. Click Through Rate has never been available in Analytics. Also, Google Webmaster Tools does give you a much bigger set of keywords for which you might be ranking that you would ever want to track in a traditional rank-tracking tool. In some cases this may include tens of thousands of Queries.
There is tremendous data available in Search Queries, and spending the time to dig it out can be a great use for a business’ engagement online. Valuable information is there for those with patience to pull it and sort it into spreadsheets for discovery.” Will Scott, CEO, Search Influence
Recently, data loss from Webmaster Tools has been established , and Dr. Pete (from Moz) has studied the accuracy of the data as well.
3) Landing-Page Query Associations
Look at filters and dashboards to show which landing pages Not Provided hit. Both Kissmetrics and Search Engine Land have covered this strategy in depth. Though, now with 100% Not Provided, you may as well reconstruct the filter. Although these filters may now be out of date, the overall strategy still lends insights into well performing pages. Josh Braaten said it well,
“Measure organic traffic in segments that correspond to specific keyword themes of interest and monitor both the traffic and conversions from those segments,” says Josh Braaten, Director of Inbound Marketing at Collegis Education. “Bonus points go to those who can tie in indirect metrics such as social media engagement and assisted conversions.”
Spot on. At this time, this strategy lends the most credibility to providing insights on organic search value by watching keyword groups ranking for the individual page where conversions are captured.
4) Branded vs. Non-branded Percentages
This metric is often used in the search industry. Utilizing Google Analytics for branded versus non-branded estimations are now extremely inaccurate. Webmaster Tools search queries can be used as a suitable workaround.
Non-branded versus branded was a great way to gauge what kind of footprint any particular site has built up across the web. It also helped understand the offline footprint as well. For example, a site with a large link profile, optimized site and a low branded-to-non-branded ratio would lend questions to offline ad spend and overall brand awarness offline.If able, utilize older data sets in Google Analytics.
5) Utilize Site Search Data
Often thought of one of the most underutilized data sources, this search box within a site can show what users are searching for on a specific page, overall queries, how different visitors use the site and more. Both free and paid versions are available for custom search options. As #3 noted, patters of searches from a landing page may lend enough insights to change focus to accommodate that user group’s initial expectations of the page.
Moving Forward: Watch it Closer & Adapt Process
Don’t give up.
There is still value in ranking organically, stay focused on valuable keywords. Adhere to best practices, watch page performance closely in Google Analytics and ALWAYS dig deeper when page performance drops due to Google / Organic.
Between Google Webmaster Tools and third party rank tracking, it shouldn’t be difficult to isolate changes. Vendors like Conductor and Search Metrics will monitor their own search indices and connect to Analytics. In fact, Conductor has had a Not Provided product in place since September.
Watch Not Provided traffic closer
- Tie Not Provided traffic to individual pages and cross-reference against rankings from Webmaster Tools and third-party rank trackers
- Segment Not Provided traffic by landing page titles
Download GA templates for Not Provided
- Six Not Provided dashboards from NotProvidedKit.com
- Not Provided – Landing Page dashboard from State of Search
Don’t put all of your eggs in one basket
- Google Site Search
- Google Keyword Planner
- Bing & Yandex Webmaster Tools
- Google Adwords
- Third party rank trackers (AWR & Positionly)
- SearchMetrics / Conductor / SEMRush
- Sorry Ahrefs.com!
The damage of 100% Not Provided is substantial. It touched hundreds of thousands of businesses from the self-employed, to small biz and enterprise in B2B and B2C. Data that businesses relied on to help generate revenue is simply no longer available in the same capacity. Every solution is a workaround rather than a fix.
There is a reason to freak out, especially if your job is riding on proving the CPA from the organic keyword level. For the rest of us, we adapt and rebuild process. Plenty of substitutes are available with adequate data to offer up. Just remember, there is no substitute for your best judgement!
- SSL Search Announcement: https://support.google.com/websearch/answer/173733?hl=en
- Conductor Not Provided Study: http://www.conductor.com/blog/2012/03/51-million-visits-analyzed-not-provided-16-of-google-organic-traffic/
- Optify Not Provided Study: http://www.marketwired.com/press-release/optify-study-googles-not-provided-rises-almost-40-organic-traffic-b2b-sites-1725242.htm
- BrightEdge Not Provided Report: http://www.brightedge.com/secure-search-report-august-2013
- Comparing Rank-Tracking Methods: Browser vs. Crawler vs. Webmaster Tools: http://moz.com/blog/comparing-ranktracking-methods-browser-vs-crawler-vs-webmaster-tools
Note from the author: we will continue to update this resource as new techniques surface on how to better isolate organic keyword performance. Please help us by sharing any missed strategies below!