The Ultimate Guide to Google’s Keyword ‘(Not Provided)’

image userpic

Hindsight being 20/20, we should have done more sooner. Seemingly overnight, all keywords were being gobbled up by the dreadful “(not provided)” category in Google Analytics.

Within a matter of two years, every person who had ever valued Google organic traffic had no choice but to deal with what was happening. As search marketers, we receive so many questions about this topic. Let’s explore!

The History of Not Provided

Google Not Provided Graph

It’s not that the traffic just stopped — not that at all — what happened was organic traffic continued to show up in analytical programs with no reference as to how they found the site on Google search.

In the Beginning

Google announces Secure Search, as a byproduct: (not provided) surfaces.

not-provided-surfaces

October 2011

To make search more secure, users logged into Google services will use search with a secure (httpS) connection.

https-google-users

With the change, website owners will no longer get “referrer” data from SSL users. Instead, Google  passes referral data through a custom token called “Not Provided.” Of course, Advertisers are still going to receive keyword referral data as before from the same set of users.

  • Official Google announcement published.
  • Keyword referral data usage and value becomes a main talking point in industry circles
  • Estimated Not Provided: 1%-2%

November 2011

Keyword referral loss becomes a concern for many in the search industry. The initial Google estimate was site owners would see percentage loss in the single digits. Within two weeks though, Google Not Provided jumped up for many to nearly 10%.

  • Heavier SSL adoption spreads across more users
  • Privacy debate ramps up
  • Optify releases study showing the Not Provided number to be much higher than perceived
  • Estimated Google Not Provided: 8%-14%

March 2012

Several months after the announcement, the number grew beyond 20%. The backlash was extensive at this point, heated debates were happening all over the place.

  • Publishers outside of Search and Tech industries begin voicing concerns
  • With millions of users, and an estimated 25% of overall searches, Firefox becomes the first major browser to begin using secure search by default
  • Estimated Google Not Provided: 16%-17%

November 2012

A drastic increase in Google Not Provided data happened in September of 2013, when Google announced all searches would go to httpS. BrightEdge released a study showing the average Google Not Provided percentage was around 50% of all keyword referrals. With the announcement, most realize, it’s just a matter of time (most likely a matter of months) until all of Google organic keyword referral data is masked.

  • Months earlier, Moz & Distilled disclose they are now seeing 50% not provided
  • One month prior, Apple defaults Safari users to secure search for current iOS catching Google off guard.
  • Estimated Google Not Provided: 40%
  • Keyword Referral Data Goes Dark

Notable Dates:

  • Jan 18th, 2013 - Chrome moves to secure search.
  • August 28th, 2013: Report reveals not provided averages to be nearly 50%
  • September??: Google quietly flips the switch. All search activity is now encrypted, except for ads.

Assessing the Not Provided Damage

person-analyzing-not-provided-data

When Not Provided first crept into our data set, the actual value of keyword targeting came into question.

Frequent questions surfaced, “Why should we pay you to optimize pages when we can’t even see if that keyword traffic is converting?” and “Why shouldn’t we just buy the keywords with PPC to see if they are of value?”.

Why would Google take something away that businesses came to rely on? Although it’s a very valid question, the purpose of this post is not to get into the politics, just know, soon, other search engines will follow suit and the majority of organic keyword data will be masked. Now it’s time to prepare.

What matters now is.. how do we get those numbers back, what can we do to show value and how can we adapt?

How Can I Get Not Provided Data Back?

You can’t.

Your organic search referrals in Google Analytics are now in a generic search bucket. Many bloggers try to lure us in with fun alliteration like “How to Unlock Not Provided” and “10 Steps to Find Not Provided,” but it’s not so simple.

searching-for-not-provided-data

You can’t “unlock it” and you can’t “find it,” it’s gone. What we can do is press harder to find data that supports organic keyword value.

Sure, Google Webmaster Tools has search query reports and Adwords still provides keyword referral data, both have their faults.

SEO’s simply want to cross-reference keywords with performance metrics.

Workarounds to Google Not Provided Data Loss

As we always do, the search industry thrives in a collaborative environment. So, we also reached out to Will Scott, Josh Braaten and Luke Alley for additional suggestions and workarounds.

1) Adwords Not Provided Options

Luke Alley explores the Paid & Organic Report in Google Adwords.

“I work in PPC, aka 100% provided, and there is a rarely mentioned feature in Adwords called the Paid & Organic Report that provides some insights on organic terms. 

Adwords and Organic Keyword Breakdwon

After connecting WMT it allows you to see data when only PPC ads were displayed, when only organic listings were displayed, or if both channels were displayed. While not ideal there is opportunity to gain some insights on organic performance of specific terms. The catch? It’s within Adwords, which means you’re spending money with Google. Also, it just happened to be released in August 2013. Coincidence? Probably not.” ~ Luke Alley, AvalaunchMedia.com

Luke raises another interesting point, you have to pay to play to see what’s going on. Adwords will require ad spend and to replicate organic keywords will cost you money. Moreover, the traffic is not truly the same as your those users on the same query. It’s a valuable way to analyze the quality of search terms, but ad performance is not necessarily going to speak on behalf of your organic rankings sharing the SERP.

2) Google Webmaster Tools 

Will Scott uncovers some pain-points and tips for using search queries within Google Webmaster Tools.

webmaster-tools-search-queries

“Google Webmaster Tools is clunky to use; it’s not terribly easy to look at keyword data and metrics using WMT. 

Two of the most disappointing non-features in WMT is you are limited to 90 days of data, so every day that passes you lose valuable reference material. And, date comparison capability is non-existent. There are ways to catalog the Webmaster Tools data. WMT is a great resource, however, to discover how your site is ranking and what people are seeing and clicking on.  

Impressions vs. Click Through Rate has never been available in Analytics. Also, Google Webmaster Tools does give you a much bigger set of keywords for which you might be ranking that you would ever want to track in a traditional rank-tracking tool. In some cases this may include tens of thousands of Queries.

There is tremendous data available in Search Queries, and spending the time to dig it out can be a great use for a business’ engagement online. Valuable information is there for those with patience to pull it and sort it into spreadsheets for discovery.”  Will Scott, CEO,  Search Influence

Recently, data loss from Webmaster Tools has been established , and Dr. Pete (from Moz) has studied the accuracy of the data as well.

3) Landing-Page Query Associations

Look at filters and dashboards to show which landing pages Not Provided hit. Both Kissmetrics and Search Engine Land have covered this strategy in depth. Though, now with 100% Not Provided, you may as well reconstruct the filter. Although these filters may now be out of date, the overall strategy still lends insights into well performing pages. Josh Braaten said it well,

“Measure organic traffic in segments that correspond to specific keyword themes of interest and monitor both the traffic and conversions from those segments,” says Josh Braaten, Director of Inbound Marketing at Collegis Education. “Bonus points go to those who can tie in indirect metrics such as social media engagement and assisted conversions.”

Spot on. At this time, this strategy lends the most credibility to providing insights on organic search value by watching keyword groups ranking for the individual page where conversions are captured.

4) Branded vs. Non-branded Percentages

This metric is often used in the search industry. Utilizing Google Analytics for branded versus non-branded estimations are now extremely inaccurate. Webmaster Tools search queries can be used as a suitable workaround.

Non-branded versus branded was a great way to gauge what kind of footprint any particular site has built up across the web. It also helped understand the offline footprint as well. For example, a site with a large link profile, optimized site and a low branded-to-non-branded ratio would lend questions to offline ad spend and overall brand awarness offline.If able, utilize older data sets in Google Analytics.

5) Utilize Site Search Data 

Often thought of one of the most underutilized data sources, this search box within a site can show what users are searching for on a specific page, overall queries, how  different visitors use the site and more. Both free and paid versions are available for custom search options. As #3 noted, patters of searches from  a landing page may lend enough insights to change focus to accommodate that user group’s initial expectations of the page.

Moving Forward: Watch it Closer & Adapt Process


The feeling of having anything taken away sucks.

mad-about-not-provided-data

Don’t give up.

There is still value in ranking organically, stay focused on valuable keywords. Adhere to best practices, watch page performance closely in Google Analytics and ALWAYS dig deeper when page performance drops due to Google / Organic.

Between Google Webmaster Tools and third party rank tracking, it shouldn’t be difficult to isolate changes. Vendors like Conductor and Search Metrics will monitor their own search indices and connect to Analytics. In fact, Conductor has had a Not Provided product in place since September.

Key Takeawys

Watch Not Provided traffic closer

  • Tie Not Provided traffic to individual pages and cross-reference against rankings from Webmaster Tools and third-party rank trackers
  • Segment Not Provided traffic by landing page titles

Download GA templates for Not Provided

Don’t put all of your eggs in one basket

  • Google Site Search
  • Google Keyword Planner
  • Bing & Yandex Webmaster Tools
  • Google Adwords
  • Third party rank trackers (AWR & Positionly)
  • SearchMetrics / Conductor / SEMRush

Conclusion

The damage of 100% Not Provided is substantial. It touched hundreds of thousands of businesses from the self-employed,  to small biz and enterprise in B2B and B2C. Data that businesses relied on to help generate revenue is simply no longer available in the same capacity. Every solution is a workaround rather than a fix.

There is a reason to freak out, especially if your job is riding on proving the CPA from the organic keyword level. For the rest of us, we adapt and rebuild process. Plenty of substitutes are available with adequate data to offer up. Just remember, there is no substitute for your best judgement!

References:

  • SSL Search Announcement: https://support.google.com/websearch/answer/173733?hl=en
  • Conductor Not Provided Study: http://www.conductor.com/blog/2012/03/51-million-visits-analyzed-not-provided-16-of-google-organic-traffic/
  • Optify Not Provided Study: http://www.marketwired.com/press-release/optify-study-googles-not-provided-rises-almost-40-organic-traffic-b2b-sites-1725242.htm
  • BrightEdge Not Provided Report: http://www.brightedge.com/secure-search-report-august-2013
  • Comparing Rank-Tracking Methods: Browser vs. Crawler vs. Webmaster Tools: http://moz.com/blog/comparing-ranktracking-methods-browser-vs-crawler-vs-webmaster-tools

Note from the author: we will continue to update this resource as new techniques surface on how to better isolate organic keyword performance. Please help us by sharing any missed strategies below! 

This entry was posted in E-commerce, Google Analytics, SEO. Bookmark the permalink.
  • Pingback: SearchCap: The Day In Search, February 17, 2014

  • http://www.seopros.org Terry Van Horne

    Hey Brent 2 things… you could get the data as the top page and top data in GWT actually does provide the corresponding pages for a keyword or vice versa… the problem here is A. the downloadable csv does not contain the pages in the query data or the query in the landing page data…even worse you can’t get it by scraping because it looks like JSON calls that only execute when the executable is clicked… ie: in other words by design they have withheld and blocked the ability to get the data and figure out the notprovided. Further more that data is not available in the GWT API either…so obviously all the guff about they aren’t trying to hide it from us is just Google pissing on our leg and telling us it is raining. They absolutely can facilitate…but have gone to fairly long lengths to block it.

    • Brent Rangen

      Hi Terry,

      As far as I can tell, search query data can’t be directly combined with any meaningful GA metric. You’re exactly right that they ARE working to remove this data from WMT’s exports and API’s. If not, this would be a great time for a Google rep to step forward.

      • http://www.seopros.org Terry Van Horne

        A partial tell is in the way the query and landing Pages reports are coded. They are executables whereas normal implementation is to write the data into divs and show the div on execute, usually click or mouseover. The way it is done simply scraping the pages (most SEO’s can do this with their eyes closed) does not expose the data.

        • Brent Rangen

          Yeah, the good ol’ fashioned URL modification (grid, in this case) works to pull more straight out of WMT’s. I’m sure you’re aware of that. I just stumbled into it and will add it to the article. Feel free to reach out to me Terry with any more insights if you’d like to see any other resources added to this article.

  • Pingback: Web Analytics | Annotary

  • Steve

    Can’t play too conservatively when you’re doing PPC. You have to bid to win that traffic. You should be receiving >90% impression share of the keywords you are bidding on and that means bidding a little higher sometimes. What happens is that you have to have an optimized sales process to justify the ability to bid higher. If you make more per sale, you can bid more per click, and squash your competition.

  • foki

    This site is very great and indispensable to my thank you to expect continued success with the authorities there are millions of people come easy memlunkalan but I guess I did not bring the language

    پارتیشن اداریسایپاپارتیشنکرکره برقیدرب اتوماتیک شیشه ایدرب اتوماتیکطب سنتیطب سوزنیدوربین مدار بسته