Posted by Dr. Pete

Just over a week ago, Google launched a massive change to search personalization, Search Plus Your World. Along with this change came a new toggle switch to shut off personalization. Below the Google search box and above the results, you’ll see something like this:

The default, person icon is personalized results, and you click on the globe to shut off “your world” (I won’t comment on how little sense that makes). Of course, we already had personalized results and a handful of ways to shut them off before, so what does “personalization” mean now, and do any of these de-personalization methods actually work? I thought it was time to put that question to the test.

The Methods

I actually started with 6 ways to de-personalize, but ended up excluding two of them for the final test (more on that below). The original 6 were:

1. De-personalization toggle
2. “pws=0” parameter
3. Signing out
4. Signing out + “pws=0”
5. Incognito (Chrome)
6. Incognito (IronKey)*

I’ve already discussed the new option (1) above, but I thought it might be a good review to talk briefly about the other options. Here’s a quick primer:

(2) “pws=0” Parameter

If you’ve been in SEO for a while, you’re familiar with the “pws=0” de-personalization parameter. By adding it to the end of a Google query URL (“&pws=0”), you can theoretically remove history-based personalization. A simplified URL would look something like this:

(3) Signing Out of Google

This one’s pretty straightforward. Just sign out of your Google account. Unfortunately, the Google interface has been changing a lot lately, but if you have Google+, click on your avatar in the top bar, and you’ll see an option for “Sign Out” at the bottom of the menu.

(4) Signing Out + “pws=0”

Option (4) just combines (2) and (3). Sign out of Google, run your search, and then append the “&pws=0” parameter to the URL.

(5) Incognito Browsing (Chrome)

Google’s Chrome browser has a built in “incognito” mode that supposedly removes any traces of your browsing activity, such as cookies or search history.  In Chrome, click on the wrench icon in the upper right, and you’ll get an option for a “New incognito window”:

(6) Incognito Browsing (IronKey)

While Chrome’s incognito mode does seem reliable, there’s something about trusting a Google product not to pass Google data that just makes me itch. So, for my “control” condition, I used another incognito browser, a version of Firefox that runs directly off of my IronKey USB drive.

(x) Stand-alone Crawler

Originally, I was going to use a stand-alone crawler (PHP-based) as the control condition. Unfortunately, my crawlers all run out of a different state from a different C-block of IPs, so I decided to confine the test to only methods I could use directly from my office setup.

The Dry Run

I’ll discuss the search queries and metrics more below, but I initially did a dry run of 5 queries, and I ran into a couple of issues and insights that caused me to scrap that data and start over. Briefly, here’s what I learned:

Google’s Toggle <=> “pws=0”

As I was collecting data, I realized that switching Google’s new de-personalization toggle was actually adding “pws=0” to my query URLs.  If you add it manually to the URL, the toggle switches itself. Options (1) and (2) are functionally identical, so I only used the de-personalization toggle in the final test.

Queries Change Frequently

I originally ran each option one-by-one, recording the data. By the time I was done (15-20 minutes), the Google results for the control had sometimes changed. I realized that I would need to run all of the versions of each query as back-to-back as possible and then collect the data. In the final experiment, I ended up using multiple windows and 2 PCs on the same connection.

Signed Out Data Didn’t Change

There was no measurable difference between options (3) and (4) in my pilot data. Adding “pws=0” to a signed out query didn’t seem to have an impact. So, I dropped option (4) in the final test. This left 4 methods:

1. De-personalization toggle
2. Signing out
3. Incognito (Chrome)
4. Incognito (IronKey)*

The Data Set

Given the labor-intensive nature of collecting this data, I decided to use a set of 10 popular queries, pulled from Google Trends Hot Searches list for 1/17. I purposely picked popular queries so that they were more likely to be personalized and/or have social results. The point wasn’t to measure how much results are being personalized, but how well methods to remove personalization work. The query list was as follows:

1. paula deen
2. jerry yang
3. seattle weather
4. victor martinez
5. mary tyler moore
6. betty white
7. jenelle evans
8. wisconsin recall
9. wikipedia blackout
10. girl scout cookies

The original #10 on the list was “school closings”, but I decided that had too much of a local SEO aspect, so I bumped up #11. Localization is a completely different issue these days (shutting off “personalization” doesn’t shut off localization), so I decided to avoid any searches that had clear local intent.

The Metrics

To compare the SERPs across methods, I tracked three different metrics, as described below:

(1) Total Results

This was a count of all non-paid results – organic, universal, and social. News, images, and TV/movie results all counted as +1 each. In other words, if news had 3 items, it was +3. If there were 6 images displayed, it was +6. I did this for two reasons: not only are these counts variable, but Google is now mixing in social images with regular image results. For example:

Here, a search for “jerry yang” (former Yahoo CEO) shows 9 image results, but 4 of them are coming from the new social integration.

(2) Social Results

I did a separate count of social results – anything with the person icon next to it. As with total results, social image results each counted as +1. So, in the Jerry Yang example above, that set of image results would count as +9 total results and +4 social results.

(3) Ranking Change

Finally, I calculated the shift between each pair of organic rankings. This ranking “delta” could range from 0-100, and was calculated with 3 simple rules:

1. Result in same position = +0
2. Result moved positions = +|change|
3. Result fell off entirely = +10

So, if the #2 result in the control SERP ended up in #5 on one of the other de-personalization methods, it would count as +3 (change was always posit