Posted by Tom Anthony

Measuring CTR data in search engine results is notoriously difficult, and with Google's recent move to HTTPS for logged in users it's only going to get worse.

The problems include, but are not limited to:

• How can you record the clicks?
• How can you know what position you were in?
• What snippet was shown?
• What did the other entries look like?
• What ads were shown?

There are so many factors and no good way to gather the data meaning that the signal-to-noise ratio basically makes the exercise worthless. Furthermore, the delay in making changes (trying a new title, for instance) and getting data is simply agonizing.

What I wanted was a simple way to measure the change in CTR for a given search query's results when I adjusted entries, but nothing existed.... so I built it. I think I got pretty close to what I wanted; it isn't perfect but it is quick, cheap and the signal-to-noise ratio is the best I've seen (certainly for the price!). Here I show you how I tested it, and how you can use it for your own tests.

Introducing SERP Turkey

My plan was simple:

1. Build a dummy search engine page.
2. Create multiple instances of the SERPs for a given keyword.
3. Push Mechanical Turk users to these pages and measure the clicks.
4. Examine analytics. Be happy.

Basically, SERP Turkey is what I came up with. It allows you to enter a keyword for a search, import the search results from Google for that search and then edit each entry's title, description/snippet, display URL and re-order them as you see fit. You can create multiple variants of the SERPs for split testing, or you can just keep to one and measure the CTR distribution. You can then take your test link and either share it with a pool of testers, send to your friends on Twitter, or do what I did and send it to Amazon Mechanical Turk. (If you don't know- it mTurk is a service that allows you to push simple 'human intelligence tasks' to a workforce of thousands, who you pay a few cents a time to complete your task.)

Each user who visits the test will then be shown the dummy search page and a randomly select variant from those you created, and their click is recorded. You can then examine (and download as CSV) the CTR of each entry for each variant and hopefully draw some conclusions from it. You can run tests that gather results from 200 users for as little as about $10 and the results will be in within 2-3 days.

Before we move on, here is how the dummy search page results look for a test:

You can visit this test page for yourself right here, if you want - feel free to click a result and see what happens. :)

You can see that the navigation links, and adverts that a user would expect to see on a search engine results page are there, but they are blurred out so as not to attract clicks or distract the user. Overall the page looks pretty much like the results pages that a user would be used to seeing. There is some instructional text and a message at the top to make clear that this isn't a real search engine (which would be against Mechanical Turk rules).

So far, so good....

But Tom... are these clicks going to be reliable?!

This was the first thing that I wondered about. Will mTurk testers or other testers (co-workers, Twitter users, or anyone else) really be motivated to do the test properly? Won't mTurk users just click the top hit to collect their payment?

With regards to mTurk, you'll find that most workers do pay attention (not all, but most) because you have to approve their work and their 'approval rate' is a criteria that can bar them from getting more work.

However, that wasn't good enough for me - I wanted data to be sure, so I ran a sanity check test...

I ran a search for 'sharks' and imported the results into SERP Turkey. I then ran a search for 'great white sharks' and I imported the top two results and placed them in positions four and six of the 'sharks' results. I setup the SERP Turkey results page to show that the search term was 'great white sharks', however, the results showed were the 'sharks' results with my two more relevant results inserted.

This is how it looked:

I pushed this out to Amazon Turk and gathered some results to see whether, as I hoped people would click the two relevant results.

I won't keep you in suspense; here is how the results look in SERP Turkey:

(click to enlarge)

The results on the left show the raw clicks (first click per user only - if they went back and clicked a second result it is ignored), and the results on the right showed the results when those faster than five seconds are filtered out. I knew some users wouldn't look properly and I found five seconds a good threshold for filtering out people who just clicked without really looking (you can view any time threshold you want).

You can see in both cases that over 65% of the clicks were focused on the two 'most relevant' results. In both cases the Wikipedia page for 'sharks' in position number 1 also attracted a lot of clicks, but it is also a relevant result and I imagine that it mimics real search results in some sense (it is in position 1, it is wikipedia, it isn't irrelevant).

Conclusion: The point of the experiment was to demonstrate that test users, on the whole, examined the results properly before making a decision. What we found was exactly that - users do seem to pay attention and hunt out the most relevant results.

This experiment involved 200 Amazon Turk users who I paid $0.05 each. When filtered I used 174 data points, as shown above. Total cost to me, with Amazon's fee, was only $11! It took about three days to gather the data- but this could be sped up with a higher bid, if you're in a rush. You can run multiple tests at the same time too.

Test 2: Does Wikipedia really get a higher CTR? Obama lets us know...

So now it seemed the tool worked I wanted to take it for a test drive, and test the split-testing part of the tool. I decided I'd test to see whether just being Wikipedia really is enough to overcome your position. Would a Wikipedia entry in position 3 beat out a relevant entry in position 2?

I ran a search for 'Barack Obama' and imported the results into SERP Turkey. Wikipedia was in predictably in position 1, but I didn't want the fact that many searchers often just click the first result to interfere too much with my experiment. So using the power of the Turkey, I created two variants; the first had the Wikipedia entry in position 2 and the second had the Wikipedia entry in position 3. Here is the first variant:

You can see the top four results are all pretty relevant. You can see the test page for yourself here. Feel free to play around.

I pushed it out to Amazon Turk again, and the results came in:

(click to enlarge)

On the left we see Wikipedia in 2nd, and on the right we see it in 3rd with whitehouse.gov taking the other slot.

Despite whitehouse.gov being a very relevant link, sure enough Wikipedia does overcome being in 3rd position to still garner 1/3rd of the clicks - doubling the whitehouse.gov in position 3.

Another interesting result we see is that when Wikipedia is further from the top of the results it seems the user is more inclined to continue searching yet further down the results, and result number 4 begins to see an uptake in clicks.

Conclusion: It seems that Wikipedia does command additional CTR just for being who they are.

Bonus Conclusion: From a single experiment with so few data points (118 users' clicks are included above) it is hard to draw an accurate conclusion as to how other categories of se