Optimizer costings for hash join versus nest loop

I have been looking at a query that involves a join between two reasonably large tables. The optimizer picks a hash join rather than a nested loop, but when I force a nested loop it uses significantly less CPU (although a lot more I/O). The elapsed time is far better for the nested loop join - so I am trying to understand how the optimizer calculates the difference between these plans and ends up selecting the poorer plan.

The example I created involved the following scenario:

* A larger table with about 8 million rows in it, and a non-clustered index on three columns that are going to be used in the join, as well as the column used in the aggregate (see below)

* A smaller table with 80,000 rows in it (1% of the larger table), with a unique clustered index on the three columns used in the join

The query is as follows:

SELECT max(l.colZ), l.colA, l.colB, l.colC
FROM SMALLER s
INNER JOIN
LARGER l
ON s.colA = l.colA
AND s.colB = l.colB
AND s.colC = l.colC
GROUP BY
l.colA, l.colB, l.colC

The query plan that is picked uses a hash join. This has relatively low logical I/O on the larger table - 28,192 but the CPU time is over 7 seconds. On my 8 core laptop this results in an elapsed time of around 1800ms. If I force a loop join, the CPU time is just over 600ms with an elapsed time of just under 800ms. The logical I/O is far higher with the nested loop - 254k but even if I drop the buffers and force physical I/O, the loop always results in a lower elapsed time than the hash join.

The real-word production example from which I extracted this sample was even more marked - in that scenario the large table had over 92 million rows and the difference between a loop and a hash join was 25 seconds.

I am trying to understand what might be going on here? The subtree cost in the plan of the hash join is far lower than the subtree cost of the nested loop, yet the actual CPU time suggests the optimizer is completely wrong here.

I am running SQL 2012 SP1 11.0.3349.0. I have also tried this on SQL 2008 with the same result.

Any insights most welcome.

Thanks,

Robert