group by select statment

I have the following table:

select '1' as OrgKey,
  '123' as PolNum
into #temp
union
select '1' as OrgKey,
  '234' as PolNum
union
select '2' as OrgKey,
  '456' as PolNum
union
select '3' as OrgKey,
  '678' as PolNum

Go

select *
from #temp

OrgKey PolNum

1          123
1          234
2          456
3          678

I only want to return the OrgKey and the PolNum of the records where the org key only shows up once.  So in the above example, I only want to return back the rows for OrgKey 2 and 3.  The place I am getting stuck is return both the OrgKey and PolNum fields.  I tried grouping on OrgKey, but then I lose the PolNum field.

thanks

Scott