Sql server bugs? Query result isn't determined when group by expression?

I have the following query

with cte1 as (
    select isnull(A, 'Unknown') as A,
           isnull(nullif(B, 'NULL'), 'Unknown') as B,
           C
    from   ...
    group by isnull(A, 'Unknown'), isnull(nullif(B, 'NULL'), 'Unknown'), C
    ),
    cte2 as (select top (2147483647) percent A, B, C from cte1 order by A, B, C),
    ctes as (
    .... -- pretty complex query joining cte2 multiple times
    )
    select count(*) from finalCTE

The result change every time when it's executed. And it's much less than the number it should be. I found any one of the following steps can make it right.

1. Materialize (temp or permanent table) the CTE cte1 and use the materialized table instead.
2. Change the group by in cte1 to any of the following forms.
   • group by A, isnull(nullif(B, 'NULL'), 'Unknown'), C
   • group by isnull(A, 'Unknown'), nullif(B, 'NULL'), C
   • group by A, nullif(B, 'NULL'), C
   • Use cte1 stead of cte2 in other CTEs. (Update: This step doesn't always work. Still has the problem when it's in a table function, although it works if run the SQL directly)

However, why the original query behave strangely? Is it a bug in SQL Server?