Hi, to answer your question, yes it is almost certainly head gasket.
It does not necessarily go milky, it depends on how it goes.
To answer your second question, yes it might also be cracked cylinder head, but either way you will need to strip her down to see.
Before you do that, what are...