my understanding is Neanderthals had larger brains and a side view limits some aspects as well
Larger by volume, but not up/down height. A side view of the skull is actually better to evaluate the Neanderthal size. However, with the flesh on, it's pretty hard to make anything out.
The basic problem is we could find 3 people today, line them up, dress them funny, one shaven and another not, and create a similar line-up.
Of the first two, certainly. Of the Neanderthal? I dunno know. There are a lot of people out there, some pretty weird looking, but no forehead, no chin is a tricky find.