Yes and no.
Recapitulation theory, as Haeckel imagined it, has been rejected by science. Haeckel believed that each embryo went through clearly defined stages of evolution, resembling a fish, a reptile, and so on, gradually moving through the evolutionary history of some species. This view was not accurate.
However, you can see many clues for a species evolutionary history in the development of the embryo. As you have rightly pointed out, many features that evolved early on are also the first to develop in the embryo. And many modern features develop from what resembles more ancient features in the embryo (like pharyngeal arches, which resemble gill slits).
The reason for this is fairly obvious. We know that evolution changes the morphology of a species. Humans don't change much after embryonal development, and neither do most tetrapods for that matter. So what evolution is really modifying is the development of the embryo. Since evolution isn't an intelligent designer, it won't go back to the drawing board everytime a species evolves a new function. Instead, evolution alters structures that the organism already has. If embryos are already developing gill slits, it's not difficult for modifications to turn those developing gill slits into something else useful.
Evolution doesn't care as much about what stages the embryo passes through as it does for the final product. If natural selection favored human ancestors who lost their tails, it only selected for the end product. If the embryo stayed true to the mold set up by many earlier generations with tails, but then lost it before it completed development, natural selection would still favor it.
Now for the disclaimer. I'm not a biologist, so I'm sure our resident scientists will explain it better for you, or point out inaccuracies in my reply. But this is my impression of how things work.
Respectfully,
-Meldinoor
Edited by Meldinoor, : Corrected typo
™ Version 4.2
