Though you did qualify your statement by saying "sometimes" the fact that small and large species of sloths, elephants, etc. lived contemporaneously argues against oxygen as the main factor in determining size. By the same arguement why arent minke whales as large as sperm whales or blue whales? Presumably there are not locallized areas of higher oxygen content exactly where these different species exist since in some cases their ranges overlap.
I don't really see a tendency in evolution regarding size...mammoths (my favorites) varied over time in size getting bigger and smaller depending on the time with Mammuthus columbi being absolutely enormous and M. primigenius smaller than modern African elephants.
If the argument is that dinosaurs, extinct mammals etc. are big versions of modern taxa as a result of different oxygen levels, then how does one explain that some modern species are larger than their ancestors while others are smaller..i.e. no generalized trend.