8 people should not unilaterally decide the fate of thousands of people.
Supreme Court of the US does not have 8 justices. It has had 6, 7, 9, 10, and FDR tried to make the maximum 15. It has been 9 since 1869.
Further, it is in their power to affect thousands of people. In the Dred Scott case they affirmed slavery. In Brown v. Topeka they destroyed segregation. All due to issues of constitutionality, which thanks to the Marbury v. Madison case is their prime directive. You also need a final court for appeals, so even if they didn't decide on cases of constitutionality, their rulings on certain laws involved in certain cases set precedents that all lower judges may and do use, so their power is inherent.