That is the basis, is it not? When a man saw a woman could not carry as much, say in paleolithic times, this may have been sparked. Today, the discrimination is based solely on difference in a world where how much you can carry doesn't count.
But patriarchy is much more than just a gendered division of labour, whether that division is based on physical differences or not. If in palaeolithic times men could carry more than women, then it might be sensible for... erm... men to carry more than women.
But the important feature of patriarchy is the fact that women's work is considered dirty work, skill-less work, low-value work, embarassing or humiliating work, etc.
i don't see how this cultural view of women's place is society would arise just from different physical abilities of women and men. Differing physical abilities might lead to a rational division of labour, but not to male control of female sexuality; men's control over the behaviour of children; and differing legal status for women and men.