Maybe the author didn't intend for man to rule everything in the sea!
According to Richard Elliott Friedman in his book "Who Wrote The Bible?", the first chapter of Genesis is considered to be written by the same priestly author that wrote Leviticus. He states that they were probably written after the fall of the northern kingdom.
This verse shows that the writer understood that there were various creatures in the water.
Genesis 1:20-21
And God said, "Let the water teem with living creatures...
So God created the great creatures of the sea and every living and moving thing with which the water teems, according to their kinds...
Now we have the verse in question:
Genesis 1:26
Then God said...and let them rule over the fish of the sea and the birds of the air, over the livestock, over all the earth, and over all the creatures that move along the ground.
Notice we don't have rule over the sea, but we supposedly do over all the earth and all the creatures that move along the ground and the birds of the air.
Since the writer did not give man rule over the sea, he understood that humans couldn't rule where they couldn't survive.
Now look at what the writer says God allows the Hebrews to eat.
Leviticus 11:9-10
Of all the creatures living in the water of the seas and the streams, you may eat any that have fins and scales. But all creatures in the seas or streams that do not have fins and scales--whether among all the swarming things or among all the other living creatures in the water--you are to detest.
IMO, the priestly author gave man rule over where he felt they could survive and over creatures that were necessary to their survival.
A gentle answer turns away wrath, But a harsh word stirs up anger.