Well, I don't know of any archaeological evidence that people even in 'pre-flood' Mesopotamia (5000 years ago?) would have averaged only 3 feet tall. People even back then were actually averaging around 5'3 or 5'5 for men, but certainly I could see how a solidly built 7' man could very well seem twice as tall as an average guy at first glance.
When the bible mentions king Og or Goliath, it does seem there are true historical kernels, but extraoplated and condensed into the Hebrew chronicles. The giants are always the bad guys as it were.
Based on the earliest readings of Samuel, Goliath was infact around 6'7 and wore 100 lbs of Mycenaean style armor. The Philistines seem to have been of a taller Anatolian or Aagean stalk, perhaps averaging 5'10 or 6 feet in stature. The king Og described in Deutoronomy and Numbers, seems to represent a conglomerant of Aramean and Ammonite traditions concerning an ancient Bashanite ruler of great stature whome the Israelites later added locale lore to. The "Iron Bedstead" may have referred to his ancient Tomb or Sarcophagus (13ft long) was known during the time Deutoronomy was compiled c.700 bc.
Whatever the case, these giants seem to have grown throughout the centuries, capturing the imagination of biblical readers by their brute strength and unimaginable wickedness. But in modern terms, I'd just say that the ancient Jews were simply prejudice against their taller neighbors who may have been of caucasoid or Dinaric bodybuild which could account for a 5-6 inch height advantage above the shorter Semites. Whatever the case, height was a respected quality in ancient times. King Saul was "head and shoulders" above average people (Sam. chpt.9) and this was evidently a noble and distinguishing quality.
The names, Anak, Rephaim, and Nephilim, do seem to echo ancient Canaanite traditions of a tall and strong warrior class in the ancient past who were later immortalized and re-introduced into ancient Hebrew history. All of this is very intriguing to hypothesis though.