We have no idea if some creatures existed in ancient or prehistoric times who never fossilized for one reaosn or another. We should never concluded that something NEVER existed simply because there is no evidence it ever existed. Do you disagree with that?
It depends upon the time frame and the numbers of the critters in question.
Fossilization events are rather scarce and hard to come by, usually river deposition is the best way. Although swamps do a good job.
There is evidence outside of traditional fossilization, peat bogs and tar pits come to mind where preservation may extend to fossilization through mineralization of bones but not necessarily. Peat bogs, caves and glaciers are a really good way to preserve things if the conditions occur properly.
Yet, if there was a really small population that was not widespread then it is possible for a species to arise and no preservation to occur.
However, the stories of giants (except for the Duatha D'ann, who became smaller after becoming less dominant, they became ants) would have required human contact for the stories to be transmitted, archaic homo sapiens sapiens is about 150-60,000 years old, truly 60 or less for gracile homo sapiens sapiens. Their bones have been found unfossilized in several situations in dry caves.
So the outside window for the existence of giants and humans to co-exist if 60,000 years. We have stone tools made by gracile homo sapiens from that time period.
Why haven't any remains of the giants been preserved from that time period, if normal bones and tools can be preserved, without fossilization, from that time period?