We have no idea if some creatures existed in ancient or prehistoric times who never fossilized for one reaosn or another. We should never concluded that something NEVER existed simply because there is no evidence it ever existed. Do you disagree with that?
Not true. We know what the skin of various dinosaurs was like, and whether some early bird-like proto-birds had feathers or not, because they die and fall into a mud flat, and the imprint of their skin then fossilizes.
So in some instances, it is indeed possible to detect creatures and features that otherwise would not fossilize by themselves.
Wood also rots away, but we've no shortage of petrified forests.
Of course this race of "giants of old, men of reknown", was just a leftover tale from some earlier religion that was still floating around in popular consciiousness back then, and hence the writers of the Bible felt they had to account for it, either by direct inclusion, or, as more likely, by excising as much as was necessary to bring the earlier works into alignment with the current theology.
Depending on the fussiness level of the Christian apologist, they are not the leftovers from when Yahweh's Canaanite pantheon members mated with mortal women, but (since Yahweh's pantheon was converted to a pantheon of size = 1) they are either children of mortal women and angels, or just really big but otherwise normal humans.
But in no circumstances may a Christian interpret that as Yahweh and his existing god-buddies pounding on some lucky human females.