The pope is only infallible when he explicitly invokes the infallibility, making an official pronouncement ex cathedra. That means he's been on the phone with the Holy Spirit

and absolutely that's the truth. Anything else, is just his personal reading of the Bible and opinion. A very educated opinion, mind you, but still just his opinion.
They're actually very careful with what they invoke infallibility about, because it's exactly what it sounds like. Once an infallible pronouncement ex cathedra has been made, it can't even be undone by another pope. You can't use your own infallibility to override a predecessor's infallibility. (Though you can override a previous pope on something that was just his educated opinion.)
It's only been used 7 times total, in 449 AD, 680, 1336, 1653, 1794, 1853 and 1950. And technically only for the last one a formal and official definition and dogma of papal infallibility existed. Though it had been kinda assumed all along that a pope could do that.
So, yes, it's not something the pope does every day. In fact most popes had never been infallible in their whole lives.