Yes, in the sense that we will have to wait an infinite amount of time to see light from the whole thing.
The confusion here is that whether something is infinite, finite, or a point can depend on how you choose to measure it. Our everyday intuition about volumes and time is just completely wrong at a singularity.
One way to define distance to a point is by how long you have to wait before light from that point reaches you. In that case, the big bang singularity in a flat or open universe is infinite. However the same definition says the horizon of a black hole is infinitely far from any point outside, and that's not very nice.
Another way to define it to take some group of stars today - say a galaxy - and trace it back in time, and ask how much volume those stars (or the particles which formed them) occupy when you get to the bang. The answer is zero - everything in the group collapses down to one point - which might lead you to say the singularity is a point.
Really you just need to write things down mathematically, and then you can figure out the answer to any definite question like that.