The symbolism of the cross of St George especially has loooooooong been co-opted by the far right, neo-fascist groups, racists and the like, as I'm sure a keen student of 20th century UK history, such as your good self, will be fully aware, let alone its symbolism going as far back as the Crusades or the Unionist version in the Six Counties with the Red Hand of Ulster in the middle.
There are many, many, many very unsavoury connotations. The Union Flag, less so, but there can still be some.
It is nowhere near as simple as "the British or English flag".