I get what you're saying, and you're probably right in terms of free speech impacts. That wasn't really my intent. Right now I think parties are a problem - not the concept, but the effect.
I'm very frustrated, because I'm not partisan and I've always been independent. I can't vote in primaries because I'm not registered with a party. In pretty much every cycle I can remember, there were candidates from each party that I thought would be good leaders, but they didn't make it through primaries. The reality is that a fairly large portion of Americans are politically homeless, and neither of the dominant parties are fielding candidates that meet the needs of normal citizens very well.
I think the system of political parties, and the way in which they're run and organized, is a barrier and a problem. I don't care if people associate, but I really dislike the closed off nature of parties, and the way in which they end up forcing people we don't like as the only choices.
FPTP is a HUGE part of that problem. Closed primaries is also a big part of the problem. I don't have a perfect solution, I just know that this isn't working very well.