I only drove LA-SF once, we took the PCH to SF because we had lots of time and it's a great drive. Back to LA we took I-5, and there was no traffic at all outside the SF and LA metro areas. I remember truck after truck after truck of garlic on the road near Gilroy on the way up!
But you hit on the major issue - traffic congestion isn't in inter-city travel, it's in the intra-city travel. And HSR doesn't address that at all. In fact, it uses dollars that could be going to maintaining and expanding existing light rail and bus routes. Here in Chicago numerous sections of the CTA rail service are slow zones because of track maintenance issues (parts of it are 100 years old), and service frequency is being cut. Bus routes are also being cut or even eliminated. And these are the things that actually address traffic congestion!
But hey, Ilinois got $2 billion to make our existing Amtrak route few people use slightly faster in some sections!