This is already the case, albeit for other pernicious issues. See Iowa and corn for example. The EC as it is exacerbate concentrating on small state / swing state, where a bit of lobbying pay much greater return than , say, big state like california.
That flaw you describe is not alleviated by the presidential election as it is. In fact it is exacerbated, you need to lobby/go to fewer people with a bigger swing.
That's why frankly Iowa for example play a much disproportional play than say california, during campaigning. By having such disproportion between votes, you encourage what you describe. With a 1 vote== 1 voter, you dilute such small swing group to nothingness.
So if you want to avoid a small group swinging an election, the EC as it is, is actually the wrong way.