So then the problem is really that the court clerk who referred the couple to him didn't tell them that he didn't do atheist marriages.
Yep. Unless the case is that doing atheist marriages is part of the job description - more likely worded as something like, "all legal marriages." As you said, a legislative fix.