Still, brutal to get the hang of; hours sometimes to translate one sentence; and to speak it!?

When people ask about the collapse of the Roman Empire, I sometimes wonder if it wasn't due to sheer exhaustion from speaking Latin.
Well, either that or too many orgies.
I was under the impression that it's generally accepted that almost nobody actually spoke Classical Latin as it was written and as it survives today, probably for the reasons you describe. Rather, it was a sort of
lingua franca (heh, there's your Latin) for the educated classes, especially as dialects in spoken language were likely to differ from one end of the Empire to the other.
More or less the same is true of classical Chinese in Japan; literati for the best part of a millenium wrote government documents, stories and poetry in Chinese while being completely unable to speak the language in any form recognizable to anyone in China.
Anyway, re: Japanese and its relative difficulty...true, the writing system is probably one of the most ridiculous ever devised. But then, as the above examples show, you have to differentiate the
language itself from the
writing system used to denote it, because they're not the same thing. OK, this is splitting hairs somewhat and purely academic from the point of view of someone learning Japanese today, I admit. But I reckon you can (and, to their eternal discredit, some long term expats do) speak about 70% fluent Japanese without being able to read it at all.
Long story short, while there are certain extralinguistic factors that can complicate things (like honorific or humilific language), I don't think the Japanese
language itself is that much harder than a lot of European languages (it's not tonal, it has two tenses, no verb conjugation or noun declensions, no plurals, no gender, adjectives don't have to agree in number and gender, etc). But then of course, there's those damned kanji...