I've studied quite a few languages, but only two in depth. I taught French and Spanish for 5 years in high school and college. Pretty close to native accent in both I am told.
My wife speaks Thai as her native language. Now, the hardest languages for me are Thai and Chinese. I have had a lot of Chinese friends over the years and learned a fair amount, but I am no good at the tones unless it is an expression I have memorized.
In Thailand, I used to watch CNN and pick out a fair amount, but it just fascinated me that (for example) every time the announcer said "song kram", ("war"), he would get the tone right. Crazy, I know, but as a native English speaker, it is so tempting to tie tone to emotion or question intonation etc., not meaning. Then there is something called "sandhi" which shifts tones depending on where they come relative to other ones.
I have American friends who speak pretty good Thai and Chinese. I'm jealous!
Chinese grammar is no problem, since there are no tense endings, verb conjugations, or plurals. But there is a really different approach to things like relative clauses, and the use of particles for tense and questions.
So, the hardest language I ever played with was an African language called Kanuri, from the Chad lake region. The morphology was unbelievably extensive and the phonology, while not mysterious was complicated. I don't think I could ever master this. Many of the Amerian Indian languages and Eskimo (Athabaskian sp?) are really something else. Oh, yes, Swahili...they have something like 22 noun classes or genders and all adjectives have to agree as I recall: kisu kizuri (good day)
For me the bottom line is whether you can be exposed to full immersion in a language. That is, if you hang out with a native speaker (preferably several) long enough, you will learn any language quite readily. Learning in a classroom is always much more difficult.
Currently trying to learn some Japanese. Loving it! Sugoi!!

Just something about SOV that makes sense to me. Not sure why.