It turns social justice on its head. If you ask the average person what they think of when they hear the word "privilege", they'll probably tell you that it's something that, by definition, some people have and others don't (e.g. your boss at work has more privileges than you do).
I think that what social justice should really be talking about is rights. After all, you don't want to take privilege away from anybody, you want to extend that privilege to everybody. The question should be "How do we help people?" not "How do we hurt the privileged people?" (Or, more often, "How can I use this concept of privilege to win some petty argument I'm having?")