I tried to research this on Google, but the only reference I could find to "It doesn't matter" were in the context of "It shouldn't matter what sex you are..." I couldn't find it in the sense of his being dismissive of people's rights, but rather of his being affirmative of non-discrimination.

But of course, Google searches can sometimes be thoroughly unproductive--just because I didn't find anything doesn't mean that it isn't somewhere where it could be easily found by a more savvy searcher...