I believe the outcry was about women who are victims of abuse and rape not having a voice. Please remember that a great majority of rape victims never tell anyone. They have no voice, no representation.
Celebrities, not just Allen, have a very big voice. A platform. A public persona. Almost no woman has that.
Free speech, I agree is vital. We should give voice to all. Yet, they who have no voice, have little recourse except a form of boycott, or protest. They will never have a book deal, or a major motion picture.