Me being free includes the possibility that I'm going to say something which is going to hurt you personally - otherwise I could not speak out of fear of hurting you, and fear is not a sound basis for happiness.
You being free, however, should also include the fact that your happiness and well-being would not be generally compromised by any individual stating their opinion - otherwise you would live in fear of someone stating what you don't agree with because it offends you personally, defying freedom.
Therefore each of us would do well to attempt to maximize our own happiness and freedom, so that both of us can speak our mind without being in fear of personal offense, never losing, however, the compassion and love and respect, not necessarily for personal views, but for each other as human beings.