It's really important that you stop right there and make the distinction clear. Islam is an ideology. It is written. Muslims are people. Some of them know a great deal about Islam, and some of them know almost nothing, and everywhere in between.
So when you notice someone talking about Muslims in response to you talking about Islam, say something like this:
I wasn't talking about Muslims, I was talking about Islam, the ideology. Speaking freely about religions and political systems is one of the things we do in a free society. We compare and critique philosophies, religions, political doctrines in the hopes that our discussions will bring us to better clarity and understanding.
You will rarely find a person who disagrees with that.