This is an easy objection to answer, but it is also an opportunity to give your listener a deeper education on the subject. Here are a few ideas of how to answer this objection:
1. Terrorism is only one of many ways to wage jihad. There are at least ten types of jihad (I recommend you memorize this list). In Islamic teachings, there are five pillars of Islam, five things every Muslim should do. But according to Mohammad, jihad is more important than any of them. It is a religious duty for each Muslim to struggle for the establishment of Sharia law everywhere in the world. Some do it with bombs. Some do it with immigration and fecundity. Some do it with relentless political actions (waging jihad by gaining concessions). Some do it with "mainstream, moderate" Muslim organizations that try to undermine Western governments. So in other words, I agree with you completely that not all Muslims are terrorists, but I disagree with you that this somehow implies the problem of Islam is something we can safely ignore. We need to know about it or we will have no ability to protect ourselves from it.
2. That's true: Not all Muslims are terrorists, but most terrorists are Muslims, and they kill in the name of Islam. Do you know why? Do you know what they're after? (When they say something like, "Yes, they want the West to stop interfering with Islamic affairs," you can answer with a lesson on Islam 101.) Since the beginning, Muslims have waged a war of expansion, and they have tried to justify their violence as a response to a grievance (read more about that here). That's the way Mohammad did it, and he's the example all Muslims forever after are supposed to follow. They want only one thing: For everybody on earth to submit to Islamic law. Fighting toward this goal is a religious duty for Muslims. And terrorism is only one of many ways to wage jihad.
3. That's true, but most Muslims believe Mohammad is a good example to follow. They believe this because it says 91 times in the Qur'an that every Muslim must follow Mohammad's example. Do you know anything about Mohammad? Knowing about Mohammad explains a lot of what otherwise is incomprehensible about what is going on in the world. (Here you can tell the story of Mohammad's rise to power and the change in the Qur'anic revelations. Talk them into reading the Qur'an. Recommend a readable version like this one: An Abridged Koran.)
4. The Muslims who are terrorists are able to do what they do because of a tremendous amount of support from their community, and that support is motivated by Islamic teachings. It is also motivated by the hope that the supporters will gain entry to Paradise. The martyr can plea to Allah (once he arrives in Paradise) on behalf of up to seventy of his relatives to get them a ticket to Paradise. For those who believe this, it is a tremendous incentive to help any of their relatives who plan on killing non-Muslims. In other words, that "small percentage" of Muslims who are active terrorists are only the tip of the iceberg of a tremendous amount of popular support for the killing of non-Muslims (and other forms of jihad). Remember the jubilation in the Muslim world when thousands of non-Muslims were killed on 9/11? Only 19 hijackers did it, but clearly millions supported it. Millions. Maybe hundreds of millions. Why? Because that's the kind of thing Muslims are supposed to do according to the Qur'an and the "perfect" example of Mohammad (read more about that here).
Make your answer to the objection as cleanly and as calmly as you can. Try to be almost casual about it. Never be intense if you can help it. Understate rather than overstate your case. Speak accurately. Never exaggerate.
We must successfully persuade people. We cannot afford to fail. Make your conversations count.