It's possible both sides are correct. Let me explain.
First of all, I think everyone can agree that there are a few dedicated jihadists who commit violence in the name of Islam. They are not "peaceloving people" by anyone's definition but their own. They may think of themselves as peaceloving because they think that once the whole world has submitted to the rule of Islam, the world will be at peace. But the methods they use to achieve that peace are car bombs, beheadings, and flying planes into buildings.
There are also a number of Muslims committed to forcing Sharia law on the world by rioting and the threat of riots. These are the ones who protest and riot when a Danish cartoonist publishes cartoons about Muhammad, for example. The ensuing riots killed 187 people. It is a violation of Muslim morality to draw Muhammad or to criticize him, and the violence intimidated many others in Western democracies into restraining themselves from re-publishing those cartoons, and in this way, the threat of violence enforced Sharia law on Western democracies.
The same thing happened with Draw Muhammad Day on Facebook, and with the pastor who burned a Koran. The violence and threat of violence by Muslims around the world affected the behavior of people in free countries, curtailing their freedom. The end result is the enforcement of Sharia law in Western democracies — not by changing what is written in the lawbooks, but by scaring people into doing what orthodox Muslims insist non-Muslims must do.
The people doing the rioting may, in fact, be "peaceloving people" in their daily lives in the opinion of everyone who knows them. It could be argued that everyone has a breaking point; anyone can lose their temper if the offense is great enough, and perhaps they love their Prophet or their Koran so much, that criticizing him or burning it was just too much for them to stand, so they went berserk, but really they are just normal, peaceloving Muslims in the rest of their lives.
Another sizable percentage of Muslims are dedicated to legally and nonviolently gaining concessions for Islam within Western democracies. They are pressing for halal food in public schools, pressing for Islamic limits on free speech (pressing for censorship in the media so Islam is never criticized — see some examples here). They are doing it in individual countries, and they're also doing it at the UN. The Organization of the Islamic Conference is the largest voting block in the UN and they are putting pressure on the rest of the countries to impose worldwide limits on free speech — the kind of limits Islamic law demands.
All these people working for the legal imposition of Sharia law may very well be peaceloving people.
A very large percentage of Muslims do not protest against the violent ones. Silence implies consent, usually, but they may keep silent out of fear. The violent ones are, of course, capable of violence, and peaceloving people could be afraid to speak out in protest against such violence in the name of Islam. And they may not feel that they have an ideological leg to stand on since the violence is sanctioned by Islamic doctrine and protesting against that violence is prohibited by it.
Another large percentage pay their zakat — it is a mandatory tithe to the mosque. This money is often used for charity (to help Muslims, according to Sharia law, and never to help non-Muslims). This money also sometimes goes to fund jihad. The people paying the zakat may be considered peaceloving people by most standard definitions.
Another group of Muslims are creating avenues for "Sharia finance," which also gives a certain percentage of that money to Islamic charities, some of which also fund jihad. Those who put their money in Sharia financial institutions or pay the fees could be peaceloving people, even though they are, wittingly or unwittingly, helping to finance the killing or subjugation of non-Muslims.
A sizable percentage of Muslims, according to polls, wish to have some measure of Sharia law, including things like Islamic limits on free speech and the death penalty for apostates (Muslims who leave Islam). In some places, a majority of the Muslims feel this way. But they do not commit any violence themselves and would be considered by many as peaceloving people.
When Muslims immigrate to Western democracies, they often form "enclaves" — whole areas where primarily Muslims live. The larger the number of Muslims in the area, the more hostile some of them are to the non-Muslims living there, so those non-Muslims move away. More and more Muslims move to the area until it becomes, for all intents and purposes, a small Muslim state within a Western democracy.
These enclaves are creating "no-go zones" where legitimate law-enforcement officers are reluctant to go, or where legitimate government authorities bend to the Muslims' demands (for fear of violent reprisals). There are more enclaves and no-go zones in Western democracies with every passing year in Sweden, France, Germany, and many other European countries. The United States just got its first Muslim enclave.
Wherever Muslims gain a sizable majority, the most dedicated among them begin pushing for local manifestations of Sharia law.
But it would probably be correct to say that most of the people who move to a Muslim enclave from a Muslim country are peaceloving people. They are just families who are moving to an area where they have relatives, and they want nothing more than to raise their children and be happy.
Let us assume they don't know much about Islamic doctrine, and even if they do, they have chosen to quietly ignore the violent or political parts of it. They are still unwittingly helping to accomplish Islam's prime directive in many ways — they are helping those who are actively trying to convert Western democracies into Islamic states — even if they don't mean to.
Muslims around the world have lots of children. Some of them immigrate to Western democracies and go on welfare, so the raising of their children is being paid for by the non-Muslim taxpayers. But most of these people are probably not violent. They raise their children, telling them that they are Muslims and that the Koran is the word of Allah, but they don't explain to their children the political mandates of Islamic doctrine.
When the kids become teenagers, some of them are susceptible to recruitment by the more orthodox (politically active or violent) Muslims because the teenager has already been primed — a primary identity they have is "I am a Muslim" and the recruiter only has to say, "read your Koran and discover your obligations." And so we see that second-generation Muslims in Western democracies are more likely to become jihadists than their parents, even though their parents are peaceloving people.
This is another way peaceloving Muslims are unwittingly helping jihadists accomplish their mission.
Another couple of groups I should mention are Muslim leaders and oil billionaires. There are quite a few prominent Muslim leaders who exhort their followers to pursue Islam's prime directive. These are not isolated leaders with little influence and few followers. These are heads of state and influencial people with huge numbers of followers (read more about them here).
And there are Muslim billionaires (primarily Saudi Wahhabis) who are pouring their money into building mosques and madrassas all over the world. They fund 90 percent of the world's Islamic institutions. Unfortunately, they are promoting Wahhabism, which is a branch of orthodox Islam — dedicated to jihad; dedicated to Islam's prime directive; dedicated to eliminating all democracies and establishing Islamic law for all people. This is not as impossible as it sounds. The world is far more Islamic today than it was even 20 years ago.
These oil billionaires have built and maintain most of the mosques in the United States and Canada, for example, and 80 percent of these mosques are actively promoting jihad (read more about that here). Promoting the violent overthrow of the government by jihad or any other means is against the law, but it is overridden by the protection of religious freedom. Because the jihad they preach is not extraneous to their religious teachings, but inherent in them, freedom of religion has protected them.
The oil billionaires and the Muslim leaders may have never done anything violent in their lives, and may only want a peaceful Islamic world, so they may be "peaceloving people" by most peoples' definitions.
One final idea we should consider is that it doesn't take a majority to cause serious trouble, which means that if the majority of Muslims are peaceful, it would be irrelevant (read more about that here).
So let's get back to our original question: If the vast majority of Muslims are peaceloving people, do non-Muslims really have anything to worry about?
Yes we do.