Genocide is rarely the answer. Sometimes it has to be, in order to ensure the survival of a nation, but it tends to be over-applied throughout history.
To borrow a phrase from a movie, wars are fought in the will. If you can break the will of the people to fight, you've won. Breaking that will is, in fact, preferable to killing everyone, since there is a distinct tendency for people to look poorly upon genocide, and a nation that practices it runs the risk of suffering it when they finally run into someone strong enough to take them down.
In the short term, breaking the will of a nation's military will suffice. The first Gulf War is a prime example of this. The downside is that this leaves the door open for further aggression on the part of the defeated nation. Even a negotiated peace after a long war can lead to difficulties, particularly if there are harsh punishments imposed. The Versailles Treaty was viewed by many Germans as both excessively punitive, and by many others as
premature. There was a strong faction of Germans who would rather have fought on to a more advantageous settlement, and many of those people would go on to join the Nazis.
In so-called "Total War", however, the situation becomes more complicated. The concept of total war is simple: The entire nation's economy is bent on the waging of war and the production of war materials. In that sense, there
are no civilians in a total war, something that was embraced by all sides in WWII, though in many cases hidden behind euphemisms and double-talk. The British weren't firebombing German civilians, they were conducting a "De-Housing Project".
In some cases, the defeat of a nations military and the capitulation of its leadership will suffice, though the people will probably have suffered extensively by this point in a total war regardless. Japan serves as something of a prime example in this case: While the home islands were never invaded, and though military casualties were fairly high, civilian casualties were less than a third that suffered by Germany, and a something like 1/30th that of China. But by breaking the will of the government, shattering key facets of Japanese culture, and basically scaring the crap out of people, Japan capitulated with relatively little post-war bloodshed during the occupation and reformation.
On the other side, however, there are places like Iraq and Afghanistan, where defeat of the government and the armies cannot guarantee a successful resolution to the conflict. There are a variety of reasons for that, mostly revolving around the concept that in Muslim countries, there are typically two sources of authority, political and religious, and given the history of the western world, we are strangely loathe to attack the latter. When the man who claims to speak for the people says surrender, and the man who claims to speak for
GOD says fight, a populace with strong religious beliefs will almost invariably side with the latter, on at least some level.
Short of attacking the religion directly, there is, in fact, relatively little you can do to
fight religious fervor. In a widespread religion, attacking it becomes significantly more difficult, of course. Force one religious leader to capitulate, and another will decry them as a traitor/apostate/insert term here, and the war will continue largely unabated. Kill them, and they become a martyr. You are basically left with two options: Either treat it as a total war, attacking all members of a given religion wherever they might live (which is obviously extremely impractical), or simply hunker down and wait for a religion to "mellow" out some. Stomp on the local brush fires, keep the depredations of the extremists to a minimum, and hope that eventually the majority will come to their senses and police their own societies.