Although Bilam failed in all his attempts to curse Bnei Yisroel, he succeeded in causing them to sin and thereby caused their downfall. He suggested that the Midianite women seduce Bnei Yisroel to sin with them. Bilam's idea was so successful that even Zimri, one of the heads of the tribe of Shimon, was seduced by a high ranking Midianite's daughter. Pinchos saw what was transpiring and he said to Moshe, "You taught us that when one lives with a gentile woman, a zealous person is permitted to kill him."
Rav Wolbe (Shiurei Chumash) defines what exactly zealousness is. He notes that we may behold a person who rants about the depravity of others while deep down he himself harbors an interest in those very same depraved actions! Chazal describe this idea with a parable.
There was once a man who had a servant who was a drunkard. One day the master passed by a different drunkard who was rolling in the mud as many children stood around taunting him. The master thought that if his servant would behold how a drunkard looks after too many drinks, the servant would certainly be cured of his alcohol abuse. He quickly ran home and brought his servant to the scene of action. When the servant saw the drunkard in the muck and mire, he ran over to him and asked him where he got such good whiskey! The servant did not even notice the drunkard's disgrace because all that interested him was the alcoholic beverage that had been imbibed. In a similar vein, it is very possible that someone might scream about another's misdeeds, not because the actions disturb him, rather, because he himself identifies with those very actions. It is this desire for these actions which prompts him to discuss them at any given opportunity. Although this person decries the improper behavior, his reaction is not zealousness at all.
Zealousness is the middah of a person who truly cannot stand evil simply because it flies in the face of the Torah. He who truly despises immorality would be permitted to kill a Jew who lives with a gentile woman. The transgression bothers him to such an extent that he is willing to take action without the authorization of beis din. Moreover, if he were to ask, beis din would not tell him to kill the perpetrator. He has to have the internal drive to get rid of the evil without being told to do so.
Rav Wolbe continues that sometimes we denounce another person's actions, not out of zealousness but out of jealousness! Deep down there is a part of us that wishes that we would be able to do what he did. However, we are too embarrassed to agree with what he did, so instead we feel a need to denounce it. This in no way resembles the zealousness that led to the covenant of peace that Pinchos merited with his actions.