Dought & clear, - It is essential to respond to those who defame the Prophet (peace andblessings of Allaah be upon him).

There is no one among us who is unaware of what the Christians say
defaming the Prophet (peace and blessings of Allaah be upon him), and
we are not unaware either of the gheerah (protective jealousy) of the
young men of the Muslim ummah towards their religion and their Prophet
(peace and blessings of Allaah be upon him). Is it permissible to
respond to those who defame the Prophet (peace and blessings of Allaah
be upon him) by insulting the speaker, knowing that I insulted one of
them and some of my relatives advised me not to do that again, because
it will make them defame and mock him even more, so their sin will be
on me?.
Praise be to Allaah.
Defaming the Prophet(peace and blessings of Allaah be upon him) is a
kind of kufr. If that is done by a Muslim then it is apostasy on his
part, and the authorities have to defend the cause of Allaah and His
Messenger(peace and blessings of Allaah be upon him) by executing the
one who defamed him. If the one who defamed him repents openly and is
sincere, that will benefit him before Allaah, although his repentance
does not waive the punishment for defaming the Prophet(peace and
blessings of Allaah be upon him), which is execution.
If the person who defames him is a non-Muslim living under a treaty
with the Muslim state, then this is a violation of the treaty and he
must be executed, but that should be left to the authorities. If a
Muslim hears a Christian or anyone else defaming the Prophet(peace and
blessings of Allaah be upon him) he has to denounce him in strong
terms. It is permissible to insult that person because he is the one
who started it. How can we not stand up the Prophet(peace and
blessings of Allaah be upon him)? It is also obligatory to report him
to the authorities who can carry out the punishment on him. If there
is no one who can carry out the hadd punishment of Allaah and stand up
for the Messenger(peace and blessings of Allaah be upon him) then the
Muslim has to do whatever he can, so long as that will not lead to
further mischief and harm against other people. But if a Muslim hears
a kaafir defaming the Prophet(peace and blessings of Allaah be upon
him) and he keeps quiet and does not respond for fear that this person
may then defame him even more, this is mistaken thinking. With regard
to the verse (interpretation of the meaning):
"And insult not those whom they (disbelievers) worship besides Allaah,
lest they insult Allaah wrongfully without knowledge"
[al-An'aam 6:108],
this does not apply in cases where they defame Allaah and His
Messenger(peace and blessings of Allaah be upon him) first. Rather
what is meant is that it is forbidden to insult the gods of the
mushrikeen first, lest they insult Allaah out of ignorance and enmity
on their part. But if they insult Allaah and His Messenger(peace and
blessings of Allaah be upon him) first, then we must respond and
punish them so as deter them from their kufr and enmity. If we leave
the kuffaar and atheists to say whatever they want without denouncing
it or punishing them, great mischief will result, which is something
that these kuffaar love. No attention should be paid to the one who
says that insulting or responding to insults will make him more
stubborn. The Muslim has to have a sense of protective jealousy and
get angry for the sake of Allaah and His Messenger(peace and blessings
of Allaah be upon him). Whoever hears the Prophet(peace and blessings
of Allaah be upon him) being insulted and does not feel any protective
jealousy or get angry is not a true believer – we seek refuge with
Allaah from humility, kufr and obeying the Shaytaan.
And Allaah knows best.
Shaykh 'Abd al-Rahmaan al-Barraak, Majallat al-Da'wah, Muharram, issue no. 1933.