When is it right to divorce your non-muslim wife. Any dalil on this?

My husband embraced Islam in 1994. He has previous marriage done in civil and church cermonies. We married in 1995, after several trials, for 2 years, he nows leaves with me and says that his marriage to his non-muslim wife is only in papers now. Is this not haram since he is not fulfilling his intimate obligations to his non-muslim wife as a husband? When I told him to divorce his non-muslim wife, since according to him he has given up hope on her embracing Islam. But he replied that there is no teaching in Islam that he should divorce his non-muslim wife even in such a situation. But if I think that I want a divorce he will give it to me since it is allowed in Islam. I find this illogical. Please explain to me in details why there is no dalil in divorcing non-muslim wife (who do not want to embrace Islam) and one can divorce the muslim wife easily. Jazak Allahu Khairun.