I'm sure you've already accomplished half your battle by now, just by pouring out your heart to Jehovah. You need to know, not what your heart wants, but what is best for the both of you. This will be difficult to find out on your own. You are undoubtedly at a point in your life where your heart will not point you in the right direction most of the time.
Since i cannot personally relate to your circumstances, i've tried to put the scenario in terms i could possibly understand better.
Lets say you have your drivers licence, you've owned a vehicle before, but met with an unfortunate accident. That car has been written off. You are now without a vehicle. You KNOW that the trauma from that accident cannot allow you to drive another vehicle for a period of time. Yet, you still want to own another car one day and drive again. . . .
So far, i would commend you for your courage!. . . That's amazing!. . Despite the tragedy that you faced, you still have the confidence and faith that you can move forward with your life and you have not lost hope or given up!. . . You are doing very well so far!. . .
Now, you come across an amazing car, in prestine condition that is for sale. You know that you want to test the car out and maybe even buy it, because you admire it. . . But you still realize that the time is not yet. You are not ready for it. The trauma hasn't worn off.
In your case though, its more like the car wants you, than the other way around.
From your comments though, you seem as though you have good control over your emotions. That's a major advantage. You know what you want. You don't want to ruin a friendship. At the same time, you want nothing more than that.
The first thing i would advise you or anyone in a similar situation to do, after making it a matter of prayer, is speaking to an elder you are comfortable with. Even if he's from your former congregation. Preferably, someone who knows both you and this brother well. You will be amazed at how helpful they can be!
The elder would surely advise you the best way of handling this situation based also on his experience. He may have experience in judicial cases, and so may even forsee things that i cannot.
Maybe, i would suggest writing your thoughts out on paper, and handing it to him in a letter form. This is where a true test of character comes into play. If this brother is genuine and considerate and truly Christian, he would respect your feelings and immediately act. He would stop phone calls and give you both space and time.
If he does not act in that way immediately, you've discovered that he does not respect your feelings. That is an immediate "red flag".
So in other words, you can really kill two birds with one stone.