Dear Dharani,
You have rejected offer from Company A. However, did you accept the offer letter then decided not to join? If yes, then were any conditions mentioned in the offer letter for turning down the offer after acceptance of the offer.
Secondly, before accepting the offer letter from Company A, did you make any deal, verbally or otherwise, with the middle-person? If you have not agreed to pay any brokerage, then he does not have any locus standi in this matter. The matter is between you and Company A. Acceptance of rejection of offer is absolutely either party's call. Third party can only facilitate employer-employee relationship and nothing else.
If the so called mediator starts troubling you then I recommend you recording telephonic discussion with him. Tell him politely that there was no deal as such to pay him any money, therefore, he should call communicating with on phone or through any other media. If he persists with his calls then you will be forced to lodge complaint at the police station.
Lastly, about the black listing your candidacy. This is a pure sterile threat. Do not attach much meaning to such threat.
Thanks,
Dinesh Divekar