It is really embarrassing unless you get a no-objection certificate from the parent department; you cannot join any other organization. In the first case, while you are applying, they might have demanded a no-objection certificate. In case your age and educational criteria do not require a no-objection certificate and you have applied directly to the PSU, you should have properly informed the parent department (GOVT.) to get yourself relieved. Otherwise, the Government department has every right to initiate a departmental inquiry (ex-parte if the situation warrants) to terminate your services, clearly notifying you are unfit for any other job by termination/removal/dismissal or according to whatever criteria.
Furthermore, it is not known why the Government authorities have not initiated any action for this lengthy time period. Regarding your entry into a 4-year bond, the bond copy should have been endorsed to the Government authorities in the department where you have served for annual service entries. Your salary should have been adjusted according to the present status, and service entries should have been made for these four years. Being a gazetted officer, the authority responsible should have been informed to further your service. In this regard, please review the service rules, CCS conduct rules, CCS pension rules if you are eligible, confirmation of service, deputation, or otherwise relief from the parent service and rejoining after foreign service. All these service rules should have been adhered to.
Best of luck! Initiate now; it's better late than never.