Dear Sandeep,
Holding of domestic enquiry into the charges against the delinquent employee is an important and inevitable aspect of disciplinary proceedings in tune with the Principles of Natural Justice. Therefore, it cannot be skipped very particularly when the charges are grave in nature warranting the maximum penalty of dismissal.
However, there can be exception to the above rule. Under certain circumstances, continuation of the enquiry proceedings may not be required in the interest of justice and fair-play and even it could be waived of in judicial scrutiny on the merits of the case. Such rare among the rarest circumstances are:
(1) continuation of the enquiry proceedings even after the retirement of the employee and prolonged period of suspension.
(2) Prevalence of certain extraordinary circumstances at the material time like commotion due to strike or lock out or other kinds of work stoppage, serious charges of intimidation or assault of other employees like co-worker or officer which require examination of witnesses who are prevented from or threatened to do so by the charge-sheeted employee which result in reasonably impossible situation to hold the enquiry in a peaceful manner
(3) Unequivocal admission of guilt by the delinquent at the pre-enquiry stage itself.
But, you have not adduced such reasons for your inability to conduct the enquiry despite after issuance of notice.
Therefore, under the limited circumstance cryptically mentioned in your post, you cannot straight away terminate the charge-sheeted employee simply skipping the phase of domestic enquiry.
Further more, since he is only an out-sourced employee, being a principal employer, you don't have the locus-standi to initiate any disciplinary action against him. You can ask the contractor to replace him immediately with a substitute.