Since, the first part, i.e. keeping documents is illegal, there is no legal validity of asking money for returning her own documents.
But before taking any steps, make sure that you have enough evidence to prove that the documents is in their custody. Otherwise, if they sense you are going legal, they might destroy the original documents and completely deny receiving such documents from you. In such a situation, you have to take all the trouble to apply for duplicate documents from the respective authority which can be time consuming.a
12th September 2016 From India, Kolkata
When you submit the original documents take receipt from the HR department on the photocopy of your certificate. Please try to avoid giving original documents which is not industry practice.
Further You don't have to give money for returning the documents. This will put you in great difficulty when you resign from the company
16th September 2016 From India, Chennai