Ownership and Custody of Documents
First of all, your wife is the owner of the documents, and no one else can keep those documents in their possession without her consent. So, the company or anyone else cannot keep her documents. If someone or a company keeps your documents, then it is highly illegal. You can ask the company in writing to return the documents and simultaneously lodge a police complaint.
Legal Implications of Document Retention
Since keeping documents is illegal, there is no legal validity in asking for money for returning her documents. However, before taking any steps, make sure that you have enough evidence to prove that the documents are in their custody. Otherwise, if they sense you are going legal, they might destroy the original documents and completely deny receiving such documents from you. In such a situation, you would have to take all the trouble to apply for duplicate documents from the respective authority, which can be time-consuming.