Dear Manisha,
Please understand that having friendly environment in office is good for cordial relationship but, there would be few employees who are intentionally trying to be overly friendly with other employee to kill their and others time resulting extended breaks and getting disadvantage of the relationship. Pl. identify the such people and counsel them keeping management in the loop.
Issuing memo to all staff would only deter the situation. Instead, you start taking lunch/tea with them, when time is over tell all to get get dispersed in gentle way.I think, as HR Director, you can say that.
Make policy and ensure compliance for arriving office in time, those who break rule, penalize them with deduction in leave/salary or asking them to offer say, chocolate to all their colleagues.
Nitu Parmar