Hi Yogita,
Its good to see and very appreciable discipline in your office log in time. No need to offense, Since this is the scenario with many of small and young organizations. You can actually stop them blaming you with the below ways.
As you are reaching early on every day, You can ask or put a request to the concerned person to handle over the key to you, Which makes more sense in a way of opening the office at-least a 30 mins before for all other employee comfort ability who logs in a little before to the office. On the other hand, You can start coming late, to convey your importance of discipline towards lo gin time, which makes them to review their blaming s on their own.
Either case is fine, But you should ensure that you will not be falling into their criteria of blaming mode.