Same issue as per our office; ours is also a start-up organization that started 2 years ago. Right from the beginning, I was the only one with the key. However, as staff joined, and on some days when I came late, they had to wait outside the office. Our office is in a complex with many other offices, so the facility of waiting space and restrooms is always available to staff whether the office is open or not.
As more people joined and their duties became more complex, we started issuing keys to those who needed them. Now, almost every staff member, a total of 8, has the office keys. Once they open the office, they can use the electronic access lock on the main door, allowing the next person to open the door using a fingerprint or card.
Another thing is that in any office, the housekeeping or cleaning staff comes half an hour before and cleans the office premises. This is a normal practice, so all offices open half an hour to 1 hour before the working times.
I can understand your problem of waiting half an hour until someone comes and opens the office. Just check; some solution will be there. We all know that when we come to the office after traveling a long distance, we need to refresh ourselves and relax for 5-10 minutes, which is not possible if the office is closed and no proper facilities are available.