Yes. Not having a credit card and availing a personal loan in the last 12 to 15 years will certainly affect your credit score.
While overusing a credit card is not advisable, not having a credit card is also equally not advisable; the caution you bring while using credit card & paying the dues on time is what is really required.
A stranger comes and ask you a loan of Rs.1 lakh; you won't give him but your close friend whose credentials are very well known to you, if he comes and ask you Rs.1 lakh which he promises to repay in 2 months time, you'll give in without even demanding interest for the amount.
Now, this is what credit rating agencies too do while ascertaining the credit worthiness of an individual.
The loans an individual has availed and the kind of transactions as well as frequency of defaults / prompt payments are tracked and based on this, the credit score is calculated.