That is not an issue for the new company where you joins but the HR or the Personnel Manager can raise a question why didn't you contribute for four years? The answer will, obviously, be that the company where I have been working since 2018 is a small company with less than 20 employees and hence not covered by EPF. It is not an issue but the experience of a person from a small organisation is always counted to be very negligible. Organisations would always prefer to have persons from reputed establishments. Whether in Sales, Operations, HR or Accounts, the experience gained from an organisation having large number of employees weigh more than from that from a small organisation. This is very significant when you join an HR team where the number of employees handled count much.
Now if your establishment was having more than 20 employees but that company did not 'believe' in statutory coverages and contributions, that also will be a negative remark on the company. Moreover, it is not just one or two years but four years and that should not be ignored.
Now coming to another scenario, the new organisation, per se, is not concerned about the PF applicability of your previous company. But they are concerned about whether you had a Universal Account Number (UAN) of PF. Since you have a UAN of your first company, ie, the company you worked prior to joining the immediate previous company in 2018, that can be given in form 11 declaration. Since that UAN is active you can continue that account with new member id created from the new company. Here also the only risk is that you have to give the name of the first company and not the immediate previous company in the form 11.