I do not know much about Karnataka Factory Rules, but there are model rules framed by DGFASLI in Mumbai, which are then adopted with some changes here and there by State Legislative Assemblies to form State Rules. According to Section 87 of the Act, State Governments are empowered to declare any operation being carried out in a factory as 'Dangerous,' which may make rules applicable. Under Section 85, State Governments are empowered to make the Act applicable to certain premises.
So, normally, as a general principle, State Governments lower the number of workers for the applicability of the Act in the case of Hazardous Operations. One may find the list of Hazardous Operations declared by State Governments in the State rules under the chapter "Special Provisions." In Gujarat Chemical works, areas in the factory with high noise (higher than 90 dB), Welding/Cutting, Foundry, Glass industry, Grinding or glazing of metals, places where Mercury, lead, Asbestos, Silica, Chromium, etc., are handled, and workers are exposed to them.
In Gujarat, the Act is applicable to the units employing 5 or more in Hazardous Operations, but in some Hazardous Operations, even if one worker is employed, the Factory Act/Rules are applicable. The Government of Gujarat published a notification in November 2008, and according to this notification, units where Asbestos is handled, the Act will be applicable even if one worker is employed. The website of the Karnataka Government should have this information.