Why do people find weavers suspicious in Silas Marner?

In the book, the people are suspicious of weavers because they hold contrasting worldviews and come from markedly different social backgrounds.
While the villagers subsist within an agrarian culture, the weavers are emigrants from the surrounding towns and cities. To the villagers of Raveloe, Silas Marner is an anomaly. His skills, worldview, and inclinations seem foreign to them.

To the peasants of old times, the world outside their own direct experience was a region of vagueness and mystery: to their untravelled thought a state of wandering was a conception as dim as the winter life of the swallows that came back with the spring; and even a settler, if he came from distant parts, hardly ever ceased to be viewed with a remnant of distrust...especially if he had any reputation for knowledge, or showed any skill in handicraft.

From the quote above, we can see that the villagers have a habit of distrusting anyone who is not part of the agrarian framework in their world.
The text tells us that Silas Marner has lived in Raveloe for fifteen years and in all that time, the villagers have held firm to their suspicions about him. It isn't until Silas adopts Eppie as his own that the people begin to adopt slightly different views about him. Although they retain some of their old suspicions and contempt for Silas, the people (especially the women) begin to hold him in higher regard after Silas takes it upon himself to raise the orphaned two-year-old Eppie.

Silas Marner is the object of intense suspicion in the village of Raveloe. As a weaver, he has special skills, and this makes the villagers resentful of him and all other weavers. Silas is an outsider in the village, having moved there from Lantern Yard. For most people living in the countryside at that time, the village was the center of their universe. They spent their whole lives in the village and rarely encountered any outsiders. When Silas Marner moves to Raveloe, he is instantly aware that he is not accepted by the locals.
Silas's isolation is further compounded by his remarkable skill for making herbal remedies. This arouses the suspicion of the villagers, who believe him to be involved in some kind of witchcraft. Superstition is rife in the locality, and any kind of behavior which stands out from the established norm is going to be regarded with considerable hostility.