Travelers, like other itinerant groups (such as the more famous Roma), are frequently greeted with negative reactions from the communities they are near.  As is frequently the case with negative stereotypes, there is often a grain of truth to the claims which are then broadened out and applied to the group as a whole.

Read this story from the beginning here.