It is common to hear in sermons that tax collectors in Jesus’ time were hated because they worked for the Romans. In Jesus’ time in Galilee, however, we are still in the era of Herodian rule (in contrast with Judea, which was under direct Roman rule by this time). So wouldn’t individuals like Matthew/Levi have collected Herod’s taxes rather than Rome’s?
You don’t have to be collecting taxes for foreign overlords in order to be unpopular, of course…