I worked at a site with a karma system years before reddit and the like ever came into being. There will always be people who just downvote anything they don’t like. Unless you start finding and removing those users, nothing is going to change with them. And if you start removing chunks of your community, you have fewer posters, less interaction, etc.