I think the foothold thought was that pasteurization destroys more than just bacteria, and milk might be healthier and/or tastier without having been changed by that process. Of course taste is largely culturally acquired - example A being Germans and UHT milk - but lots of people fancy themselves taste-o-philes.

Then the mistrust of “them” kicked in, and if “they” said the risk of pathogenic bacteria far outweighed any marginal health benefit, the “truth” must be the opposite.