There is a shitload of legal precedent limiting just how much you can soak a non-resident taxpayer. The State of Minnesota has a relative well written 2 pager about it:
https://www.house.mn.gov/hrd/pubs/ss/clssnonr.pdf
This is why you often see homestead exemptions (often with qualifiers based on age) or homestead vs. non-homestead tax rates for education taxes but not a straight up soaking of a non-resident property owner. There needs to be relative justification for the different rate.
Like Maz said, rarely does it make a huge difference. Non-resident tax payers are already a pretty sweet deal for most towns, where they pay in significantly more than they take.