The words privilege and privacy have the same root, and they overlap in meaning. The "privileged" class within a town has historically meant those with homes set in garden space, under the shade of trees and behind discreet hedges. In other words, privilege means privacy.
Naturally enough the folks who are crowded into dense housing crammed alongside public thoroughfares, subjected to prying eyes and other people's noise, resent those who have space and peace. Today, we talk about making things equitable. Regarding urban planning this sadly means densifying neighbourhoods and lowering quality of life -- but equally.
Quality of life is lowered when nature-space and green aesthetics are eliminated, gardens paved, the hedge replaced by the surveillance camera, the luxurious lawn criminalized. When this happens people's health suffers, physical and mental. They do less outdoor exercise and experience stress from overcrowding and noise.
Not only people but wildlife suffers: without gardens to forage in, pollinators (bees and other insects plus non-insect pollinators such as hummingbirds) decline -- over 30% of insect and bird species are now going extinct with the loss of plant diversity. And the big trees go; no room for them. Then drought and heat become extreme. The equitably-shared decline in quality of human life and health is also a setback for nature.
The current push for high-density multi-unit housing tightens this downward spiral. That other kind of "setback" -- property setback rules created by urban planners of the past which specified legal distance between houses, roads and other houses -- was meant to protect landscape and ecology. In our haste to undo the privilege of privacy we have taken away ambient ecological and aesthetic assets from everybody.
To meet the need for housing we could either dampen demand (noting the "build it and they'll come" rule), or try to meet that demand (although we never will, given the above-noted rule) while retaining ecological and aesthetic values. In other words: privilege everybody. If we must endlessly build, we could at least maintain real estate setback rules around the multi-unit dwellings being proposed, leaving room for big trees, and not killing the pollinators.