The city has lost tens of thousands of residents in the past several years. And much of what’s driving them away is well beyond the control of the city’s next mayor.