As small businesses crumble, shelves get emptier and the care-worker shortage intensifies, life outside the EU is having a dire effect on many of us. Why aren’t politicians talking about it?