The ex-£140,000-a-year aide also admitted the government had been “swamped by the Brexit problem”, which he said the PM branded a “huge nightmare”.