Everyone should worry about no deal, the civil servant who was, until March, head of the Brexit department has said.