David Walker, who voted for Brexit "on impulse", says he did not think about how the vote could affect his lifestyle.