Move means PM must be in Westminster for prime minister's questions on the scheduled day of his leader's speech to activists.