The UK Brexit minister has been urged to take responsibility for making the deal he negotiated work, after he suggested the UK could take drastic action.