A minister has been criticised for suggesting that "many people" believe judges are biased about Brexit.