The author of a new paper on what a future united Ireland could look like has said lessons should be learned from Brexit when it comes to avoiding a "chaotic" poll.