The White Cliffs Hotel is 140 years old but the owner said it suffered a steep downturn after Brexit and the pandemic.