You are right! In fact most of the people in the southern coastal provinces were once considered as nan man or southern barbarians by the mandarins in the central plains. Hokkien, Canton, Hainan etc slowly got occupied by the northerners because of invasion by the northern barbaric hordes...