Selten and Warglien in P.N.A.S. try to experimentally evolve a new language:
We investigate in a series of laboratory experiments how costs and benefits of linguistic communication affect the emergence of simple languages in a coordination task when no common language is available in the beginning. The experiment involved pairwise computerized communication between 152 subjects involved in at least 60 rounds. The subjects had to develop a common code referring to items in varying lists of geometrical figures distinguished by up to three features. A code had to be made of a limited repertoire of letters. Using letters had a cost (…) Our experiments show that a too small repertoire of letters causes coordination failures. Cost efficiency and role asymmetry are important factors enhancing communicative success. In stable environments, grammars do not seem to matter much, and instead efficient arbitrary codes often do better. However, in an environment with novelty, compositional grammars offer considerable coordination advantages and therefore are more likely to arise.
Update: John Hawks goes into more detail