Mapping the Muddle: "It's __________ to me."

Strange Maps has a fascinating post about how different cultures fill in the blank of the expression we English speakers formulate as "It's Greek to me." As this cartogram shows we're not alone: Persians, Swedes and Spanish speakers are among groups that refer to Greek when considering something incomprehensible. But it's not the most popular expression of the idea:

But it is Chinese that, according to this cartogram, is the incomprehensible lingo of (p)reference for almost a dozen other languages, from Greek and Polish to Dutch and Lithuanian. Spanish, Hebrew and Greek are also quite popular, understandably so in the case of the latter two languages (isolate, relatively small languages) but more inexplicably so in the case of Spanish - a world language in its own right.

Which begs the fundamental question: why is language X considered the summit of incomprehension by language Y? Doesn’t that at least require some passing knowledge (or to be more precise, an awareness of the existence) by Y of X?

A fun tidbit at the end of the post:

Even Esperanto-speakers have been endowed with their own expression, pointing the finger at another constructed language: “Estas Volapuk al mi!” (”It’s Volapük to me!”)