Comparison of letter positions in eight languages
Click to enlarge:
EDIT 2014-07-08: Sometimes the readers are just smarter than me! The original graphic (which you can see at the end of the post) had the letters in rows; I got six e-mails in the first few hours suggesting it would be a lot easier to compare across languages if they were in columns. When you're right, you're right.
My May 27 blog post of the distribution of letters in English toward the beginning, middle and end of words seemed well-received, and generated quite a few compliments, and not a few requests to do the same for other languages. One reader was even inspired to do a similar project in French.
Since I already had the code, I thought, why not? Now the only problem was getting my hands on a corpus; you can read about my adventures in this regard, as well as some more esoteric analysis of this data set, on my other, geekier blog; suffice it to say I was quite fortunate to find the Europarl Parallel Corpus, a collection of proceedings of the European Parliament with simultaneous translations in twenty languages. Since every language has the same subject matter, we're maximizing the chances that any differences we see are actually due to the language, not because of differences in the corpus.
I chose the seven languages with the most speakers in the European Parliament, plus Finnish because I thought it would be interesting to have a non-Indo-European language to compare as well.
Note that characters outside of the Basic Latin Unicode block* (accents, digraphs, etc.) are aggregated with their non-ornamented versions; this is not ideal, since they're not at all interchangeable (if you ask someone where a congrès is in Paris without the accent, they'll point you towards a seafood shop), but it's really the only way we can make the datasets comparable in this limited and hardly scholarly context.
Note that to determine the shape of the individual graphs I followed the same methodology as last time, outlined here.
There are a lot of interesting features in the chart, I could stare at the thing for hours (and I have.. well, 45 minutes, anyway). I'll just name a few here:
Here's the original graphic:
* Thanks to readers for pointing out that my quickly googled source on German orthography was quite wrong, and that my phrasing seemed to be saying that only English ever had a d as the last letter of a word (which would, of course, not be la verdad). Also, since I deal with Unicode and UTF-8 difficulties every day, I forgot that some might be confused that I seemed to be claiming the Latin language had 26 letters.
EDIT 2014-07-08: Sometimes the readers are just smarter than me! The original graphic (which you can see at the end of the post) had the letters in rows; I got six e-mails in the first few hours suggesting it would be a lot easier to compare across languages if they were in columns. When you're right, you're right.
My May 27 blog post of the distribution of letters in English toward the beginning, middle and end of words seemed well-received, and generated quite a few compliments, and not a few requests to do the same for other languages. One reader was even inspired to do a similar project in French.
Since I already had the code, I thought, why not? Now the only problem was getting my hands on a corpus; you can read about my adventures in this regard, as well as some more esoteric analysis of this data set, on my other, geekier blog; suffice it to say I was quite fortunate to find the Europarl Parallel Corpus, a collection of proceedings of the European Parliament with simultaneous translations in twenty languages. Since every language has the same subject matter, we're maximizing the chances that any differences we see are actually due to the language, not because of differences in the corpus.
I chose the seven languages with the most speakers in the European Parliament, plus Finnish because I thought it would be interesting to have a non-Indo-European language to compare as well.
Note that characters outside of the Basic Latin Unicode block* (accents, digraphs, etc.) are aggregated with their non-ornamented versions; this is not ideal, since they're not at all interchangeable (if you ask someone where a congrès is in Paris without the accent, they'll point you towards a seafood shop), but it's really the only way we can make the datasets comparable in this limited and hardly scholarly context.
Note that to determine the shape of the individual graphs I followed the same methodology as last time, outlined here.
There are a lot of interesting features in the chart, I could stare at the thing for hours (and I have.. well, 45 minutes, anyway). I'll just name a few here:
| Vowels | Unsurprisingly, Spanish, Italian and Portuguese have the vowels "a" and "o" shifted towards the end of words, plus "e" and "i" in Italian. Non mi credi? È vero! |
| Foreign letters | Some languages use certain letters only in foreign or borrowed words (French, Spanish and Portuguese, k, w; |
| D | English is alone having the "d" most commonly* at the end of words, thanks to the past tense. Who'd have guessed? Well, anyone who thought about it, I suppose. |
| H | To my eye, "h" shows the most difference in distributions patterns: the most representative words are the, chaque, nicht, ha, che, senhor, tych and puhemies. |
| L | Because of their articles, French and Spanish's "l"s are much more front-heavy. Le phénomène, es la verdad. Similarly, due to their grammar, German and Finnish's "n"s are towards the end. |
| P | "P" is mostly* at the beginning of words in all eight languages; due to its source, this particular corpus has a preponderance of cognates of president, parliament and political, but even removing these words leaves the phenomenon intact. This makes intuitive sense to me (n.b. IANAL -- I am not a linguist); as a bilabial plosive, it's awkward at the end of words; consider the word "pep". The same phenomenon occurs to a lesser extent for p's voiced analogue, the letter "b". [relevant Sesame Street] |
| W, Y, Z | Letter frequencies aren't the point of this chart, but they're interesting in their own right. Look how much more often "w", "y" and "z" are used in Polish than in any other language (the word for "all" is "wszystkich"), and I never would have guessed "f" to be such a particularly English letter (though it's obvious in retrospect, given that among the most common words are "of" and "from"). |
Here's the original graphic:
Graphing the distribution of English letters towards the beginning, middle or end of words
Some data visualizations tell you something you never knew. Others tell you things you knew, but didn't know you knew. This was the case for this visualization.
Many choices had to be made to visually present this essentially semi-quantitative data (how do you compare a 3- and a 13-letter word?). I semi-exhaustively explain everything at on my other, geekier blog, prooffreaderplus, and provide the code I used; I'll just repeat the most crucial here:
| • | The data is from the entire Brown corpus in the Natural Language Toolkit. It's a smaller and out-of-date corpus, but it's open source and easy to obtain. I repeated the analysis with COHA, the Corpus of Historical American English, a well-curated, proprietary data set from Brigham Young University for which I have a license, and the only differences were in rare letters like "z" or "x". | |
| • | I used a corpus rather than a dictionary so that the visualization would be weighted towards true usage. In other words, the most common word in English, "the" influences the graphs far more than, for example, "theocratic". | |
| • | The ordinal (y) scales are obviously not equal: "e" is used 100-200 times more often than "z", and while I could have fudged everything with log scales, letter frequency is not the point of the graphs. As long as I had to fudge anyway, I did so in a way that, I believe, makes it easiest to understand what the graph shows. Your mileage may, of course, vary. The color coding is a quick guide to help understanding, since letter frequency is of course relevant to the shapes you see. | |
| • | There are 15 "bins" of letter positions, as a purely qualitative comparison suggested to me this was about the ideal number to show the underlying trends without under- or overfitting. Therefore the "t" in "the" takes up positions 1 through 5, the "h" 6 through 10, etc. When letters straddle a boundary they are apportioned proportionately. |
Now then: I became curious about how letters are placed in English while doing many different, often quick, sometimes pointless, pattern analyses of letters for a wide variety of reasons. (One example: for one art project that will hopefully be posted on this blog one day, I found all the anagrams of "Hollywood", and noticed that words beginning with "w" were overrepresented.)
I've had many "oh, yeah" moments looking over the graphs. For example, words almost never begin with "x", but it's quite common as the second letter. There's a little hump near the beginning of "u" that's caused by its proximity to "q", which is most common at the beginning of a word. When you remove "q" from the dataset, the hump disappears. "F" occurs toward the extremes, especially in prepositions ("for", "from", "of", "off") but rarely just before the middle.
A final thought: the most common word in the English language is "the", which makes up about 6% of most corpuses (sorry, corpora). But according to these graphs, the most representative word is "toe".
Google ngrams of 'google' and 'ngram'
I like to test data tools and data sets with "edge cases", a fancy word for using them in ways they were not designed to be used (which is, by the way, the definition of hacking). It's informative to see how far things will bend before they break -- and the good thing with data is it's easy to un-break.
Rare occurrences make good edge cases; so do recursive cases, i.e. run a data tool on itself. We looked briefly at the Google Ngram Viewer a couple of weeks ago; what happens if we determine Google Ngrams of the words "Google" and "Ngram"? (By the way, I like to call this kind of approach 'selfremetacursironiferentiality'. I'm sure it will catch on one day so I look like less of a dork when I say it.)
Of course, the frequency of the word "google" after the company was incorporated in September 1998 is predictable: it becomes a very common word (and is even adopted into that hallowed club, The Verb, where Xerox briefly rested and from which Kleenex was inexplicably barred). The only interesting thing about its 2001-2008 (where the data set ends) rise is that it's pretty linear; I would have intuited either positive or negative curvature, but don't forget this is the word's appearance in published, printed matter, not in conversation.
That's a lot of use of the word "google" before the company we all know and... well, know... existed. Using Google Books, the mystery is easy to solve: there was a newspaper comic strip character named Barney Google, and a lot of anthologies were published over the years. Not unusually, the technical term "ngram" lags far behind a term used in pop culture; however, it is surprising that around the dawn of the 20th century a term used in computational linguistics would turn up. Again, Google Books solves the mystery: this is an artifact of a lot of directories of names from around this time being poorly scanned; the name "Ingram" is being recorded as "I, ngram" (which sounds like a terrible book title).
The moral of this story, as with all data sets too huge to be curated by humans (and, coincidentally, every other Aesop's fable): things are not always what they seem, so we'll be sure to dig a little before drawing conclusions, especially in edge cases. The next time someone brings up over the water cooler how ngrams were being studied in 1902, you can nod to yourself knowingly.
* of course, sometimes that means it's just noise, but I find noise interesting too
*** that's what she said.
Rare occurrences make good edge cases; so do recursive cases, i.e. run a data tool on itself. We looked briefly at the Google Ngram Viewer a couple of weeks ago; what happens if we determine Google Ngrams of the words "Google" and "Ngram"? (By the way, I like to call this kind of approach 'selfremetacursironiferentiality'. I'm sure it will catch on one day so I look like less of a dork when I say it.)
Of course, the frequency of the word "google" after the company was incorporated in September 1998 is predictable: it becomes a very common word (and is even adopted into that hallowed club, The Verb, where Xerox briefly rested and from which Kleenex was inexplicably barred). The only interesting thing about its 2001-2008 (where the data set ends) rise is that it's pretty linear; I would have intuited either positive or negative curvature, but don't forget this is the word's appearance in published, printed matter, not in conversation.
Let's have a look at "google" and "ngram" (both case-insensitive) from 1880 to 2000, before the rise of Google and with a vertical axis about fifty-fold lower so we can see the edge cases (in my experience, the more jagged a line is*, the more interesting it is.**)
The moral of this story, as with all data sets too huge to be curated by humans (and, coincidentally, every other Aesop's fable): things are not always what they seem, so we'll be sure to dig a little before drawing conclusions, especially in edge cases. The next time someone brings up over the water cooler how ngrams were being studied in 1902, you can nod to yourself knowingly.
* of course, sometimes that means it's just noise, but I find noise interesting too
*** that's what she said.
The Nteresting Nnovation of Google Ngrams
If you're unfamiliar with the term or concept of ngrams in general or Google Ngram Viewer in particular, a look at it in action is the best explanation:
This shows how often the words "overrated" and "underrated" appear in Google Books from 1800 to 2008 -- sort of. There are a few caveats, which Google is upfront about (although I wish they'd post a précis of the shortcomings of the database and the main erroneous conclusions that can be drawn from them on the main page of the Ngram Viewer). I'll get into the unique problems of computerized curation of a dataset so huge it comprises 6% of all the books in existence (so they claim, it depends how you count them, but it's a defensible number).
So as the title says, what they heck is an ngram? Well, what you see above are 1grams. If I look up phrases, they become 2grams (or bigrams), 3grams (trigrams), 4grams, 5 grams (not to be confused with pentagrams). Some fascinating things can be revealed by searching for multiword units; we'll look at them in later blog posts.
You have to be careful what conclusions you draw: from the above graph, could you say people were more pessimistic in 1850? No, we haven't run the proper controls: for instance, are there synonyms for "overrated" that took over in 1900? Are there certain kinds of books overrepresented in the database that are more likely to use these terms? Google published a paper with some interesting results (such as the effects of Nazi censorship), but they had the resources to have verifiable control experiments.
Still, it's an interesting database, and one I find myself turning to a lot. Just as there are those who pore through Google Street View to find oddities like people wearing horse head costumes; I do the same with Google Ngram Viewer. I don't like Google's presentation, though, so I wrote a script to automatically import results into python and create prettier graphs (that use per million instead of per cent so you don't have all those leading zeroes, for one):
That's a dramatic rise for "onto the". What could it possily mean? Well, I'll telll you... later.
This shows how often the words "overrated" and "underrated" appear in Google Books from 1800 to 2008 -- sort of. There are a few caveats, which Google is upfront about (although I wish they'd post a précis of the shortcomings of the database and the main erroneous conclusions that can be drawn from them on the main page of the Ngram Viewer). I'll get into the unique problems of computerized curation of a dataset so huge it comprises 6% of all the books in existence (so they claim, it depends how you count them, but it's a defensible number).
So as the title says, what they heck is an ngram? Well, what you see above are 1grams. If I look up phrases, they become 2grams (or bigrams), 3grams (trigrams), 4grams, 5 grams (not to be confused with pentagrams). Some fascinating things can be revealed by searching for multiword units; we'll look at them in later blog posts.
You have to be careful what conclusions you draw: from the above graph, could you say people were more pessimistic in 1850? No, we haven't run the proper controls: for instance, are there synonyms for "overrated" that took over in 1900? Are there certain kinds of books overrepresented in the database that are more likely to use these terms? Google published a paper with some interesting results (such as the effects of Nazi censorship), but they had the resources to have verifiable control experiments.
Still, it's an interesting database, and one I find myself turning to a lot. Just as there are those who pore through Google Street View to find oddities like people wearing horse head costumes; I do the same with Google Ngram Viewer. I don't like Google's presentation, though, so I wrote a script to automatically import results into python and create prettier graphs (that use per million instead of per cent so you don't have all those leading zeroes, for one):
That's a dramatic rise for "onto the". What could it possily mean? Well, I'll telll you... later.
Use of the f-word in Eddie Murphy: Delirious
I had planned to take a break from blogging during the holidays, but today I saw this post on reddit about the use of the f-word in movies in the dataisbeautiful subreddit, and I was inspired. The top movie on the list I had seen was Eddie Murphy: Delirious; I was 13 when it came out, but nobody I knew had HBO, so my best friend and I had to wait till it showed up in the Betamax tape rental place. We made a lo-fi audio recording (a microphone held up to the TV speaker), and soon had it memorized and spent several years quoting it in all sorts of inappropriate situations.
So, let's break down the use of the f-word (I admit, I'm being a total wuss, Google hosts this blog and I'd rather not deal with any automated fallout from using profanity, so I'm going to asterisk out all the naughty words) during the movie. Some simple poor man's calculus (for each use of the word at time x, y equals the inverse of the average of the times of the previous and next use) shows the clustering of swearing during different parts of the film:
It would be great to know what parts of the movie those clusters correspond to: if you go to the bottom of the post, there's a reversed version of the graph that allows you to see the dialogue (lightly Bowdlerized, again, I'm sorry) line by line.
I've been learning how to do Natural Language Programming in Python, and while I didn't bring out the big guns, I thought it would be interesting to look at some of the simple patterns in word use in the movie:
Normally I would use a stop list to remove common words like "the" and "and", and a corpus to compare word frequencies, but I think the raw data is the most informative perspective, showing how the profanity rivals the most common syntactic words in Delirious. Here are the top N-grams (words that appear side-by-side):
I'm a contributor to the FullMovieGifs subreddit, so I couldn't resist the temptation to make one of Delirious. Hopefully Google doesn't OCR these things; if you want to see it larger, click on it.
Finally, here's a big, vertical version of the first graph in the blog, which you can mouseover to read the lines of dialogue (is it still called dialogue when only one person's talking?) to your heart's content. If you can't see a really huge graph right underneath this sentence, click here to see it.
I think I'll be hearing from my mom about this post.
Update Jan. 1, 2014: Whaddaya know, my mom was fine with it.
So, let's break down the use of the f-word (I admit, I'm being a total wuss, Google hosts this blog and I'd rather not deal with any automated fallout from using profanity, so I'm going to asterisk out all the naughty words) during the movie. Some simple poor man's calculus (for each use of the word at time x, y equals the inverse of the average of the times of the previous and next use) shows the clustering of swearing during different parts of the film:
I've been learning how to do Natural Language Programming in Python, and while I didn't bring out the big guns, I thought it would be interesting to look at some of the simple patterns in word use in the movie:
Normally I would use a stop list to remove common words like "the" and "and", and a corpus to compare word frequencies, but I think the raw data is the most informative perspective, showing how the profanity rivals the most common syntactic words in Delirious. Here are the top N-grams (words that appear side-by-side):
I'm a contributor to the FullMovieGifs subreddit, so I couldn't resist the temptation to make one of Delirious. Hopefully Google doesn't OCR these things; if you want to see it larger, click on it.
Finally, here's a big, vertical version of the first graph in the blog, which you can mouseover to read the lines of dialogue (is it still called dialogue when only one person's talking?) to your heart's content. If you can't see a really huge graph right underneath this sentence, click here to see it.
I think I'll be hearing from my mom about this post.
Update Jan. 1, 2014: Whaddaya know, my mom was fine with it.
Frequency of Scripps National Spelling Bee winning words in Google Books, 1925-2008
Please note that this is a logarithmic scale; every vertical division to the right makes the word is ten times more common, two divisions make it 100 times more common, etc.
If you want to see a standalone image at full resolution, it's hosted here.
The diagonal black line shows the overall trend: the Scripps National Spelling Bee winning words have become more and more unusual as the years pass.
As noted above, this is a logarithmic scale, and thus there cannot be a zero value. (It's a math thing.) Words without bars at all did not appear at all in the Google Ngram Viewer for that year. According to Google, this means the word appeared fewer than 40 times in their corpus for that year.
The only word in this list that does not appear in the Google Ngram Viewer for any year is esquamulose, the winning word of 1962. It is an adjective meaning "not covered in scales".
The list of winning words was from Wikipedia; for more information about the Scripps National Spelling Bee, see the Wikipedia page or their home page.
EDIT: There was a discussion on Reddit's Data Is Beautiful forum about this graph.
No smoothing was used in the Google Ngram Viewer, in order to get the most accurate results. The Google Ngram Viewer only goes to 2008 at present, which is why the graph stops there; the winning words since then were Laodicean, stromuhr, cymotrichous, guetapens and knaidel.
Gender in Language
Sources:
German/Spanish bridge perception: Guy Deutscher, Through the Language Glass: Why the World Looks Different in Other Languages (Picador USA, 2010), also mentioned in a New York Times article by the author.
Further reading:
Grammatical gender on Wikipedia
Luganda on Wikipedia
