I’m not sure if there’s any good solution for this out there, but I cooked up my own. Each language translation has its strings in a text file formatted like an ini file with id keys and localized strings separated by an equals sign. You can view the English and Japanese raw text files if you like. These are read into a PHP array (i.e. dictionary), after first looking at what language is specified by the URL (/en for English, /ja for Japanese or any other code), and if that is not specified then looking at what languages the browser is set to prefer via the Accept-Language HTTP header. If the requested language is not available then default to English.

To get the html output localized, the php script that reads through and configures the app (the plain html file itself is set up to run offline for debugging purposes only) looks for string ids enclosed in percent signs, i.e. like %strings id%. These are then replaced with the localized strings from the dictionary. The php-generated content is trivially changed to look up strings from the dictionary. On the JavaScript side, I wanted access to the same string dictionary that I had on the php side, so this is inserted into a <script> block of the generated html output as a JavaScript object (i.e. dictionary). String id lookups can then be done on this object from the JavaScript code just like on the php side. In other words, the php string dictionary is converted into JSON, which is used from the JavaScript side.

あれ何語？ What Language Is This? in 日本語

It all works pretty well and meets my goals. The only downside is that it relies on the server to do some processing, so when I develop on the offline version the strings aren’t available, instead I get to see the raw string ids, which can be useful too, but you have to rely on imagination to envision the end result. Isn’t programming always like that anyway, though?

The first translated version of What Language Is This? is of course Japanese, done by myself and my wife (初めての共同作業? lol), not just because it’s easy for me to do, but also because when looking at the AddThis stats, Japan is the top ranking country, and also as you know the average English skills in Japan are pretty bad, so I suspect there is a demand for a Japanese translation. Looking at the access stats, and discounting those with good English skills (India, Netherlands, Scandinavia, for example), next in line would most likely be Spanish, French, and German, in that order. Anyone feel like helping? Please drop me a comment in that case. I can offer proper credit and a link back from the site in return.

Also this service is very good at identifying indic languages (where as many other services fail to understand).

Well, thanks. And yes, I have been making sure that the languages of the Indian subcontinent and its surrounding areas are thoroughly supported for identification.

But two notable languages have been missing, and I finally got around to adding them. Namely the two Dravidian languages Malayalam (not to be confused with Malay, to which it is unrelated) and Kannada (not to be confused with Canada, to which it is unrelated).

Together with the already supported Tamil and Telugu, this means that all four literary Dravidian languages are supported now! I hope this will be of use to many, and I’d like to thank the Dravidian-speaking bloggers for their support in the form of writing about the site.

The entered text can also be sent with the feedback, allowing me to gather more sample texts to use as material for the statistical analysis used as a basis when identifying the language, and for testing (there’s an automatic test feature built in to What Language Is This?, just run selftest() from a JavaScript console on the page and it’ll test all supported languages to check for regressions – very handy when updating the database, since it’s easy to accidentally break some of the fine tuning).

Anyway, I think it’ll be useful, and I hope everyone will use it a lot since it’ll help me improve the site. I’m already getting a lot of useful and encouraging comments so it’s really fun to keep on developing it. For the next update I’ll probably add more languages.

But I’ve been wondering what languages people use it for, so yesterday evening, while drinking shochu (in spite of which I could only find one bug today! but I did write a processing and database-intensive function, n00b style, which I replaced with a single SQL query today…), I added logging of the results. Only when the language identification certainty is reasonably high is it logged, and only the result; the actual text inputted is not sent. This, of course, happens in the background. A language is only logged once per client, and results from clicking the “example” button (Tower of Babel extracts – I like that story) are not logged.

This morning I added the top ranking to the page. It’s generated on the server side in order for the search engines to see it. The top 5 languages for the past seven days are printed. At this time, i.e. about 15 hours after the result logging started, these are Spanish, Korean, Portuguese, and Thai.

You can see the currently most inputted languages live: http://henrikfalck.com/languageanalyzer/

I thought it would be a really hard problem to solve – writing a piece of software that could figure that out. It turned out not to be so hard though. Just hours of programming, and probably a lot of luck. Because my initial hunches on how to tune the algorithms proved to be pretty right, and I was, and still am, really startled at how good the software became.

I released it as the Wørd – Language Analyzer Opera widget. Unfortunately the target audience for Opera widgets is quite small, so I always thought of making it into a web page. I don’t know why it took so long, but here it is!

The web page version has some new, cool improvements. It will try to detect as you’re typing, for instance. It also has improved support for Swedish, Serbian, and Afrikaans. And the UI is in my opinion better than the widget version.

So please try it yourself and see how it works. It’s pretty fun to just copy-paste any piece of text your can find on the Internet into it, or just type something in a language you know yourself and see when it gets it right. Here’s the address again:

http://henrikfalck.com/languageanalyzer/