I’m not sure if there’s any good solution for this out there, but I cooked up my own. Each language translation has its strings in a text file formatted like an ini file with id keys and localized strings separated by an equals sign. You can view the English and Japanese raw text files if you like. These are read into a PHP array (i.e. dictionary), after first looking at what language is specified by the URL (/en for English, /ja for Japanese or any other code), and if that is not specified then looking at what languages the browser is set to prefer via the Accept-Language HTTP header. If the requested language is not available then default to English.

To get the html output localized, the php script that reads through and configures the app (the plain html file itself is set up to run offline for debugging purposes only) looks for string ids enclosed in percent signs, i.e. like %strings id%. These are then replaced with the localized strings from the dictionary. The php-generated content is trivially changed to look up strings from the dictionary. On the JavaScript side, I wanted access to the same string dictionary that I had on the php side, so this is inserted into a <script> block of the generated html output as a JavaScript object (i.e. dictionary). String id lookups can then be done on this object from the JavaScript code just like on the php side. In other words, the php string dictionary is converted into JSON, which is used from the JavaScript side.

あれ何語？ What Language Is This? in 日本語

It all works pretty well and meets my goals. The only downside is that it relies on the server to do some processing, so when I develop on the offline version the strings aren’t available, instead I get to see the raw string ids, which can be useful too, but you have to rely on imagination to envision the end result. Isn’t programming always like that anyway, though?

The first translated version of What Language Is This? is of course Japanese, done by myself and my wife (初めての共同作業? lol), not just because it’s easy for me to do, but also because when looking at the AddThis stats, Japan is the top ranking country, and also as you know the average English skills in Japan are pretty bad, so I suspect there is a demand for a Japanese translation. Looking at the access stats, and discounting those with good English skills (India, Netherlands, Scandinavia, for example), next in line would most likely be Spanish, French, and German, in that order. Anyone feel like helping? Please drop me a comment in that case. I can offer proper credit and a link back from the site in return.

The entered text can also be sent with the feedback, allowing me to gather more sample texts to use as material for the statistical analysis used as a basis when identifying the language, and for testing (there’s an automatic test feature built in to What Language Is This?, just run selftest() from a JavaScript console on the page and it’ll test all supported languages to check for regressions – very handy when updating the database, since it’s easy to accidentally break some of the fine tuning).

Anyway, I think it’ll be useful, and I hope everyone will use it a lot since it’ll help me improve the site. I’m already getting a lot of useful and encouraging comments so it’s really fun to keep on developing it. For the next update I’ll probably add more languages.

But I’ve been wondering what languages people use it for, so yesterday evening, while drinking shochu (in spite of which I could only find one bug today! but I did write a processing and database-intensive function, n00b style, which I replaced with a single SQL query today…), I added logging of the results. Only when the language identification certainty is reasonably high is it logged, and only the result; the actual text inputted is not sent. This, of course, happens in the background. A language is only logged once per client, and results from clicking the “example” button (Tower of Babel extracts – I like that story) are not logged.

This morning I added the top ranking to the page. It’s generated on the server side in order for the search engines to see it. The top 5 languages for the past seven days are printed. At this time, i.e. about 15 hours after the result logging started, these are Spanish, Korean, Portuguese, and Thai.

You can see the currently most inputted languages live: http://henrikfalck.com/languageanalyzer/

I thought it would be a really hard problem to solve – writing a piece of software that could figure that out. It turned out not to be so hard though. Just hours of programming, and probably a lot of luck. Because my initial hunches on how to tune the algorithms proved to be pretty right, and I was, and still am, really startled at how good the software became.

I released it as the Wørd – Language Analyzer Opera widget. Unfortunately the target audience for Opera widgets is quite small, so I always thought of making it into a web page. I don’t know why it took so long, but here it is!

The web page version has some new, cool improvements. It will try to detect as you’re typing, for instance. It also has improved support for Swedish, Serbian, and Afrikaans. And the UI is in my opinion better than the widget version.

So please try it yourself and see how it works. It’s pretty fun to just copy-paste any piece of text your can find on the Internet into it, or just type something in a language you know yourself and see when it gets it right. Here’s the address again:

http://henrikfalck.com/languageanalyzer/

But there are big differences as well. Where minesweeper starts with a randomly generated field, sudoku requires a puzzle to be correct, i.e. soluble with only one valid solution. The greatest challenge though, is to rate the difficulty of a given puzzle. Fortunately, I found one good program that can both generate and rate puzzles. My Sudoku game has an online database of thousands of puzzles, separated into five difficulty levels. Of course, it also has online high scores, just like Minesweeper.

I’m calling it an “alpha” though, because I actually suck at sudoku, so I can’t really test it myself yet. I’ve invited people whom I know like solving sudoku puzzles to test it, and you’re invited as well! Please add your comments to this blog post, and I will be extremely thankful. Things I’m looking at specifically are:

• Are the difficulty levels correctly rated?
• Are the puzzles “good”?
• How can the interface be improved? I know sudoku software usually has some methods for making notations as you solve the puzzle. The next step will be to add that, but how should they be designed?
• Any improvements you’d like to see.
• Any bugs you find.

So please, start solving those puzzles!
http://henrikfalck.com/sudoku/

“Web 2.0 is a term that corresponds to Ajax.”
My standard answer to “what is web 2.0?” would be something along the lines of “it’s about control of users and data“, i.e. you build a big web site where users can generate the content, market it, and pray you’ll be among the 1% that are somewhat successful. Then you capitalize on providing pieces of that data. Google if anyone should know that. Ajax (or whatever you choose to call it) is a technology that is certainly a part of the modern web, but really, aren’t pastel colors and rounded corners more important for a web 2.0 site? even if you build it with old-fashioned server-side scripts only. Ajax as a technology is an enabler, not a necessity.

“[Web 3.0] is a different way of building applications.”
Yes, maybe that too. The web by the time 3.0 comes around is bound to have some new technology – or rather some new uses of the technologies we have. And this will allow us to make applications in a different way. So the statement is trivially true, but it does not provide a definition or even a speculation of what web 3.0 will be like. Let’s move on…

“Web 3.0 will be applications that are pieced together.”
This is more interesting. Like mashups? I’m sure all of us computer software users would like to have that. But I’ll believe it when I see it: applications from different vendors cooperating. So far unix system tools are the only ones to come close to this ideal. Unfortunately, I don’t think this is realistic for web apps.

“The applications are relatively small”
Now this is one of my favorite pet peeves. I nagged a bit about this on my CSS Nite presentation as well: Right now, many people, and especially Apple, are thinking of web apps/widgets/gadgets as small, more-or-less useless applications for trivial tasks such as analog clocks or calculators, or something that in the very least is separated from the tasks of “real” applications. I don’t agree with this at all…

Consider Mac OS with its Dashboard (even though it is a great reification of the ideal of web tech-based apps): you have a “normal” mode; your normal apps, running along in their windows as usual, and a “widgets” mode; your normal apps—no wait, these aren’t normal apps – these are widgets, special kind of apps made in a special, even naive, kind of way, to be run in a sandbox isolated from your normal desktop working environment. Even Opera has copied this thinking.

This is definitely how it is right now, so I can understand Apple’s decision. But I don’t think it’s how it will be, in the future, boys and girls. Web technology-based application will be just as common as other applications, I think. And in a few years they will be just as big as well. Many have tried to achieve this before – Java and Dotnet come to mind – but web technology has already won. I thought a lot about this, and I’ll write about it some time.

“The data is in the cloud”
I want some of what you’re smoking too dude, but there is no cloud. Data belongs to somebody, access is restricted, bandwidth is limited. The Internet is not a cloud (it’s a series of tubes).

There are a few more goodies in the video. Anyway, what will the web 3.0 be like? I mean, if you can figure that out now, you’ll be a billionaire. I think I have pretty decent idea, and I’ll continue writing about web 3.0 here in this blog. I think and hope that Google won’t be playing a big part in it. (They might be absorbed into something bigger, though).

This trick is so simple and incredibly easy to pull off that you have no reason not to do it. There are essentially three things involved:

1. Make apache serve .gz.js files with gzip transfer encoding
2. Bundle up all your JavaScript files into one (or a few) big file(s)
3. Gzip your One Big File(s)

Let’s go through these step by step…

1. Make apache serve .gz.js files with gzip transfer encoding
Gzip transfer encoding is great! The only bad thing about it is that on dynamic content, the web server will have to use up some processing power to compress the files on the fly, but in this case, we’re serving static JavaScript source files, so that’s not even an argument! (As for the client, if it’s got enough CPU to run your web app, it’s not gonna have any trouble doing a little unzipping…)

The easiest way to do this is to edit the .htaccess config file in the directory where you put your JavaScript files, or any directory above that one to make it more global. The only thing you have to do is add the following line:
AddEncoding x-gzip gz

This adds the gzip transfer encoding to apache for files with a “gz” file extension.

Notice the title says “.gz.js” though: by adding “.js” at the end, the files will be served correctly with the text/javascript mime type. Otherwise the browser might go berserk. That’s also why I wouldn’t recommend using “.js.gz”; it’ll get served with gzip transfer encoding alright, but not with the JavaScript mimetype.

This solution works even with clients that don’t support the gzip transfer encoding, because web browsers send an “accept-encoding” header that tells the server which encodings the client supports, so apache won’t use gzip transfer encoding if it’s not supported by the client.

2. Bundle up your JavaScript source file into one (or a few) big file(s)
This means simply appending all your source files to one file. Or a few files. I usually bundle up the libraries/utility code etc, i.e. code that you don’t change very often, into one file and the rest of the app into one file that gets updated more often. You can do this by copy-pasting in your text editor if you like, or make a script that does it. I usually have the whole deployment process automated on the server, which I’ll write about some other time perhaps.

Another good way to do it is using Dojo ShrinkSafe. ShrinkSafe will not only append all your source files into one big file, but also reduce the size even further by optimizing the JavaScript code. However, I have had problems with scripts not running correctly (in all browsers) after running them through ShrinkSafe, despite their claims of it being “safe”. I would especially recommend not using the “strip newline chars?” option. Anyway, the file size reduction the results from using ShrinkSafe is small compared to just gzipping the files (I’ll show you some numbers later) so if you’re afraid of regressions, don’t use it.

3. Gzip your One Big File(s)
Now that you’ve got one or two or so JavaScript files, you need to gzip them into a gzip file. You can use the command-line gzip tool on the server to do this, like:
gzip -n9c scriptfile.js > scriptfile.gz.js

The -n option means to not store the the original file name in the gzip file. No big deal if you do, but there’s no reason to do it either. -9 means maximum compression. I like doing things to the extreme! And -c means to write to standard output instead of modifying the source file, so we need to pipe it “>” to a .gz.js file at the end. Or you can use a UI tool such as 7-Zip that supports gzip compression. That’s just damn easy.

Then upload the .gz.js script file to your server, and change all your old <script> tags for every file you had to one pointing to it. I.e. like:
<script type=”text/javascript” src=”scriptfile.gz.js”></script>
That’s all there is to it!

Show Me The Numbers!
Let’s use my Minesweeper game for comparison. The code is served in two files: libs.gz.js containing Prototype and Scriptaculous core and Effects libraries, and minesweeper.gz.js containing the actual game. The breakdown of the files in these libraries before combining them is as follows (at the time of writing):

Original
libs.js: 95 kB + 3 kB + 38 kB = 135 kB
minesweeper.js: 3 + 9 + 9 + 4 + 2 = 25 kB

Shrunk
libs.js: 92 kB (32% smaller)
minesweeper.js: 13 kB (48% smaller)

Gzipped
libs.js: 30 kB (78% smaller)
minesweeper.js: 7 kB (72% smaller)

Shrunk + gzipped
libs.js: 27 kB (80% smaller)
minesweeper.js: 4 kB (84% smaller)

Now, add to that the fact that every HTTP request for a file incurs a 1 kB overhead in HTTP headers, and you get an original total amount transferred of 135 + 25 + 8 = 168 kB compared to 27 + 4 + 2 = 33 kB, i.e. a total saving of 135 kB, i.e. 80% less bandwidth used, not to mention that making two http requests is much faster than making eight http requests!

The next step would be to bundle up html, css, scripts, and everything you possibly can into one request… (In fact, I have scripts doing that automatically at runtime on some sites…) But you have to draw the line somewhere. There’s a definite trade off in maintainability, cachability, and compatibility of the site. So I’ll stop here.

Developing this game has taught me one something about the time it can take from prototype to somewhat finished product: Developing the completely playable “offline” prototype version was quick and easy – after all, I developed minesweeper already in high school on my programmable Casio calculator. So I thought it would take about trice the time to finish it, but in the end it took about ten times that time.

One reason for this is the online highscore functionality. But T-rex, online highscores just means keeping a list of the best times on the server and sending it to the client. Yes, but then it wouldn’t take much to figure that all you need to do is type “register_highscore.php?time=0&name=script%20kiddie” into your browser’s address field to get an instant all-time high. So how did I solve that?

The game is played simultaneously on the client (i.e. in the web browser) and on the server (in php). Sweeps are sent at regular intervals to the server, which then plays the same moves and checks the results. So in the end it’s the server that decides that a game is over, how long it took, and whether that’s a new highscore or not. That, ideally, is not so complicated either, but well there are lots of opportunities to make stupid mistakes. (Especially when you’re having a shochu on the rocks while coding (I did it during my vacation).)

The other reason is that it’s hard to make a web app behave like a nifty game. You need graphics, animations, and that kind of stuff that often causes you to run into problems with layout positioning and, most of all, browser inconsistencies. Web browsers just aren’t ideal for making applications yet. (Well, tell me a platform that is, anyway…) The game runs best, as always, in Firefox. Although I admit I haven’t tried IE 6 yet. I’ll tackle that beast tomorrow. IE 7 works good, though.

I still have a number of features planned that I’ll implement during the coming weeks, but at least now it’s good enough to play. Try and beat my times! Here’s the url again:

http://henrikfalck.com/minesweeper/

How do I know this works? Because it’s based on a much cleaned up version of the code used on paintmyblog.com – which has proven itself already. I look forward to writing some cool apps using this library, and I hope other people will use it as well. Expect the first release next week.