Self-fertilization, or: web 3.0, or: Mixi, or: One of those engrish.com moments

Today I visited the brand new, hip and fancy offices of Mixi (in Harajuku, overlooking Yoyogi Park with a spectacular view of Shinjuku and Shibuya...). Now, my work, both as under-stimulated code monkey (by day) and as a web 3.0 consultant (by night), is of course highly classified shit. But I'd like to write a bit about Mixi, because I find the phenomenon interesting, and I really like Mixi (the site) and visit it daily.

If you haven't heard of Mixi that means you aren't Japanese or Japanophile. To put it generalized and bluntly: Mixi is the only social networking site in Japan. Japan is the second largest economy in the world (★pause for reflection★). The reason it's so popular is basically the same as why Microsoft products are: they were there first, and everyone else uses them, and the basic functionality is actually good.

Mixi, technically, is stone age. Although recently they've introduced video upload etc that we have become accustomed with on the modern web, the basic technology is just server-side perl scripts outputting broken html with a table-based design. In other words: it's web 1.0, although they have a pastel color, but it's the wrong hue, and pastel color alone doesn't make web 2.0 - you need rounded corners and rss too.

But as a consumer-oriented product, Mixi is really state of the art. It's actually statier than the statiest art. I started using the predecessors to nowadays' social networking sites in junior high school, back in Sweden. That was like 10 years ago now I guess. (Heh, when I think back, that was about the time I got my first mobile phone. Was that only ten years ago?!) . Even though they used about the same technology then as Mixi does now, the culture and usage patterns are completely different. They were about kids doing their best to make their pages look as hideous as possible (like today's Myspace) and presenting themselves as generally emo and cool. And guys (both young and very old) trying to pick up young girls, of course. But Mixi is not like that.

Oh well, there's that too. But Mixi is much more woven into the fabric of Japanese society. It's like an ad-sponsored public service page (fortunately, and strangely, the mobile version doesn't have ads). And fortunately, you can't design your own page, and there are no widgets etc, so it's actually possible to browse around people's profiles and community pages. Really nice, although I bet it's more because the Mixi people haven't figured out how to implement it technically than a conscious decision.

I joined Mixi when I realized my Japanese language skillz had gotten good enough for me to actually understand pretty much all of the communication taking place there. And the reason I keep using it is still mostly to practice reading Japanese; every day on the train I read some new, interesting tidbits from the parts of Japanese society that concern me. Like what's happening in my town, what's happening along the train lines I use, what events are going on at my favorite bars and clubs, or if there's a Swedish-speaking off-kai soon (off-kai: オフ会, people who talk online meet up in real life), etc. I give it three thumbs up!

Anyway, now for the real anecdote here, and the reason I figured I'd write this blog post at all: In their reception they had this wall with all kinds of catchy words and phrases written on it in the style of a tag cloud. Very, very web 2.0 hip I must say... If anything proves that you're falling behind current developments in the world of the web, it's that you're trying to mimic a Google office, I'd say. (I'd like my office to look classical and sophisticated, and there's always music in the air.)


Now, you can notice that, just beside "web 3.0", they've included the word "self-fertilization". I don't suppose I'm the only one who kinda gets a bit suspicious because of that. And I find the graphical proximity to "web 3.0" especially intriguing. I don't suppose it's a statement of theirs? Nah, it's probably one of those engrish.com kinda moments, you know, when Japanese people confuse R and L, or use Google Translate to translate business emails. Anyways, it's funny.

What is Web 3.0?

I found this video today where Eric Schmidt (of Google) answers the question "what is web 3.0?". The guy (with The Christian Look) who asks the question boldly asserts that we know what web 2.0 is... do we? Anyway,



"Web 2.0 is a term that corresponds to Ajax."
My standard answer to "what is web 2.0?" would be something along the lines of "it's about control of users and data", i.e. you build a big web site where users can generate the content, market it, and pray you'll be among the 1% that are somewhat successful. Then you capitalize on providing pieces of that data. Google if anyone should know that. Ajax (or whatever you choose to call it) is a technology that is certainly a part of the modern web, but really, aren't pastel colors and rounded corners more important for a web 2.0 site? even if you build it with old-fashioned server-side scripts only. Ajax as a technology is an enabler, not a necessity.

"[Web 3.0] is a different way of building applications."
Yes, maybe that too. The web by the time 3.0 comes around is bound to have some new technology - or rather some new uses of the technologies we have. And this will allow us to make applications in a different way. So the statement is trivially true, but it does not provide a definition or even a speculation of what web 3.0 will be like. Let's move on...

"Web 3.0 will be applications that are pieced together."
This is more interesting. Like mashups? I'm sure all of us computer software users would like to have that. But I'll believe it when I see it: applications from different vendors cooperating. So far unix system tools are the only ones to come close to this ideal. Unfortunately, I don't think this is realistic for web apps.

"The applications are relatively small"
Now this is one of my favorite pet peeves. I nagged a bit about this on my CSS Nite presentation as well: Right now, many people, and especially Apple, are thinking of web apps/widgets/gadgets as small, more-or-less useless applications for trivial tasks such as analog clocks or calculators, or something that in the very least is separated from the tasks of "real" applications. I don't agree with this at all...

Consider Mac OS with its Dashboard (even though it is a great reification of the ideal of web tech-based apps): you have a "normal" mode; your normal apps, running along in their windows as usual, and a "widgets" mode; your normal apps---no wait, these aren't normal apps - these are widgets, special kind of apps made in a special, even naive, kind of way, to be run in a sandbox isolated from your normal desktop working environment. Even Opera has copied this thinking.

This is definitely how it is right now, so I can understand Apple's decision. But I don't think it's how it will be, in the future, boys and girls. Web technology-based application will be just as common as other applications, I think. And in a few years they will be just as big as well. Many have tried to achieve this before - Java and Dotnet come to mind - but web technology has already won. I thought a lot about this, and I'll write about it some time.

"The data is in the cloud"
I want some of what you're smoking too dude, but there is no cloud. Data belongs to somebody, access is restricted, bandwidth is limited. The Internet is not a cloud (it's a series of tubes).

There are a few more goodies in the video. Anyway, what will the web 3.0 be like? I mean, if you can figure that out now, you'll be a billionaire. I think I have pretty decent idea, and I'll continue writing about web 3.0 here in this blog. I think and hope that Google won't be playing a big part in it. (They might be absorbed into something bigger, though).

Reducing Bandwidth Usage when Deploying Web Applications

... by serving your JavaScript source code files as one big gzipped file. Nowadays when we're getting more and more nice applications (and widgets, gadgets, mashups, and whatnots) running on the web, there's a lot of talk about user experience and stuff, which involves the whole experience of using a web site. Good content, design, and fancy programming, of course make up the core of a good user experience. But we all wants sites to be quick and responsive to load too, right?

This trick is so simple and incredibly easy to pull off that you have no reason not to do it. There are essentially three things involved:

1. Make apache serve .gz.js files with gzip transfer encoding
2. Bundle up all your JavaScript files into one (or a few) big file(s)
3. Gzip your One Big File(s)

Let's go through these step by step...

1. Make apache serve .gz.js files with gzip transfer encoding
Gzip transfer encoding is great! The only bad thing about it is that on dynamic content, the web server will have to use up some processing power to compress the files on the fly, but in this case, we're serving static JavaScript source files, so that's not even an argument! (As for the client, if it's got enough CPU to run your web app, it's not gonna have any trouble doing a little unzipping...)

The easiest way to do this is to edit the .htaccess config file in the directory where you put your JavaScript files, or any directory above that one to make it more global. The only thing you have to do is add the following line:
AddEncoding x-gzip gz

This adds the gzip transfer encoding to apache for files with a "gz" file extension.

Notice the title says ".gz.js" though: by adding ".js" at the end, the files will be served correctly with the text/javascript mime type. Otherwise the browser might go berserk. That's also why I wouldn't recommend using ".js.gz"; it'll get served with gzip transfer encoding alright, but not with the JavaScript mimetype.

This solution works even with clients that don't support the gzip transfer encoding, because web browsers send an "accept-encoding" header that tells the server which encodings the client supports, so apache won't use gzip transfer encoding if it's not supported by the client.

2. Bundle up your JavaScript source file into one (or a few) big file(s)
This means simply appending all your source files to one file. Or a few files. I usually bundle up the libraries/utility code etc, i.e. code that you don't change very often, into one file and the rest of the app into one file that gets updated more often. You can do this by copy-pasting in your text editor if you like, or make a script that does it. I usually have the whole deployment process automated on the server, which I'll write about some other time perhaps.

Another good way to do it is using Dojo ShrinkSafe. ShrinkSafe will not only append all your source files into one big file, but also reduce the size even further by optimizing the JavaScript code. However, I have had problems with scripts not running correctly (in all browsers) after running them through ShrinkSafe, despite their claims of it being "safe". I would especially recommend not using the "strip newline chars?" option. Anyway, the file size reduction the results from using ShrinkSafe is small compared to just gzipping the files (I'll show you some numbers later) so if you're afraid of regressions, don't use it.

3. Gzip your One Big File(s)
Now that you've got one or two or so JavaScript files, you need to gzip them into a gzip file. You can use the command-line gzip tool on the server to do this, like:
gzip -n9c scriptfile.js > scriptfile.gz.js

The -n option means to not store the the original file name in the gzip file. No big deal if you do, but there's no reason to do it either. -9 means maximum compression. I like doing things to the extreme! And -c means to write to standard output instead of modifying the source file, so we need to pipe it ">" to a .gz.js file at the end. Or you can use a UI tool such as 7-Zip that supports gzip compression. That's just damn easy.

Then upload the .gz.js script file to your server, and change all your old <script> tags for every file you had to one pointing to it. I.e. like:
<script type="text/javascript" src="scriptfile.gz.js"></script>
That's all there is to it!

Show Me The Numbers!
Let's use my Minesweeper game for comparison. The code is served in two files: libs.gz.js containing Prototype and Scriptaculous core and Effects libraries, and minesweeper.gz.js containing the actual game. The breakdown of the files in these libraries before combining them is as follows (at the time of writing):

Original
libs.js: 95 kB + 3 kB + 38 kB = 135 kB
minesweeper.js: 3 + 9 + 9 + 4 + 2 = 25 kB

Shrunk
libs.js: 92 kB (32% smaller)
minesweeper.js: 13 kB (48% smaller)

Gzipped
libs.js: 30 kB (78% smaller)
minesweeper.js: 7 kB (72% smaller)

Shrunk + gzipped
libs.js: 27 kB (80% smaller)
minesweeper.js: 4 kB (84% smaller)

Now, add to that the fact that every HTTP request for a file incurs a 1 kB overhead in HTTP headers, and you get an original total amount transferred of 135 + 25 + 8 = 168 kB compared to 27 + 4 + 2 = 33 kB, i.e. a total saving of 135 kB, i.e. 80% less bandwidth used, not to mention that making two http requests is much faster than making eight http requests!

The next step would be to bundle up html, css, scripts, and everything you possibly can into one request... (In fact, I have scripts doing that automatically at runtime on some sites...) But you have to draw the line somewhere. There's a definite trade off in maintainability, cachability, and compatibility of the site. So I'll stop here.

Caching in Php

Php by default tries as hard as it can to make the web browser not cache pages. While I can understand the rationale behind this a bit, sometimes you want caching. Caching is actually a good thing! you know. It means faster load times and lower bandwidth and processing requirements.

So I was surprised by how hard it is to turn off this aggressive non-caching policy. I googled for a few minutes and browsed the php documentation without finding an easy way of doing it. Ok, so you can use the following code snippet to enable caching in php (the argument to the function is number of seconds the page is valid):

function send_cache_headers($expire) {
  header("Cache-Control: max-age=$expire");
  header("Pragma: cache");
  header("Expires: " . gmdate('D, d M Y H:i:s \G\M\T', time() + $expire));
}


The Days of Web Standards

As I mentioned before, I was going to appear at The Days of Web Standards, 2007. And as also mentioned, I was working on an ajax messaging library. I'm sitting here at the Days in Akihabara now, listening to one of the last presentations for the day. There's quite a lot of people here, and there's a certain conferency mood in the air. I have the W3C on my left and the Mozilla Corp. on my right. Got some nice Firefox stickers.

I combined the two mentioned matters and did a presentation and demo developing a chat widget using the freshly-released Ajax Messaging Library. Well, it's in early beta quality with many features planned but not yet implemented, but it works! Even on stage (except for a small bug, hehe). Anyway, it's out there now! I'll write more about it later.

P.S. I did the presentation in my socks; my shoes were drenched this morning by Typhoon no. 4.

CSS Nite Vol 19 Recap, and Sequel

As I mentioned in my other blog, I was going to hold a presentation at CSS Nite Vol. 19. Considering I haven't held a presentation since my university days (what the heck kinda company am I working for, really?), I think it went really well, and it was very fun to do. I've uploaded the slides as well - not that they're much to see in themselves. I did a demo constructing an Opera widget live on stage, and contrary to what one would have expected, it actually went quite without mishaps.

I'll also have a presentation on The Days of Web Standards 2007 on 14~15 of July. My presentation is on Sunday the 14th at 14:30. It'll be a bit longer and also on widgets (although this time too I'll try to sneak in as much talk about web application development as possible without Marketing noticing). I've got some pretty nifty ideas for a live coding session and demo... :) I hope someone will attend. Anyway, there are many very interesting speakers scheduled...