We'll Always Have C

The other day there was an interview in Dr. Dobb's Journal with the managing director of TIOBE Software, who publishes the TIOBE Programming Community Index, a ranking of programming language popularity. It was also discussed on Slashdot.

The methodology used by TIOBE to calculate a language's popularity is basically the good old google hits ad-hoc voodoo index, using "[language] programming" as the query. This measures the "web presence" of a programming language.

First of all, it's obvious to you and me that this measures something, that something being the amount of web pages including the term "[language] programming", obviously. There's nothing wrong with this method, as long as one is aware of what they're measuring. But is it fair to call this the popularity of a language?


Look at this blog, for example. I mostly mention JavaScript and PHP here, just like everyone else. Throw in some Ruby and Python too to max out the buzz factor. There is no mention of relics such as C in this blog. But you know what language I use ten times more than any other? C. I'd love to have a job hacking away in JavaScript, Ruby, and Python all days, but I'd have to settle with half the salary. So here it goes: C programming. Index that. Embedded, heavily multi-threaded, efficient, minimum memory, hardcore badass C programming, that's what I do, and I love doing it.

Most coders can't do C. That's why you see all these Visual This and Dot That and scripting languages on the ranking, because these kids blog about every little insignificant hobby project they manage to cut and paste together, just like I do. But let there be no mistake about it: real programmers can code in C. They do syntactically correct typedefs of function pointers in their sleep. (just kidding that's impossible.)


At work I also hack in Python, Perl, and Makefile. At home it's mostly JavaScript, PHP, Ruby, Python... Lately Python has replaced Ruby as my language of choice for home hacking because of its decent unicode support. (Although I've had to hack the Python standard library in some places where it didn't properly support unicode. I read the next version of Python (2.6?) will use unicode strings by default, which is great, and only ten years late.) I also sold my soul the other day and installed Visual C# 2008 Express Edition for some hobby hacking. Turned out not very fun though, but I haven't given up yet.

At my previous job I used C++ for doing essentially the same thing as I do in C now. I'm completely convinced that C is the right tool for the job. I'm also convinced C does object orientation better than C++, but that is a topic for another post. And I used to be a Java fan, but now I'm considering Java the best examples of software suckiness ever. It's a volatile industry, technologies come and go, but no amount of blogging will convince me that the C programming language is anything but #1.

I'm saying it because it's true: We'll always have C. Because we've got jobs to do.

Reducing Bandwidth Usage when Deploying Web Applications

... by serving your JavaScript source code files as one big gzipped file. Nowadays when we're getting more and more nice applications (and widgets, gadgets, mashups, and whatnots) running on the web, there's a lot of talk about user experience and stuff, which involves the whole experience of using a web site. Good content, design, and fancy programming, of course make up the core of a good user experience. But we all wants sites to be quick and responsive to load too, right?

This trick is so simple and incredibly easy to pull off that you have no reason not to do it. There are essentially three things involved:

1. Make apache serve .gz.js files with gzip transfer encoding
2. Bundle up all your JavaScript files into one (or a few) big file(s)
3. Gzip your One Big File(s)

Let's go through these step by step...

1. Make apache serve .gz.js files with gzip transfer encoding
Gzip transfer encoding is great! The only bad thing about it is that on dynamic content, the web server will have to use up some processing power to compress the files on the fly, but in this case, we're serving static JavaScript source files, so that's not even an argument! (As for the client, if it's got enough CPU to run your web app, it's not gonna have any trouble doing a little unzipping...)

The easiest way to do this is to edit the .htaccess config file in the directory where you put your JavaScript files, or any directory above that one to make it more global. The only thing you have to do is add the following line:
AddEncoding x-gzip gz

This adds the gzip transfer encoding to apache for files with a "gz" file extension.

Notice the title says ".gz.js" though: by adding ".js" at the end, the files will be served correctly with the text/javascript mime type. Otherwise the browser might go berserk. That's also why I wouldn't recommend using ".js.gz"; it'll get served with gzip transfer encoding alright, but not with the JavaScript mimetype.

This solution works even with clients that don't support the gzip transfer encoding, because web browsers send an "accept-encoding" header that tells the server which encodings the client supports, so apache won't use gzip transfer encoding if it's not supported by the client.

2. Bundle up your JavaScript source file into one (or a few) big file(s)
This means simply appending all your source files to one file. Or a few files. I usually bundle up the libraries/utility code etc, i.e. code that you don't change very often, into one file and the rest of the app into one file that gets updated more often. You can do this by copy-pasting in your text editor if you like, or make a script that does it. I usually have the whole deployment process automated on the server, which I'll write about some other time perhaps.

Another good way to do it is using Dojo ShrinkSafe. ShrinkSafe will not only append all your source files into one big file, but also reduce the size even further by optimizing the JavaScript code. However, I have had problems with scripts not running correctly (in all browsers) after running them through ShrinkSafe, despite their claims of it being "safe". I would especially recommend not using the "strip newline chars?" option. Anyway, the file size reduction the results from using ShrinkSafe is small compared to just gzipping the files (I'll show you some numbers later) so if you're afraid of regressions, don't use it.

3. Gzip your One Big File(s)
Now that you've got one or two or so JavaScript files, you need to gzip them into a gzip file. You can use the command-line gzip tool on the server to do this, like:
gzip -n9c scriptfile.js > scriptfile.gz.js

The -n option means to not store the the original file name in the gzip file. No big deal if you do, but there's no reason to do it either. -9 means maximum compression. I like doing things to the extreme! And -c means to write to standard output instead of modifying the source file, so we need to pipe it ">" to a .gz.js file at the end. Or you can use a UI tool such as 7-Zip that supports gzip compression. That's just damn easy.

Then upload the .gz.js script file to your server, and change all your old <script> tags for every file you had to one pointing to it. I.e. like:
<script type="text/javascript" src="scriptfile.gz.js"></script>
That's all there is to it!

Show Me The Numbers!
Let's use my Minesweeper game for comparison. The code is served in two files: libs.gz.js containing Prototype and Scriptaculous core and Effects libraries, and minesweeper.gz.js containing the actual game. The breakdown of the files in these libraries before combining them is as follows (at the time of writing):

Original
libs.js: 95 kB + 3 kB + 38 kB = 135 kB
minesweeper.js: 3 + 9 + 9 + 4 + 2 = 25 kB

Shrunk
libs.js: 92 kB (32% smaller)
minesweeper.js: 13 kB (48% smaller)

Gzipped
libs.js: 30 kB (78% smaller)
minesweeper.js: 7 kB (72% smaller)

Shrunk + gzipped
libs.js: 27 kB (80% smaller)
minesweeper.js: 4 kB (84% smaller)

Now, add to that the fact that every HTTP request for a file incurs a 1 kB overhead in HTTP headers, and you get an original total amount transferred of 135 + 25 + 8 = 168 kB compared to 27 + 4 + 2 = 33 kB, i.e. a total saving of 135 kB, i.e. 80% less bandwidth used, not to mention that making two http requests is much faster than making eight http requests!

The next step would be to bundle up html, css, scripts, and everything you possibly can into one request... (In fact, I have scripts doing that automatically at runtime on some sites...) But you have to draw the line somewhere. There's a definite trade off in maintainability, cachability, and compatibility of the site. So I'll stop here.

A JavaScript messaging library

I'm working on a library for sending messages between browsers using JavaScript, XMLHttpRequest, and PHP/sqlite on a server. The idea is that even though it's not possible to open a connection of sorts directly between the clients (i.e. web browsers viewing a page, or even different pages, or running a widget), you can pull for updates in a way that will make it seem like you have a connection open. Not only that, but all client listening on the same channel, so to speak, will receive the messages, so it's like broadcasting. Technically this is nothing fancy, it's just that you don't see it much, and if implemented and packaged nicely I think it'll be useful and fun to use.

How do I know this works? Because it's based on a much cleaned up version of the code used on paintmyblog.com - which has proven itself already. I look forward to writing some cool apps using this library, and I hope other people will use it as well. Expect the first release next week.

Useful JavaScript Programming Sites

There are some sites I tend to consult all the time while programming in JavaScript. Let's see what they are...

Core JavaScript 1.5 Reference - Mozilla Developer Center
A thorough and complete reference for the JavaScript language and standard library, with examples and links to more information, etc. Really good.
http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Reference

Gecko DOM Reference - Mozilla Developer Center
Complements the above reference with a complete reference to the document object model, i.e. all the methods and classes that are available in JavaScript for manipulating the web page in the browser.
http://developer.mozilla.org/en/docs/Gecko_DOM_Reference

Canvas Class Reference - Apple Developer Connection
If you like playing with the canvas tag like I do, then this is the place to go for reference documentation on the 2d graphics context object. Complete and handy with all in one page.
http://developer.apple.com/documentation/AppleApplications/Reference/SafariJSRef/Classes/Canvas.html

Prototype JavaScript Framework API
The reference documentation for the Prototype framework. Of course, this is only useful if you're using Prototype. But if you're not, you better have a really good excuse... such as using some bloated alternative like the Yahoo UI Library or Google Web Toolkit.
http://www.prototypejs.org/api

Dojo ShrinkSafe
Dojo ShrinkSafe shrinks the sizes of your JavaScript source files by removing whitespace and renaming local variables to shorter names, and some other stuff. Saves bandwidth and loading time without really giving much up, except debuggability. But you're not doing debugging on your live site anyway, right? Riiight?!
http://alex.dojotoolkit.org/shrinksafe/