Rational Action

I’ve been analyzing the WBAR server logs, trying to find out how popular we are and which browsers people use to access the site. Looking through the Browser report, I found something interesting…

Listing the top 20 browsers by the number of requests for pages, sorted by the number of requests for pages.

no.:    reqs:  pages: browser
---: -------: ------: -------
  1: 1682570: 249863: MSIE
  2:  568298: 152831: Mozilla
  3:  102801: 100051: msnbot
  4:   80980:  80920: Googlebot
  5:   70576:  65242: Netscape (compatible)
  6:  163830:  40430: Safari
...
 20:     246:    246: Jesus is the only way to God.

I’m not sure what to make of this, but it seems that a bunch of people have set their web browser’s referrer to “Jesus is the only way to God.” Thoughts?

Apologies to all readers of Mozilla Planet. I changed my blog so that it used permalinks, and now my old Mozilla posts are flooding the main page. This is probably due to a bug in the Planet software.

For those interested, I’ve got a thesis for a new Mozilla post bumping around the walls inside my skull. Let’s hope Tor doesn’t kick me off Moz Planet just yet.

Just letting everyone know: the new browser prototype issued by “Netscape”, which is based on Firefox 0.9.3, does not include any security updates created since that release. Don’t install it on friends’ computers and don’t use it on untrusted sites.

Even worse, users are reporting that it embeds IE via a plugin accessible to any website. That makes it vulnerable to both 0.9.3 and IE security holes.

You can verify yourself by checking the Mozilla Known Vulnerabilities page with the “Netscape Browser”. Click on a recent bug number and run its testcase. (For example, the Javascript clipboard access testcase worked, and the extremely wide BMP crashed the browser) You can also watch a random website embed IE by visiting this example. Users of the latest Firefox release may click all of these links in confidence.

It looks like a few people liked my Mozilla Bug Fix timeline. And a few didn’t. Unfortunately I ended up posting the thing at something like 3AM, and I didn’t get to write out everything I wanted to say. Now I have some incentive to finish.

The Mozilla developers did a great job handling this extremely critical security hole. After getting word of an exploitable hole, they pushed out a fix in less than 36 hours. Applause. The system worked.

…but not perfectly. The critical bug got fixed in a jiffy, but it was actually the result of known weaknesses in the way Mozilla handles external protocol handlers. These weaknesses came up many times on Bugzilla, starting around 2 years ago. The developers quelled some concerns by implementing an insecure protocol blacklist, but this solution did not fix everything. What I propose here would not have plugged this security hole entirely, but it would have mitigated it to a great deal. I can’t see how the Mozilla team could have prevented this problem entirely, but had they handled several known bugs, it could have been a minor issue.

Those who feel Mozilla completely botched this bug often point to Bugzilla bug 167475 and bug 173010. Although the latter bug would have fixed the problem completely, it also would have made Mozilla far less usable. The former bug would have mitigated but not fixed the problem. Let’s take a deeper look into the issue:

An Internet Explorer user on Windows has many protocols available to him within his web browser: the protocols IE supports itself, certain add-on protocols supported by Windows, and more add-on protocols implemented by other applications. See http, shell, and ed2k as respective examples. On Windows, Mozilla recognizes these protocol handlers in addition to its own. And, as these bugs point out, Mozilla will launch the associated application whether the link to the handler is in an A tag, where it would make sense, or in an IFRAME, where it really wouldn’t. Since the shell: protocol is insecure, an unpatched Mozilla user could theoretically infect himself by simply going to a webpage with a malicious IFRAME.

The solution in bug 167475 initially is to block external protocol handlers outside of A tags. It’s not a real fix, though: one commenter notes that “It’s not hard for a malicious site to get a visitor to click a link”. Fixing the bug in that way would have resulted in a false sense of security, as websites could still trick users into clicking on malicious anchors. Still, Mozilla should have blocked external protocols from tags like IFRAME. There’s just no reason to allow them. That wouldn’t have fixed this vulnerability, but it would have reduced its severity greatly. It is harder to exploit this hole when the user must click.

Another potential solution is to use a whitelist of allowed protocols (See bug 173010). All other protocols would get blocked. But what happens to our precious convert from IE, who clicks on his ed2k link but sees nothing because ed2k isn’t in the whitelist? Or the GNOME user, who expects all of gnome’s silly little protocols to work when he types them into the address bar? Any Mozilla with a whitelist would quickly become out of date as new protocols became important; the Mozilla developers would be swamped with maintaining an ever-growing list.

The reporter of the blacklist bug then modifies his strategy in a later comment, saying that if a link referring to an external protocol handler is clicked, Mozilla should pop up a warning box. Now he’s on to something.

But what happens in this new case, if we are using a blacklist? The user clicks on a malicious link. Mozilla pops up a warning dialog. The user thinks, “I want to go to that link. Let me click OK without reading the warning first.” The user has now been infected, even though the fix stated in bug 167475 has been implemented.

This confirmation dialog could have been included in the whitelist plan as well. Unfortunately, that puts us into an even more dangerous position: with just a whitelist, the user would see the same dialog for a (dangerous) shell: link as for a (benign) ed2k: link. Inevitably many Mozilla users would accidentally accept the shell: links. On the other hand, Mozilla should get confirmation from the user whenever it is about to launch an external program. I would appreciate the notification, and it wouldn’t make me less secure.

Which brings me to my proposed solution:

1. Block external protocol handlers from tags like IFRAME where they do not make sense. (See bug 167475)
2. Continue to utilize a blacklist of known dangerous protocols. Do not allow Mozilla to trigger them. We can’t give users the option of using known-dangerous protocols, because they will inevitably find a way to use them. (Implemented in bug 163648)
3. Implement a whitelist of known-safe protocols. This should include protocols Mozilla handles along with ones needed for GNOME integration. (See bug 173010)
4. Create UI so that if a protocol is not on the blacklist or the whitelist, Mozilla prompts the user as if it were about to open a helper application. (See bug 167473) Malicious links will inevitably be accepted by many, but the confirmation will stop many more.

Even with this solution, Mozilla still would have been vulnerable to the latest attack. “shell:” was not on the blacklist (and it certainly would not have been on the whitelist.) Users would thus see a confirmation dialog for opening the link. The careful ones could avoid infection, but many would simply click “OK”. Which brings me to my most important gripe:

Get automated critical updates working!

Had the Critical Updates system been functioning at the time of the exploit, its severity would have been minimal, even with only a blacklist. Thankfully, the developers are hard at work getting it up and running. (Or is it working already? I never saw a critical update for the vulnerability. Did anyone else?)

I made my first post about the shell: exploit because I thought Mozilla deserved credit for their open development process. With that process so open, I’ve found that it’s also much easier to find areas of weakness — and that’s a plus. With constructive criticism from the outside, the process can change due to changing needs. I hope the Mozilla team gives itself a pat on the back for its handling of the shell: vulnerability, but I also hope that it will be more responsive to the long-term security holes that many users often point out.

Update: I have created an additional document that points out some failures surrounding this security hole. It should balance well with this post.

I’ve had many conversations lately about the security policies surrounding the Mozilla family of web browsers. The number of people fed up with the terribly vulnerable Internet Explorer has reached a critical mass. Now they want to know why Mozilla is better.

I suggest that those people who think “If IE is so bad, then Mozilla probably is too” take a look at this timeline. It tracks the identification, fixing, and notification of a potentially serious security hole affecting all of the Mozilla Organization’s consumer applications: the Mozilla Suite, the Firefox browser, and the Thunderbird e-mail client. I compiled it using the same publicly accessible, easy-to-find pages that Mozilla developers use all the time.

Here we go:

July 7 - 13:46 GMT - Keith McCanless files a bug in the Bugzilla Database reporting a new vulnerability. It exploits the windows “shell:” handler and allows a malicious web page to execute a program on a client’s computer (The program has to already be present on the computer). McCanless notes that the bug is “BOTH a security concern and a DOS,” since if the link points to a nonexistent file, it makes the Mozilla browser spawn off endless amounts of new windows. The bug is marked private since it is security-related; only developers with proper clearance can see it. (source)

July 7 - 16:26 GMT - Josh Perrymon sends the first e-mail to the “Full-Disclosure” mailing list about the vulnerability. (He presumably reported this bug independent of Keith McCanless.) The mail states that Mozilla will execute programs when passed the proper shell: URL. The vulnerability is now known to the world, ready to be exploited. (source)

July 7 - 18:16 GMT - Mozilla developer “timeless” creates patch closing vulnerability. He posts the patch on the Bugzilla Database so that other developers can approve it. (source) The bug had been known to the world for a matter of hours before a patch was created to fix it

July 7 - 18:19 GMT - Mike Shaver gives the patch a “superreview” flag, which is effectively a “thumbs up” for merging the code into Mozilla. The code may now be checked in to the Mozilla development and stable branches. (source)

July 7 - 18:55 GMT - Patch checked in to Mozilla development branch. (source)

July 7 - 18:58 GMT - Patch checked in the Mozilla Suite stable branches for versions 1.4 and 1.7 (source)

July 7 - 19:25 GMT - Back on the “Full-Disclosure” mailing list, Andreas Sandblad notes that the exploit “is dangerous” and explains how an attack might work (source)

July 7 - 22:07 GMT - Patch checked in to “Aviary” (Firefox and Thunderbird) branch (source)

July 8 - 01:59 GMT - Patch checked in to a new Mozilla branch; this branch represents the code for the new version, 1.7.1. Developers will then download this code and compile a new, secure version of the Mozilla Suite. (source)

July 8 - 03:23 GMT - A new branch is created, out of which developers will build new versions of Firefox and Thunderbird. The patch is checked into this branch. (source) In less than 11 hours after the vulnerability was reported to the public, all up-to-date Mozilla code was secure

July 8 - 10:56 GMT - Developers place binaries for Mozilla 1.7.1, the new secure release, on Mozilla FTP site. (source)

July 8 - 13:30 GMT - Thunderbird 0.7.2 binaries placed on the Mozilla FTP site. This release contains the patch for the security hole. (source)

July 8 - 14:13 GMT - An XPI “add-on” is uploaded that allows Mozilla Suite users to patch their browser without downloading an entire new release. (source)

July 8 - 14:27 GMT - Firefox 0.9.2 binaries placed on the Mozilla FTP site. This version contains the security patch. (source)

July 8 - 15:13 GMT - Thunderbird XPI “add-on” package placed on FTP site. This fixes the bug without requiring Thunderbird users to download an entire new setup file. (source)

July 8 - 16:13 GMT - Firefox XPI “add-on” package placed on FTP site. Once again, this fixes the bug without making users download a whole new setup file. (source) Before the vulnerability was known to the public for 24 hours, Mozilla had released updated versions of its poducts and patches for users running previous versions

July 8 - 17:38 GMT - Asa Dotzler opens the security bug to the public. Now anyone can go to the Bugzilla page and see the details of the vulnerability. By this point, however, patched versions of all affected software were available, and the bug had already been disclosed to the public on the “Full-disclosure” mailing list. (source)

July 8 - 20:53 GMT - David Baron updates the Mozilla.org main page so that it links to fixed copies of the Mozilla Suite, Firefox, and Thunderbird. (source)

July 8 - 21:57 GMT - Asa Dotzler checks in an official Mozilla.org notice of the vulnerability and the fix (source) In the course of less than a day and a half of public vulnerability, all Mozilla versions were updated, a security note was released, and new downloaders were secure by default

I think the Mozilla team deserves a round of applause for their speedy handling of a potentially serious problem. Furthermore, they have accomplished this feat in a transparent manner. And all at no cost to their users. From what I can see, most security bugs are handled this way, and that makes me happy to be a Firefox user.

Convinced? Get Firefox!

Ever since I received the prestigious title of web master/mistress for next year at wbar.org, I dreamed of updating the streaming server. I wanted, first of all, to have a unix-like operating system to play with. I also wanted to stream in Vorbis. Using Windows made doing such a thing very tough, because in order to stream MP3’s on Shoutcast, you (normally) run Winamp and configure it to use a special output plugin for encoding. Well, the output plugin only knows how to encode MP3, so encoding Vorbis requires another plugin. But you can only use one plugin at once, and when it comes down to it, you have to choose between the two formats.

I found a nifty program called Darkice that could simultaneously encode MP3 and Vorbis. It required an Icecast server. I wanted to run all of this on FreeBSD, but the system’s sound card had only commercial drivers on that platform. Linux with ALSA it would be.

After consulting the techie brain trust, I decided on Debian testing, the pretty-stable-but-we-can’t-guarantee-it version of Debian. I was excited. I read Crime and Punishment and sat through a few hours of class, anticipating the joy of installing Linux (oddly enough, the previous statement bears some truth).

The station has two servers: a “real” Dell server, and an old Compaq desktop computer. They both ran Windows 2000, but the Dell box did pretty much all of the tough work. I started by downloading a Debian netinstall image and setting up the Compaq box to act as a temporary server. When it came time to burn the netinstall image, though, it turned out that wbar didn’t actually have any burners in their computers. Somehow one turned up in the closet, and after about an hour of fiddling (don’t ask) I managed to get it installed in one of the servers (don’t ask).

With the netinstall CD burned, I turned off the Dell server and gave the Compaq its old IP address. Instant changeover! The listeners wouldn’t know the difference… or they shouldn’t have, if I had done things right. But there were only around 5 listeners, and, well, you know…

I used the rescue mode of my SUSE install CD to resize the Dell’s main partition. GNU parted worked like a charm and within minutes I had space for Debian. I rebooted into the Debian installer and immediately noticed that it was different from previous versions. The dreaded kernel module configuration dialog was no longer, and I didn’t have to enter any partitioning information. Excellent! My only gripe was that I couldn’t figure out a way to add my own source for grabbing the base system.

Once base was installed the system rebooted and I did some final configuration. I chose packages to install, which worked fine except for the fact that I needed to do several runs through the package installation step. I’m not sure why this always happens with Debian — for large apt-get’s, you always seem to need to run apt-get several times. Eventually apt-get kept failing on inn, so I just removed it. My system was installed.

X worked more or less out of the box, although some mouse driver tinkering in XF86Config-4 was necessary. I installed ALSA so that I could drive the high-end sound card. I also downloaded alsa-tools and compiled envy24control by myself — Debian doesn’t include it for some reason. Well the damn program kept on refusing to detect the sound inputs and segfaulting! After endless tinkering, I realized that it was a version incompatibility — 1.0.4 of alsa-tools, 1.0.3 of alsa-base. Jesus. One minor version! The slightly older envy24control worked like a charm, showing the input levels and letting me adjust gain. Why Debian won’t include alsa-tools, I have no idea.

With sound set up, I began working on Icecast/Darkice. Icecast2 wasn’t included in testing, so I downloaded the deb from unstable and installed it. I gave the streaming server a base configuration, then configured Darkice. Oh yeah, because Debian has some worries about patents, they didn’t compile Darkice with LAME (mp3 encoding) support. Thus making it useless. Compile-from-source, here we come.

I downloaded the sources for Darkice and LAME, compiling them both into Debian packages. In all my years using Debian I had never compiled a Debian package from source (not counting Kernel), so this was a big thing for me. Surprisingly easy! I modified the Darkice package so that it used LAME, and soon thereafter I had a functioning MP3 streaming server.

With the high bitrate MP3 stream working, I added a low bitrate MP3 stream and a high bitrate Vorbis stream, all of which the server handled very well. Unfortunately Darkice can’t downsample/downmix audio for Vorbis (even though the Vorbis encoder supports just downmixing), so I couldn’t get a low bitrate Vorbis stream running.

I made an init script for Darkice by basically copying the init script for icecast-server. I really like start-stop-daemon: I was able to figure out how to use it in almost no time. I made some simple modifications and quickly had Darkice automatically starting up and shutting down. After a reboot to test if the server knew what to do automatically, I declared my work more or less done.

I switched the stinky Compaq machine back to DHCP, and gave the Dell machine back its old IP address. By the way, Debian’s “interfaces” file is an amazingly simple way to manage network configurations. I had my IP info typed in in a few minutes, even though I had little experience with this system beforehand. As a final measure, I added some aliases on the server to make the Icecast server look more like the old Shoutcast server (e.g. requesting http://server:8000/ gets you an MP3 stream, instead of a 404 Not Found error). I tried the links on the site for one more sanity check. Everything a-ok. Time to go.

It was 9:20PM when I left. I had arrived at 12:30.

(This is probably going to be boring. Skip to the “My Analysis” section for something short.)

Yesterday I was working on my new site when I discovered that Mozilla 1.6 alpha wouldn’t publish it properly. I tried to downgrade to Mozilla 1.5, but it would go pause on startup, remaining alive but using 0% CPU. I decided to reboot.

When Windows XP started up, I got an error that shlwapi.dll could not be found, and hence winlogon.exe (the program that logs you into your computer) could not run. Huh? Where did it go?

Well, it was gone and I needed to get it back. I booted into Knoppix (a Debian-based Linux distribution on a CD) and checked out my directories. shlwapi.dll had ceased to exist. I made a friend send me her copy of it. Then I did something pretty stupid:

I enabled experimental NTFS write support on my Windows XP partition.

The attempt at copying shlwapi.dll back failed with some system error, and Knoppix refused to show my c:windowssystem32 directory from that point forward. Ooops, just a little filesystem corruption.

I used an old, burned copy of Windows XP to attempt to get to the recovery console, but lo and behold it wouldn’t accept my Administrator password. Huh? ntpasswd did the trick, resetting the Administrator password.

After some fiddling and following of the directions on some forum, I was able to get my system to start up and permit me to login. But Windows then refused to give me a start menu because it was missing shdocvw.dll. Huh? Looks like I wasn’t having a problem anyone had diagnosed before! I copied the file over from an XP cd, but Internet Explorer still wasn’t working!

I threw up my hands and did an in-place reinstall of XP. An hour later I was presented with an activation dialog, which refused to work over the Columbia ethernet because XP seemed to think I needed to dial up to my ISP. I called ‘em up and all went smoothly. Norton Antivirus popped up a dialog box saying that the Welchia worm had been detected. I quickly installed a few patches to keep my computer from being infected.

Mozilla still refused to work, so I created a new profile and copied the necessary files from my old one. I should not have had to do that. At least it’s working now.

My analysis:

Microsoft has released software with tons of security holes. They’re doing the right thing by patching them, but these patches may have little conflicts that can cause big problems. Especially with computer users like myself, who like to install things but reboot later. Somehow or another, some rather important Windows system files were deleted rather than updated.

The solution: more review of code, better testing of patches, and an increased focus on getting things right the first fucking time. I should not have to download 2 patches per week. How come the average Linux box can get by with so many fewer patches, when it has to focus on remote and local holes? Sigh.

(As for Mozilla crapping itself: at least I had the know-how to fix it. I was using an Alpha version, so if it happened because of that, I have only myself to blame.)

I have started working on a website. If you can’t tell, I’m going for the minimalist design — it seems to require the least amount of work. Right now I’m making it Mozilla Composer.

I visited Frankie and Erica yesterday and fixed up their computers a bit. And I came to a conclusion. Browse the web with Mozilla: it blocks popups and suffers from none of the security holes that afflict Internet Explorer. As the web stands right now, if you browse with Moz, you will be basically immune to any web-based exploit.