It turned out that the only reason for this was that the size of the custom error page I had designed was lower than the threshold set by IE and Chrome for the 404 page. Basically, the error pages must be greater than 512 bytes to be viewable on all browsers. For IE, the settings of the error pages are stored in the registry at
The following image shows the registry entries for the various error codes in RegEdit:
The setting in Chrome (for 404) is tied into the source code and can't be modified just as easily. Check out the comment in this source code of Chrome around line 259.
// If it's a 404 page, we wait until we get 512 bytes of data before trying
// to load the document. This allows us to put up an alternate 404 page if
// there's short text.
Surprising that Chrome had this setting as well, especially considering that Safari also uses the same WebKit HTML and CSS rendering engine and it doesn't exhibit this quirk.