Matt Cutts (of Google fame for those not in the know) has a short post on an interview about site maps and mentions one particular point about Googlebot treating the 401 (page not found, and it’s gone forever) and 404 (page not found, but it may reappear) status codes as a 401.
The reasoning is fine and it makes a lot of sense – most people use 404s when they should use a 401. However, this is like ‘quirks mode’ for HTML in web browsers. I believe the uptake of Firefox, which is much more strict on standards compliance, has forced web developers to pay more attention to standards.
Considering the huge reliance on Google for many webmasters, I believe Google should be taking this opportunity to teach webmasters to use the status code that suits in each situation. The result would be more considerate use of HTTP status codes and hopefully more effort devoted to fixing broken links.
All towards a better web…
A Pinch of Salt 