A user agent is a computer program representing a person, for example, a browser in a Web context.
Besides a browser, a user agent could be a bot scraping webpages, a download manager, or another app accessing the Web. Along with each request they make to the server, browsers include a self-identifying User-Agent HTTP header called a user agent (UA) string. This string often identifies the browser, its version number, and its host operating system.
Spam bots, download managers, and some browsers often send a fake UA string to announce themselves as a different client. This is known as user agent spoofing.
A typical user agent string looks like this: "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0".
If you go to the web test editor and find the failing requests, look at the form parameters. But the reason for this behavior is that the web test was recorded with IE6.
But the reason for this behavior is that the web test was recorded with IE6. When the web test was created, the recorder automatically fixed up the hidden field binding to pass the values from one request to another.
I kind of knew why the requests were failing but I wasn't aware of the fact that the hidden fields the web test was looking for were IE6 specific! I just discovered the reason why my notebook computer isn’t playing nice with certain ASP applications: I had apparently, in a fit of moral outrage over how Internet Explorer pretends to be Mozilla in its useragentstring, changed that string to something else.
We did the Flash part, and Intel’s web team built all the ASP and back-end infrastructure to support it. I changed the string back using the registry-editing instructions at the bottom of Eric Figure’s tool page, and now everything works.
This identifier is part of the HTTP protocol, and is a string that usually gives the name and version of the browser being used. Unfortunately, there is no real consistency in the format of this string, which makes analysis very difficult and statistics suspect.
There was much debate among developers and testers of Mozilla in its early days on what to do about its useragentstring (which starts with “Mozilla/5.0” even though this did not correspond to the actual version number of any Mozilla-based browser until many years later when Firefox 5.0 came out, rapidly succeeded by 6.0 with their current rapid-release strategy), with some wanting a “clean start” by changing its opening word to something else (even though the old pre-Firefox Mozilla Suite, once the flagship project of the Mozilla organization, was actually the only browser that could honestly call itself “Mozilla”), while others were deathly afraid to make the slightest alteration (even to change the version number with each release as Netscape always did) lest it discombobulate “browser sniffers” and lock Mozilla users out of sites. So it seems like we're stuck for the indefinite future with user agent strings that get further and further away from honestly describing the browser name and version they represent, and contain increasing amounts of fossilized deadwood that can't be removed because some site, somewhere, allegedly depends on its presence.
In the short run, such dodges help users get around clueless browser detection in Websites, but in the long run it causes those same clueless webmasters to see statistics that confirm their belief that “everybody uses ”, even if a large chunk of those users are really using something else but pretending to be using the popular browser. Thus, I have all the browsers I use configured to use a completely honest useragentstring wherever this is an available option (e.g., my copy of Opera used an “Opera” string with no mention of Mozilla or MSI, even before they made this the default), and wish that this were the default for all browsers (with a “spoofing” string, if available at all, only present as a settable option for the special purpose of going to a site that otherwise doesn't work).
Speaking of Opera, after a long time of defaulting to a “spoofing” identifier, they finally got honest and started using a logical useragentstring with “Opera/x.xx”. They did, however, trim a good deal of fat out of the Firefox useragentstring as of version 5.0, though still leaving some historical nonsense for “compatibility” with other browsers.
(And that isn't even considering various Web caching systems that make all site hit counts suspect, and the fact that any stats based on hits to inline images like counters or ad banners will exclude text-mode browsers, browsers with image loading turned off, and accesses by users with filtering programs that skip loading online ads, etc.) So some weird names like “Scooter” you might see in your logs are not “brand X” browsers, but indexers from a search engine.
Rather, it is simply to highlight specific features (intentional or accidental) of the linked sites which cause problems that could have been avoided by better design. The Oregon health insurance exchange site in 2013 says it is designed only for Internet Explorer, and shouldn't be used with other browsers.
The New York Post has intentionally blocked access from the iPad's Safari browser to get people to use their paid app to read the paper, but bizarrely left it readable by other browsers like Sky fire and Opera Mini which can run on that device. The FedEx site reportedly shoos away people using the “wrong” operating system, such as Linux users, telling them that they're using an unsupported browser and should switch to IE or Firefox (even if they're using a Linux version of Firefox).
Hall of Shame Dishonor Roll Champion: The FEMA disaster relief application required MSI 6.0 at the time of Hurricane Katrina, and turned you away if you were using anything else. In a just world, the person responsible for this would be sentenced to a week of living in what remains of the New Orleans Super dome among the piles of excrement left by the evacuated refugees.
They still deserve a “shame” note for ever designing a site with such a stupid restriction. Hilton's secure site redirects anybody whose user agent string doesn't start with “Mozilla” to this so-called Web Standards Page ; I guess I missed the part of the W3C specs that made it a “Web Standard” that user agent strings must start with “Mozilla”.
Big Noise Music sends anybody not using IE for Windows to a page that says you need “Internet Explorer 5 (or better)”. Modeling blocks every browser but MSI, every platform but Windows, and also refuses you if cookies or scripting are disabled, your connection speed seems to be too slow, or you seem to be outside the United States.
Reports are that even the new beta release of Internet Explorer is blocked, as the developers of this site seem to take the attitude “Ban everyone and everything, unless specifically permitted.” Another site that brushes off Opera but lets in Mozilla is Photonic (Getty Images).
Its “Get Lost” page tells you to get IE or Netscape, failing to mention Mozilla or Safari. Since their offerings are of particular interest to graphic professionals, many of whom use Macs which come with Safari as their standard browser, it makes no sense for these people to go out of their way to tick them off like this.
(This latter piece of advice can be translated: “Please make your browser lie about what it is in order to get past the cluelessness of idiot webmasters like us.”) Nevertheless, they still said that they had some pages that are blocked from nonbasic users because they “do not show these pages the way we would like them to be shown” and “cannot handle some web technologies from Microsoft” -- in other words, the site author still couldn't keep himself from using proprietary stuff and depending on browser quirks.
This credit union site tells many users (including those of the Mozilla Suite) that their browser is “nonstandard”, then gives them a link to enter the site anyway; however, in some cases, this link doesn't work (I think it depends on cookies being enabled). Fidelity Investment's benefits section is reported to turn away the “wrong” browser types, but it seems to work for me in Mozilla.
So I guess Mozilla isn't the “wrong” browser, but reports are that Opera is, at least when it's set to identify itself honestly. Google Maps has a broader range of supported browsers than most of the “browser-sniffing crowd”, but if you're using something other than IE, Mozilla, Firefox, or Netscape, or too old a version of them, you still get brushed off.
Put simply, the 4G Blacklist is a more effective way to protect your site against a wide variety of spam, exploits, and malicious attacks. Unlike huge lists of banned user agents, the 4G Blacklist requires zero maintenance, consumes fewer resources, and may retain its effectiveness indefinitely.
As you may recall, the original Ultimate .Htaccess Blacklist was released here at Perishable Press a couple of years ago. Then, several months later, I added more bad user agents, compressed the list into single-line format, and released the Ultimate .Htaccess Blacklist 2.
This new list features a whopping 1211 blacklisted user agents, including three of my own creation 2 to be used exclusively for my diabolical and obsessive monitoring purposes (insert maniacal laughter here). Simply copy and paste the following code into the root .Htaccess file of your site to enjoy a serious reduction in wasted bandwidth, stolen resources, and comment spam.
:) 2 Free iPod NATO plus honorable mention in my next article for the first person to identify correctly the three “imaginary” (i.e., fake) user agents.