Yesterday coleague of mine came to me with interesting problem.
He created app in Silverlight using WCF RIA services with some Telerik controls. When under development, in our environment everything worked fine. Then few day ago we were testing the app deployed at our customer. And some windows using grids showed us ugly windows with errors saying that server returned internal error. But when accessed from inside of the organization, everything worked fine.
What the heck? :/ Now what?
So I was the one coleague had chosen to help him, as always. :D
First thing that I told him – fire up Fiddler. Then we looked at requests and found out, that only requests that had special characters URL encoded returned error 500.
In this example we want to filter letter Z in query :
So, this is the request :
GET /ClientBin/BLABLABLAAPP-Web-Services-MainService.svc/binary/GetBLABLABLA?$where=(it.IdBLABLABLA%253d%253d%2522Z%2522) HTTP/1.1
And server responded with normal HTML page telling that 500 error occured and response like this :
HTTP/1.1 500 ( The request was rejected by the HTTP filter. Contact the server administrator. )
/* Please note that I used string BLABLABLA to “obfuscate” our real names. And secondly note that we din’t created BLABLABLAAPP-Web-Services-MainService.svc file itself, the WCF RIA services had and you can get more info about it here :
You can try to create the request with your browser and watch traffic with fiddler, its the same as Silverlight does. :) And you can also try to for example use unencoded url like this to find out, if the server or something in the way isn’t blocking the traffic. In our case request like this :
GET /ClientBin/BLABLABLAAPP-Web-Services-MainService.svc/binary/GetBLABLABLA?$where=(it.Type==”Z”) HTTP/1.1
passed. OK, we are getting closer.
Then we checked IIS 7.5 logs and found out that unencoded request made it to the ISS but encoded din’t. So undoubtly, something was blocking out requests and this something was ISA server of our customer.
Another coleague that is responsible for the infrastructure at our client fixed the problem by unchecking the Verify normalization checkbox in ISA server configuration.
More on this toppic here : http://technet.microsoft.com/en-us/library/cc302627.aspx where under Verify normalization you can find this information :Web servers receive requests that are URL encoded. This means that certain characters may be replaced with a percent sign (%) followed by a particular number. For example, %20 corresponds to a space, so a request for http://myserver/My%20Dir/My%20File.htm is the same as a request for http://myserver/My Dir/My File.htm. Normalization is the process of decoding URL-encoded requests.
Because the % can be URL encoded, an attacker can submit a carefully crafted request to a server that is basically double-encoded. If this occurs, Internet Information Services (IIS) may accept a request that it would otherwise reject as not valid. When you select Verify Normalization, the HTTP filter normalizes the URL two times. If the URL after the first normalization is different from the URL after the second normalization, the filter rejects the request. This prevents attacks that rely on double-encoded requests.
Note that while we recommend that you use the Verify Normalization function, it may also block legitimate requests that contain a %.
So remember, HTTP traffic doesn’t lie, fire up your favourite HTTP debugger and analyze what is really going on behind the scenes.
Good luck ;)