[mod-security-users] Perplexing Issue: "400 Bad Request"

Discussion:

Art Age Software

2010-05-07 17:09:38 UTC

I've been using mod-security successfully for a couple years now as an
additional layer of protection for a web-based app. Recently, a new
behavior has emerged that I do not understand. I am using core
ruleset/1.6.1 and have made no changes to this configuration for
several months, so I do not know why this behavior has suddenly
emerged.

What is happening, according to the logs, is that certain requests are
being rejected with "Access denied with code 400 (Request Missing a
User Agent Header)." However, these log entries are immediately
preceded by successful GETs from the same IP address. I have followed
up with the users behind these IP addresses and all report **no**
strange behavior and **none** is seeing a "400 Bad Request" error page
or any similar error page. In other words, they all report that the
application is performing perfectly normally.

Mod-security is definitely "On" and the apache web logs also show that
these requests are being rejected with status code 400. I have
included an example (partially scrubbed) of the log entries for one
such instance below. If anybody has any idea for what could possibly
be going on here, I would appreciate hearing it. Thanks.

------

Mod-security Log Entries:

--986cbd32-A--
[07/May/2010:16:39:53 +0000] tWCnzAoABQMAABVYW3wAAAAu xxx.xxx.xxx.xxx
50018 yyy.yyy.yyy.yyy 80
--986cbd32-B--
GET /this/page HTTP/1.1
Host: my.domain.com
Accept: */*

--986cbd32-F--
HTTP/1.1 400 Bad Request
Set-Cookie: SESSID=vlu8kib6b2ovekkbad8mnc43c3; path=/; domain=.domain.com
Expires: Fri, 31 Dec 1999 23:59:59 GMT
Cache-Control: max-age=0, must-revalidate, no-cache, post-check=0, pre-check=0
Pragma: no-cache
Vary: User-Agent
Content-Length: 3887
Connection: close
Content-Type: text/html; charset=UTF-8

--986cbd32-H--
Message: Access denied with code 400 (phase 2). Operator EQ matched 0
at REQUEST_HEADERS. [file
"/etc/httpd/modsecurity.d/modsecurity_crs_21_protocol_anomalies.conf"]
[line "48"] [id "960009"] [msg "Request Missing a User Agent Header"]
[severity "WARNING"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"]
Action: Intercepted (phase 2)
Stopwatch: 1273250392877004 127315 (347 525 -)
Producer: ModSecurity for Apache/2.5.6 (http://www.modsecurity.org/);
core ruleset/1.6.1; core ruleset/1.6.1.
Server: Apache

--986cbd32-K--
SecRule "REQUEST_METHOD" "@rx ^(?:GET|HEAD)$"
"phase:2,status:400,t:lowercase,t:replaceNulls,t:compressWhitespace,chain,t:none,deny,log,auditlog,msg:'GET
or HEAD requests with
bodies',severity:2,id:960011,tag:PROTOCOL_VIOLATION/EVASION"
SecRule "&REQUEST_HEADERS:User-Agent" "@eq 0"
"phase:2,status:400,t:lowercase,t:replaceNulls,t:compressWhitespace,t:none,deny,log,auditlog,msg:'Request
Missing a User Agent
Header',id:960009,tag:PROTOCOL_VIOLATION/MISSING_HEADER,severity:4"
SecRule "RESPONSE_STATUS" "@rx ^400$"
"phase:5,status:403,t:lowercase,t:replaceNulls,t:compressWhitespace,t:none,chain,log,auditlog,pass,msg:'Invalid
request',id:960913,severity:2"
SecRule "RESPONSE_STATUS" "@rx ^400$"
"phase:5,status:403,t:lowercase,t:replaceNulls,t:compressWhitespace,t:none,chain,log,auditlog,pass,msg:'Invalid
request',id:960913,severity:2"

--986cbd32-Z--

Apache Error Log Entry:

[Fri May 07 16:39:52 2010] [error] [client xxx.xxx.xxx.xxx]
ModSecurity: Access denied with code 400 (phase 2). Operator EQ
matched 0 at REQUEST_HEADERS. [file
"/etc/httpd/modsecurity.d/modsecurity_crs_21_protocol_anomalies.conf"]
[line "48"] [id "960009"] [msg "Request Missing a User Agent Header"]
[severity "WARNING"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"]
[hostname "my.domain.com"] [uri "/this/page"] [unique_id
"tWCnzAoABQMAABVYW3wAAAAu"]

Apache Access Log Entry:

my.domain.com xxx.xxx.xxx.xxx - - [07/May/2010:16:39:52 +0000] "GET
/this/page HTTP/1.1" 400 3887 "-" "-"

Art Age Software

2010-05-07 19:36:36 UTC

Permalink

Unfortunately, I do not have access to the client side as these are
remote clients - and I have been unable to reproduce the problem
locally. Thanks for the suggestion, though.

Just a thought - how about running the traffic through some sniffer like
wireshark or perhaps Fiddler/burp etc on client side to see if web server
really sent out those code 400 response. It could be possible that Apache is
sending out these 400 responses too in addition to correct (or expected)
response and the web browser is not honoring the 400 responses.
Above test will also give you insight into whether header was actually
missing or not.
Good luck,
Gaurav Kumar

Post by Art Age Software
I've been using mod-security successfully for a couple years now as an
additional layer of protection for a web-based app. Recently, a new
behavior has emerged that I do not understand. I am using core
ruleset/1.6.1 and have made no changes to this configuration for
several months, so I do not know why this behavior has suddenly
emerged.
What is happening, according to the logs, is that certain requests are
being rejected with "Access denied with code 400 (Request Missing a
User Agent Header)." However, these log entries are immediately
preceded by successful GETs from the same IP address. I have followed
up with the users behind these IP addresses and all report **no**
strange behavior and **none** is seeing a "400 Bad Request" error page
or any similar error page. In other words, they all report that the
application is performing perfectly normally.
Mod-security is definitely "On" and the apache web logs also show that
these requests are being rejected with status code 400. I have
included an example (partially scrubbed) of the log entries for one
such instance below. If anybody has any idea for what could possibly
be going on here, I would appreciate hearing it. Thanks.
------
--986cbd32-A--
[07/May/2010:16:39:53 +0000] tWCnzAoABQMAABVYW3wAAAAu xxx.xxx.xxx.xxx
50018 yyy.yyy.yyy.yyy 80
--986cbd32-B--
GET /this/page HTTP/1.1
Host: my.domain.com
Accept: */*
--986cbd32-F--
HTTP/1.1 400 Bad Request
Set-Cookie: SESSID=vlu8kib6b2ovekkbad8mnc43c3; path=/; domain=.domain.com
Expires: Fri, 31 Dec 1999 23:59:59 GMT
Cache-Control: max-age=0, must-revalidate, no-cache, post-check=0, pre-check=0
Pragma: no-cache
Vary: User-Agent
Content-Length: 3887
Connection: close
Content-Type: text/html; charset=UTF-8
--986cbd32-H--
Message: Access denied with code 400 (phase 2). Operator EQ matched 0
at REQUEST_HEADERS. [file
"/etc/httpd/modsecurity.d/modsecurity_crs_21_protocol_anomalies.conf"]
[line "48"] [id "960009"] [msg "Request Missing a User Agent Header"]
[severity "WARNING"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"]
Action: Intercepted (phase 2)
Stopwatch: 1273250392877004 127315 (347 525 -)
Producer: ModSecurity for Apache/2.5.6 (http://www.modsecurity.org/);
core ruleset/1.6.1; core ruleset/1.6.1.
Server: Apache
--986cbd32-K--
"phase:2,status:400,t:lowercase,t:replaceNulls,t:compressWhitespace,chain,t:none,deny,log,auditlog,msg:'GET
or HEAD requests with
bodies',severity:2,id:960011,tag:PROTOCOL_VIOLATION/EVASION"
"phase:2,status:400,t:lowercase,t:replaceNulls,t:compressWhitespace,t:none,deny,log,auditlog,msg:'Request
Missing a User Agent
Header',id:960009,tag:PROTOCOL_VIOLATION/MISSING_HEADER,severity:4"
"phase:5,status:403,t:lowercase,t:replaceNulls,t:compressWhitespace,t:none,chain,log,auditlog,pass,msg:'Invalid
request',id:960913,severity:2"
"phase:5,status:403,t:lowercase,t:replaceNulls,t:compressWhitespace,t:none,chain,log,auditlog,pass,msg:'Invalid
request',id:960913,severity:2"
--986cbd32-Z--
[Fri May 07 16:39:52 2010] [error] [client xxx.xxx.xxx.xxx]
ModSecurity: Access denied with code 400 (phase 2). Operator EQ
matched 0 at REQUEST_HEADERS. [file
"/etc/httpd/modsecurity.d/modsecurity_crs_21_protocol_anomalies.conf"]
[line "48"] [id "960009"] [msg "Request Missing a User Agent Header"]
[severity "WARNING"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"]
[hostname "my.domain.com"] [uri "/this/page"] [unique_id
"tWCnzAoABQMAABVYW3wAAAAu"]
my.domain.com xxx.xxx.xxx.xxx - - [07/May/2010:16:39:52 +0000] "GET
/this/page HTTP/1.1" 400 3887 "-" "-"
------------------------------------------------------------------------------
_______________________________________________
mod-security-users mailing list
https://lists.sourceforge.net/lists/listinfo/mod-security-users
http://www.modsecurity.org/breach/index.html

Gaurav Kumar

2010-05-07 19:21:43 UTC

Permalink

Just a thought - how about running the traffic through some sniffer like
wireshark or perhaps Fiddler/burp etc on client side to see if web server
really sent out those code 400 response. It could be possible that Apache is
sending out these 400 responses too in addition to correct (or expected)
response and the web browser is not honoring the 400 responses.

Above test will also give you insight into whether header was actually
missing or not.

Good luck,
Gaurav Kumar

Christian Bockermann

2010-05-07 19:54:16 UTC

Permalink

Hi Art,

looking at the audit-event you attached, this looks as expected behaviour.
The request does not contain a request-header for the User-Agent and is
rejected with code 400 by the rule with ID 960009.

Judging on the fact that only the "Host"- and the "Accept"-Headers are sent
by the affected client, then this might be a hint for an anomalous client.
This is the intention of that rule 960009.
(Usually, each browser provides the "User-Agent" header).

So the question is - is the IP which sent this event related to a "normal"
user? Is the web-application available for public use? Are the 400-events
coming from a user, who confirmed the app is running fine?

Another interesting question would be - do the subsequent requests also
have the same port? Usually, a browser will use a persistent connection.
In the worst case, this could be a hint to some client running evil code
which tries to do some probing. Or - to not count with the worst - there
is some buggy java-script sending plain http requests on behalf of the
browser.

I'd put modsec into audit-log-everything for a short while and check the
complete events for an IP-address which raised this issue. You can do this
with the AuditViewer, which allows you to filter all events by a session-ID
or the client-address.

Regards,
Chris

Brian Rectanus

2010-05-07 21:17:23 UTC

Permalink

Agreed with Chris here. These are suspicious. Additionally, is the 400 requst a legitimate page to be requesting?

I'd guess malware on the client or bad javascript. You should be able to see everything delivered to the client with full auditlogging as Chris suggests. For clients that are getting 400 take a look at what the browser served and verify it contains something that would be issuing the bad request.

If there is nothing like this sent to the client just before a bad request the probably malware.

-B

--
Brian Rectanus
Breach Security

-----Original Message-----
From: Christian Bockermann [***@jwall.org]
Received: 5/7/10 12:57 PM
To: Art Age Software [***@gmail.com]
CC: mod-security-***@lists.sourceforge.net [mod-security-***@lists.sourceforge.net]
Subject: Re: [mod-security-users] Perplexing Issue: "400 Bad Request"

Hi Art,

looking at the audit-event you attached, this looks as expected behaviour.
The request does not contain a request-header for the User-Agent and is
rejected with code 400 by the rule with ID 960009.

Judging on the fact that only the "Host"- and the "Accept"-Headers are sent
by the affected client, then this might be a hint for an anomalous client.
This is the intention of that rule 960009.
(Usually, each browser provides the "User-Agent" header).

So the question is - is the IP which sent this event related to a "normal"
user? Is the web-application available for public use? Are the 400-events
coming from a user, who confirmed the app is running fine?

Another interesting question would be - do the subsequent requests also
have the same port? Usually, a browser will use a persistent connection.
In the worst case, this could be a hint to some client running evil code
which tries to do some probing. Or - to not count with the worst - there
is some buggy java-script sending plain http requests on behalf of the
browser.

I'd put modsec into audit-log-everything for a short while and check the
complete events for an IP-address which raised this issue. You can do this
with the AuditViewer, which allows you to filter all events by a session-ID
or the client-address.

Regards,
Chris

Art Age Software

2010-05-07 21:57:37 UTC

Permalink

Post by Brian Rectanus
Agreed with Chris here. These are suspicious. Additionally, is the 400
requst a legitimate page to be requesting?

Yes, that is what is surprising. The page requests are legitimate and
could not be known by a bot or easily guessed, because the URLs are
only accessible once logged into the application and there are no
public external links to these internal application pages.

Post by Brian Rectanus
I'd guess malware on the client or bad javascript. You should be able to see
everything delivered to the client with full auditlogging as Chris suggests.
For clients that are getting 400 take a look at what the browser served and
verify it contains something that would be issuing the bad request.

I have no access to the browsers because they are remote users. I can
only assume that for some reason, these requests are coming in with no
user-agent string - but I don't know why...

Christian Bockermann

2010-05-08 06:20:03 UTC

Permalink

Post by Art Age Software

Post by Brian Rectanus
Agreed with Chris here. These are suspicious. Additionally, is the 400
requst a legitimate page to be requesting?

Interesting incident, I'll notify Dr. Watson :-)

Q1: How do you know that these are legitimate requests? From what we saw until
now, we cannot assume this.

Q2: What is your authentication scheme? Is it a app-internal session-based
login? Is it using HTTP-Authentication?

Since there is no cookie in the request, I'd assume the app relies on some
parameter-based auth-token, which you pruned before sending the event to the
list (which is perfectly reasonable).

With todays CSRF attacks, we cannot simply say it's a legitimate request since
it came from a legitimate client-IP. (Unless you have not *secure* IP-to-client
verification, e.g. by VPN or the like, this hasn't been true for a long time :-))

Post by Art Age Software

I have no access to the browsers because they are remote users. I can
only assume that for some reason, these requests are coming in with no
user-agent string - but I don't know why...

Right, but you have all the power of modsecurity available - this just cries
for using its audit-log capabilities :-)

Some things I'd look for:

(1) Start by looking at the plain access-log files. The AuditViewer can read
these as well and might be helpful to extract the sequences of requests
for each separate IP.
Otherwise you can simply write your own perl-or-whatever skript.

Q3: Are these 400-requests showing up in EVERY clients access-sequence?

If the answer to Q1 is 'No', then we distinguish two groups of clients
in the following: Client400 and ClientClean.

Q4: Are there requests to some JS-pages (i.e. REQUEST_URI "@rx .*\.js$") in
the Client400 sequences which are not requested in ClientClean sessions?
In the worst case, you might want to have a deeper look at the JS code :-)

If there are no JS-pages being accessed, then there might still be JS-code
in the HTML pages -> Use some regex for matching JS-snippets to check your
server's response and log the full server-response for matching responses.

So the bottom line: just assuming it is bad JavaScript, then this needs to be loaded
somewhere. Assuming further, that the app is self-contained, then any JS will be loaded
from your server => ModSecurity can help you see every bit of that by logging the HTTP
response body as well.

Just my 2cents for this. Rather exciting case :-)
Keep us informed (if you can provide more detailed events, you could sent them privately
for having a closer look).

Best regards,

Chris

Art Age Software

2010-05-08 17:12:23 UTC

Permalink

Q1: How do you know that these are legitimate requests?

Christian Bockermann

2010-05-09 06:49:05 UTC

Permalink

Hi Sam!

Post by Christian Bockermann
Q1: How do you know that these are legitimate requests? From what we saw until
now, we cannot assume this.

I do **not** know the requests are legitimate. However, the requests
are for valid URLs that are not externally linked. How would the
requester know what URLs to visit?

Guessing, Browser-History, Probing, etc. Is it a "standard" or a proprietary
application?

Post by Christian Bockermann
Since there is no cookie in the request, I'd assume the app relies on some
parameter-based auth-token, which you pruned before sending the event to the
list (which is perfectly reasonable).

--986cbd32-F--
HTTP/1.1 400 Bad Request
Set-Cookie: SESSID=vlu8kib6b2ovekkbad8mnc43c3; path=/; domain=.domain.com

No that is not a client cookie.
This is a new cookie, sent by the server, since the client-request did not
provide a cookie. That is a tremendous difference. Thus, the request does
*not* contain a cookie and is therefore not associated with the session.

Post by Christian Bockermann
Q3: Are these 400-requests showing up in EVERY clients access-sequence?

No, only a small handful of clients (out of thousands).

Post by Christian Bockermann
If the answer to Q1 is 'No', then we distinguish two groups of clients
in the following: Client400 and ClientClean.
the Client400 sequences which are not requested in ClientClean sessions?
In the worst case, you might want to have a deeper look at the JS code :-)

Not that I have seen, but I will check deeper. This is not a
javascript-heavy application.

That is probably the INTENDED way of your application. If there are any text-fields
where users can store text, someone might be able to store malicious JS-code as well.
This all-of-a-sudden makes your App more JS-heavy than you probably want ;-)

Just as a note: I am not saying that your application has been compromised or anything
like that. The weird requests MIGHT be an indication for JS stuff or simply just
result from a stupid browser-plugin or similar.

The hint that there are only a small handful clients causing these problems pops
up some more questions:

Q5: Is there any specific property which is common to these handful of clients
but not for the other? E.g. OS/Browser-Version? Are they coming from the
same subnet?

Q6: Does this small group ALWAYS sent this strange requests?

Regards,
Chris

Brian Rectanus

2010-05-10 16:08:00 UTC

Permalink

Sam,

Post by Christian Bockermann
Hi Sam!

Post by Christian Bockermann
Q1: How do you know that these are legitimate requests?

Art Age Software

2010-05-11 16:23:18 UTC

Permalink

There **is** a cookie.

Brian Rectanus

2010-05-11 17:18:00 UTC

Permalink

There **is** a cookie.

Art Age Software

2010-05-11 18:03:35 UTC

Permalink

And by client you mean "IP Address"? What is your definition of "client"?

Yes, IP address.

Is it possible that all requests are served via the same gateway? I'd
look into that as the gateway may be compromised (spyware) or broken.
Since you are not using SSL, all your sessions are available to any
proxies and thus accounts are hackable by anyone with access to the traffic.

Yes, this is a known tradeoff we have made due to the performance
impact of SSL on every request. Logins are always done via SSL, but
the session cookie is sent in the clear (although it contains not
sensitive data, it is subject to hijack).

Are the 400s always GET? *Should* these requests be GET normally? This
may indicate CSRF style attacks as GET is the typical for of request for
CSRF.

GETs and POSTs

OK, I have made a very interesting determination. There are actually
several application servers clustered to serve requests. What I have
found, is that for every request that results in a 400 error on one
server, there is an identical successful request filled on one of the
other cluster members **at the same timestamp.**

It seems as though either the load balancer is erroneously sending the
request twice, to two different servers, or the client's browser is
sending duplicate requests. For some reason, the duplicate requests do
not include all the header info, so mod-security traps and refuses
them, seemingly saving my bacon in a quite unintended way. I have
**never** seen this behavior before it started cropping up a few weeks
ago, and there have been zero software or configuration updates that
could explain why this behavior has emerged. Furthermore, I do not
understand why it only occurs with a small subset of clients. So, the
mystery remains, but at least there is a bit more info to go on...

Continue reading on narkive:

Search results for '[mod-security-users] Perplexing Issue: "400 Bad Request"' (Questions and Answers)

replies

Who anyone feel bad about not tipping in this situation??

started 2008-05-16 07:50:38 UTC

etiquette

replies

Looking for personal references: I have a custody issue...?

started 2007-07-27 12:52:47 UTC

marriage & divorce

replies

Tell me why you think King David was a good person or bad person?