CPDoS: Cache Poisoned Denial of Service

Posted on December 12, 2019

Web cache poisoning and deception attacks are two related and well-know type of web attacks against caches. These attacks usually target CDNs as massive web caching services. We implemented a functional test reproducing web cache deception attack about one year ago to make sure that Tempesta FW doesn't have this flaw. However, we made only manual tests to check that our web cache is robust against web cache poisoning attacks.

The recent researches CPDoS: Cache Poisoned Denial of Service and Responsible denial of service with web cache poisoning introducing a new class of web cache poisoning attack made us back to the topic the year later. CPDoS has three variants, let's have a close look onto them.

HTTP Header Oversize (HHO)

The name speaks for itself and you can read details of the attack on the original resource. In short, the attack exploits HTTP headers which exceeds web server or proxy limits, so they process an HTTP request differently. This is a misconfiguration problem and you can and should prevent it using a proper limits on your web cache proxy. For Tempesta FW you can do this with Frang HTTP limiting module:


        # Global limit for HTTP header length.
        frang_limits {
            http_field_len 4096;
            ...
        }

        # HTTP header length limit specific for a particular virtual host.
        vhost example.com {
            frang_limits {
                http_field_len 8192;
                ...
            }
            ...
        }
    

HTTP Meta Character (HMC)

This is a bit trickier attack, which requires several steps to prevent. An attacker sends a request for a cacheable resource and the request contains a header (usually non standard, which isn't verified by a proxy, but is processed by the victim application) with a special character, e.g. \a.


        GET /some.app HTTP/1.1
        Host: example.net
        X-My-App-Header: foo \a bar
    

Since the value of the header X-My-App-Header can't be normally processed, the application returns an error response with code 400. If a web proxy caches such responses, then all legitimate users requesting the same resource will get the error response.

If your application expects a normal string in the header, then it's quite unlikely that it normally process the special character. To illustrate the problem more let's have a look how Nginx, one of the most popular HTTP servers, processes HTTP headers. The HTTP parser function ngx_http_parse_header_line() and the state sw_value (see the original code at GitHub) are responsible for the parsing:


        case sw_value:
            switch (ch) {
            case ' ':
                r->header_end = p;
                state = sw_space_after_value;
                break;
            case CR:
                r->header_end = p;
                state = sw_almost_done;
                break;
            case LF:
                r->header_end = p;
                goto done;
            case '\0':
                return NGX_HTTP_PARSE_INVALID_HEADER;
            }
            break;
    

I.e. all characters, except \0, \r, \n, and space are allowed.

Tempesta FW doesn't cache 400 error responses, so basically you're safe against this type the CPDoS attack without any additional steps. However, if an attacker just sends bunch of such requests, then they all pass to the victim application and typically execute probably expensive logic, i.e. this cache behavior opens a DDoS attack vector. To protect against the DDoS attack, use the limit for HTTP error responses.

I didn't mention above, but Tempesta FW is quite strict about HTTP headers processing and it doesn't pass special characters like \a. In general, it strictly follows the RFC (HTTP/2 refers HTTP/1.1 in headers processing, so it obeys the same rules) and passes only characters allowed by the RFC. However, \a is just an example and your application may not behave correctly with RFC allowed characters, e.g. $ used for remote code execution (RCE) attacks. Tempesta FW allows you to specify explicitly which characters are allowed in different HTTP fields by your web application and this is recommended practice to use the configuration options to protect against various web attacks including injections. Note that Tempesta FW uses AVX2 for the HTTP fields validation, so it eats 32 bytes at once which is much faster than the more permissive singe-character-at-a-time parsing as in the code example above. A configuration example for the two limiting settings described above would look like:


        # Block a client after generating 3 error responses with code 400
        # within 10 seconds.
        http_resp_code_block 400 3 10;

        # Allow only [a-zA-Z0-9\s] for all HTTP headers obeying token
        # grammar by RFC 7230.
        http_token_brange 0x61-0x7a 0x41-0x5a 0x20;
    

HTTP Method Override Attack (HMO)

According to HTTP standard, a client sets request method in the first line of a request. But some Web Application Firewalls (WAF), DDoS mitigation providers, and other intermediaries block requests with relatively small white list of methods (usually only GET and POST are allowed), which may break a protected web application if it need other methods.

It's a bad practice, but if you definitely want to do this, you can define a white list for HTTP methods with Tempesta FW configuration option http_methods, for example


        http_methods GET POST;
    

To pass a not allowed method over a WAF some backend servers allow to override request methods. In this case a request will contain a whitelisted GET or POST method, but URI or a special header can define absolutely different request method.

For example, IBM Operational Decision Manager requires DELETE and PUT methods for its operation and if the methods aren't allowed by a firewall, the documentation suggests to use X-Method-Override or X-HTTP-Method-Override HTTP headers or x-method-override or x-http-method-override URI parameters to bypass a firewall.

Yes, it may look weird to deploy a security solution and then bypass it in your own application, but this is how the things work: one purchases a CDN service and happily uses it for some time, but at some point they deploys a new application functionality requiring additional HTTP methods and now realizes that the CDN provider is unable to service these types of requests.

Using the flaw an attacker may override for example GET method, but make a proxy to cache an appropriate response as GET so all other clients will receive incorrect cache version. The original article provides verbose description of the attack scenario and we won't cover them here. Instead, let's focus on the attack prevention.

The worst thing about the 'self-bypassing' is that basically you can use any custom header in your application to bypass the WAF restriction. This type of the attack is also mostly about the intermediary misconfiguration and violation of the HTTP RFCs, so the intermediary is unable to provide adequate user experience. Thus, firstly we didn't want to take any specific steps to handle the attack anyhow. However, after some thinking we decided that the problem can be solved in relatively cheap way, so our developer Ivan Koveshnikov implemented an HTTP parser extension to protect Tempesta FW's cache against the HMO attack.

Let's start with the attack in URI and consider two subsequent HTTP requests: a malicious HMO request and following normal user request to the same resource:


        GET /some.app?var=Foo&x-method-override=PUT HTTP/1.1
        Host: example.net
    


        GET /some.app?var=Foo HTTP/1.1
        Host: example.net
    

Both the requests have different URIs, so a web cache stores them in different places. This means that HMO doesn't work with URI version of HTTP method overwrite. Thus, we only need to protect against the version of the attack for HTTP headers, e.g.


        GET /some.app?var=Foo HTTP/1.1
        Host: example.net
        X-HTTP-Method-Override: PUT
    

The fix becomes obvious: just parse X-Method-Override and X-HTTP-Method-Override HTTP headers and set a request method to actual value defined by any of the headers. Now the whole Tempesta FW logic considers the request as PUT and treats it appropriately.

It's worth mentioning that with Tempesta FW you can also block any HTTP request containing a particular header, e.g. to block all requests trying to overwrite HTTP method you can use HTTP tables in following way:


    vhost app {
        ... # your application configuration
    }

    http_chain {
        hdr "X-HTTP-Method-Override" == "*" -> block;
        hdr "X-Method-Override" == "*" -> block;

        # Pass through all other HTTP requests
        -> app;
    }
    

Conclusion

Two of the three types of CPDoS attacks are about proxies and backend application misconfiguration and HTTP RFCs violations. Generally speaking, intermediaries, such as CDN proxies or WAFs, must obey the RFCs to provide robust service to end users and clients. However, in real life it's hard to avoid all types of misconfigurations and sometimes it does make sense to restrict the HTTP standard to guarantee stronger security. With all the restrictions it's still possible to mitigate or even prevent the CPDoS attacks.

If you like Tempesta FW, you can play with our alpha.

Share on