Cache headers will only be respected when using "Site Caching" or "Disabled" as your Cache Level.
In order to analyze the cache headers, a tool called curl will be used. You can also use the Developer Tools of your browser.
If you are using Microsoft® Windows, install Git for Windows before proceeding.
Open the Terminal/Git for Windows and run the following command:
curl -IL http://yourdomain.com/Change "yourdomain.com" for your domain or the complete URL of the page/file. For example, if we want to check Sucuri Blog’s front page http headers:
$ curl -IL https://blog.sucuri.net/
HTTP/1.1 200 OK
Server: Sucuri/Website Firewall
Date: Thu, 19 Jan 2017 15:18:31 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: PHPSESSID=aha8r2fs3m0njv8h3be14hbtt2; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Cookie
Link: <https:></https:>; rel=https://api.w.org/
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-Sucuri-ID: 11005Here are the lines we need to pay attention to for the purpose of this article:
Vary: Accept-Encoding
Set-Cookie: PHPSESSID=aha8r2fs3m0njv8h3be14hbtt2; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: CookieSucuri’s blog uses "Enabled" cache level, so these http headers are completely ignored. However, if it were using "Site Caching" cache level, those headers would be honored and as a result Sucuri’s blog would never be cached by the Firewall.
Do you know why? Let’s understand a bit more about how cache headers works.
– Vary
It’s a powerful header that is frequently used incorrectly. Here’s a list of the possible values:
- Vary: Accept-Encoding
When the origin server (your server) doesn’t send this header, two things can happen:
1) If the content isn’t compressed, you will use more bandwidth, but all generations of browsers will be able to render the page.
2) If the content is compressed, but there isn’t a Vary: Accept-Encoding header, older browsers won’t be able to render the content.
To avoid this kind of issue, the origin server must send the Vary: Accept-Encoding when the content is compressed. That way the Sucuri Firewall will keep two separate versions of your content: one without compression and one with compression. Depending on the browser, the Sucuri Firewall will serve the right version, saving bandwidth, speeding up your website and keeping compatibility with older browsers.
- Vary: Cookie
This header usually prevents authenticated users from seeing cached versions of pages with Set-Cookie as guide. However, due to security reasons, we won’t cache any page that has "Vary: Cookie" and "Set-Cookie" as the http header. It’ll basically have the same effect as "Cache-Control: private, no-cache" on the Sucuri Firewall.
- Vary: User-Agent
If your website has a mobile version, this header can help you with your SEO. It’ll make the cache servers create different versions of the pages depending on the User Agent.
Attention: Mobile version is different from Responsive design. Responsive design is interpreted differently by the browser, but the page content is exactly the same for all user agents. You shouldn’t use this header for responsive designed websites.
- Vary: Referrer
This header will instruct the browser to re-check with the server for each different referrer. It isn’t recommended, unless you know exactly what you are doing.
- Vary: *
Don’t use it. It’ll prevent any sort of caching and greatly increase the server load.
– Set-Cookie
Cookies are usually used to identify a user session. Therefore, if Set-Cookie is present, the Firewall will not cache the page. In case your page is being cached even with Set-Cookie present, please also add "Vary: Cookie" to make sure it’ll not be cached.
– Cache-Control
It is the most common and important cache header. It has 2 possible values: public and private.
public
Mainly used for static content and public pages (guests only), public always comes with max-age=X (X is seconds) to control the cache expiration. When you specify the max-age=X, it isn’t necessary to include public, as it is implicit.
It’s especially useful for event-driven content websites, like Wikis, news portal, highly updated blogs, etc. You can use the Sucuri Firewall cache as a micro-caching layer as explained in step 8 of the Troubleshoot Caching Issues article.
For websites that don’t update their public content frequently, it is possible to use a longer max-age.
To have even better control, you can also use "s-maxage". The Shared Max Age is a specific max-age only for reverse caching or other public proxy servers and is generally set lower than Max Age.
If your server doesn’t provide a Cache-Control header specifically for static content, like images, js, css, swf, mp3, mp4, pdf or fonts, our system will automatically set Cache-Control:max-age=315360000 for those files.
Side note: the Firewall will check and revalidate (if necessary) those files every 3 days. However, static files will always be cached, regardless of the cache level or the non-cached URLs. Check step 4 of the Troubleshoot Caching Issues article in order to know how to avoid static content cache.
private
It tells the cache server not to cache the content at all. You must combine this with no-store, no-cache, must-revalidate, post-check=0, pre-check=0 to also avoid browser caching. It is commonly used for authenticated users sections and to prevent dynamic content caching.
– Pragma
It’s basically an "obsolete" cache-control, used for HTTP/1.0 requests. It has two variables of self-explained meaning: cache and no-cache. It’s still in use because older browsers may not support the newest protocols, like HTTP/1.1.
– Expires
After the date specified by the Expires header, the browser has to request the newest copy of the content. It won’t be considered valid after this date, but until then, the local copy should be used. It’s expected that Expires matches the max-age value of Cache-Control, but it’s not necessary.
Just like Cache-Control, if your server doesn’t provide an Expires header, our system will automatically set the cache lifetime of the page to the Caching Level default value.
Final Notes
Caching is one of the most important techniques available to speed up your website, save resources and improve your website’s resilience against DDoS attacks.
If you don’t want to cache a page, you can either use "Vary: *", "Cache-Control: private", "Pragma: no-cache" or "Expires: Thu, 19 Nov 1981 08:52:00 GMT", such as in the example before:
Vary: Accept-Encoding
Set-Cookie: PHPSESSID=aha8r2fs3m0njv8h3be14hbtt2; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: CookieJust one of them would make the Firewall not cache the page if you were using Site Caching or Disabled mode.
On the other hand, if you do want to cache a page, make sure none of the headers mentioned here are set to stop caching. You could use just "Cache-Control" for example:
Cache-Control: max-age=900This header alone, without the others to interfere, will instruct the Firewall to cache your page for 900 seconds.
Keep in mind: If the page is not accessed in hours, even if you set a really long cache lifetime, it’ll expire from the caching. The Sucuri Firewall caching happens on demand and retains only the frequently accessed assets/pages.
ETag and Last-Modified headers weren’t mentioned because they don’t affect the Sucuri Firewall cache behavior.