Warm up + Refresh WordPress Varnish 4 Cache + Cloudflare

Varnish Cache is the best speed boost to give your WordPress site. Varnish is a reverse proxy that caches your WordPress site as HTML and serve them up quickly, bypassing the slow PHP backend (Apache or nginx). With this WordPress Varnish cache method you will always be serving up fast Varnish cached versions of your page.

Varnish has a clever feature: using hash_always_miss lets Varnish continue to serve its cache version while it refreshes the cache from the PHP backend. This means you can slowly refill the Varnish cache instead of purging it all at once and being stuck with slow pages as new users request the uncached versions. I will assume you have access to your server (like with Digital Ocean) to install and edit files and have XML sitemaps enabled. Yoast SEO creates sitemaps for you and is what I use for this WordPress Varnish 4 Cache tutorial. To configure WordPress Varnish 4 cache see this tutorial.

I have a WordPress plugin which automates this, use the contact form to help test it and get a free copy

Warm up your WordPress Varnish 4 Cache Behind CloudFlare

I will assume you already have Varnish cache installed and configured to work with WordPress.

Check your Varnish version

varnishd -V

If it shows 4.x then this method will work for you behind CloudFlare, if you are on Varnish 3 then see my other tutorial.

Configure Varnish for Smart Refreshing and Real IP from CloudFlare

Open your Varnish vcl file, usually default.vcl

sudo nano /etc/varnish/default.vcl

Add import std; at the top under vcl 4 to activate the std module which converts strings to IP addresses so you can match the real IP to your editors access control list you will create below.

vcl 4.0;
import std;

Add an access control list (acl) section for editors, this is for security so that other machines cannot force refreshes on your page. Add it after the backend section but before the sub vcl_recv section. Change IP.of.Server to the source IP address that will be sending the refresh commands, I am just using the web server to run the script so it is the same IP as what CloudFlare points to for this domain.

acl editors {
  "127.0.0.1";
  "IP.of.Server";
}

Adjust your VCL file to include always miss in your sub vcl_recv section, you only need to add the red section. It looks for the header Cache-Control with no-cache (which we will send later using curl) and whether the sender is a member of the editors acl. It uses the std.ip function in Varnish to get the real IP from CloudFlare and defaults to client.ip if it cannot detect the real IP. If you are using the DDoS protection guide, put this code before it so you ensure you get the right IP for the curl request to match in the editors acl.

sub vcl_recv {

# set realIP by trimming CloudFlare IP which will be used for various checks

set req.http.X-Actual-IP = regsub(req.http.X-Forwarded-For, "[, ].*$", "");

# check if the real IP is coming from your web server

if (std.ip(req.http.X-Actual-IP, "1.2.3.4") ~ editors) {

# if the header is set to no-cache set always miss

if (req.http.Cache-Control ~ "no-cache" ) {
set req.hash_always_miss = true;
      }
   }
}

I have also set my Varnish time to live – how long Varnish should keep cached version of the page – to 1 year in the sub vcl_fetch function

set beresp.ttl = 52w;

Test your adjusted Varnish default.vcl works

varnishd -C -f /etc/varnish/default.vcl

If you didn't get any errors reload your Varnish configuration

sudo service varnish reload

Manual Varnish Smart Refresh for Single URL

Create a new script that will send the always miss request for a URL you paste into the script when prompted

nano smartvarnishrefreshsingle.sh

Paste this code

#!/usr/bin/env bash

# WordPress Varnish Cache Refresh from HTPCGuides.com

echo Enter full URL to purge
read url

curl -s $url -H "Cache-Control: no-cache" -o /dev/null
echo Refreshed $url

Ctrl+X, Y and Enter to save the script

Make the script executable

sudo chmod +x smartvarnishrefreshsingle.sh

Run the manual single Varnish purge script like this

bash smartvarnishrefreshsingle.sh

You will see this output, just paste your URL and press Enter

Enter full URL to purge

Now that page has been intelligently refreshed using Varnish

Refresh Entire WordPress Varnish Cache Automated Method

The automated slow Varnish cache refresher script requires xml2 and curl so install them

sudo apt-get install xml2 curl -y

Create the varnishsmartrefresh script

nano varnishsmartrefresh.sh

Paste this, adjust site to your site's name

#!/usr/bin/env bash
site=https://www.htpcguides.com

#Download post sitemap
wget -q $site/post-sitemap.xml -O postsitemap.xml

#Parse the xml file and put it into posts.txt
xml2 < postsitemap.xml | grep /url/loc= | sed 's/.*=//' > posts.txt

# Loop through the posts.txt and use curl to send an always miss request

while read post; do
  curl -s $post -H "Cache-Control: no-cache" -o /dev/null
  echo Refreshed $post
  echo Waiting
  sleep 10
done < posts.txt

#Download page sitemap
wget -q $site/page-sitemap.xml -O pagesitemap.xml

#Parse the xml file and put it into pages.txt
xml2 < pagesitemap.xml | grep /url/loc= | sed 's/.*=//' > pages.txt

# Loop through the pages.txt and use curl to send an always miss request

while read page; do
  curl -s $page -H "Cache-Control: no-cache" -o /dev/null
  echo Refreshed $page
  echo Waiting
  sleep 10
done < pages.txt

#Download category sitemap
wget -q $site/category-sitemap.xml -O categorysitemap.xml

#Parse the xml file and put it into categories.txt
xml2 < categorysitemap.xml | grep /url/loc= | sed 's/.*=//' > categories.txt

# Loop through the categories.txt and use curl to send an always miss request

while read category; do
  curl -s $category -H "Cache-Control: no-cache" -o /dev/null
  echo Refreshed $category
  echo Waiting
  sleep 30
done < categories.txt

# Warm up and refresh blogroll pages, change 30 to the number of pages back you show posts
for i in {1..30}
 do
 echo Refreshing $site/page/$i
 curl -s $site/page/$i/ -H "Cache-Control: no-cache" -o /dev/null
 echo Refreshed $site/page/$i
 echo Waiting
 sleep 1
done

Ctrl+X, Y and Enter to save the script

Make the script executable

sudo chmod +x smartvarnishrefresh.sh

Run the Varnish cache refresh script to test it

bash varnishmartrefresh.sh

You will see a lot of curl commands and Refreshed URL names, this will take a while depending on how many posts, pages and categories you have.

I have also added a daily cronjob for refreshing the WordPress Varnish Cache

crontab -l | { cat; echo "@daily /path/to/varnishslowrefresh.sh"; } | crontab -

Manual WordPress Varnish Refresh Method

Go to your sitemaps, you can use as many or as few as you want

• Posts – http://www.yourwebsite.com/post-sitemap.xml
• Pages – http://www.yourwebsite.com/page-sitemap.xml
• Categories – http://www.yourwebsite.com/post-category.xml

Highlight and copy the entire table to the clipboard

Go into Excel or any spreadsheet program and paste it

Do a search or find and replace, make find your full URL http://www.yourwebsite.com and leave the replace blank

Highlight the whole column and copy it to the clipboard, you will paste it in the manual script below

sudo nano smartvarnishrefresh.sh

Change your site name and paste your WordPress URLs from the spreadsheet to refresh in the Varnish cache (adapted from here)

#!/usr/bin/env bash
site="https://www.htpcguides.com"
pages="
/
# Paste your URL sitemap here
/100-amazon-gift-card-giveaway-july-2015/
/40-amazon-gift-card-giveaway/
"

echo -----------------------------
echo Refresh old pages from cache
echo -----------------------------
for page in $pages; do 
	echo Setting always miss for $site$page
	curl -s $site$page -H "Cache-Control: no-cache" -o /dev/null
	sleep 10 
done

Ctrl+X, Y and Enter to save the script

Make the script executable

sudo chmod +x smartvarnishrefresh.sh

Run the manual Varnish cache script

bash slowvarnishpurge.sh

These WordPress Varnish Cache scripts should ensure you are always serving content quickly to your site's visitors even if you are behind CloudFlare.