您當前位置：首頁 > php框架 > 框架設計 > Varnish4.1.2配合FastDFS構建內存緩存

Varnish4.1.2配合FastDFS構建內存緩存

來源：程序員人生發布時間：2016-08-17 09:55:29 閱讀次數：3593次

Varnish是甚么

Varnish是1款高性能的開源HTTP加速器，挪威最大的在線報紙 Verdens Gang (http://www.vg.no) 使用3臺Varnish代替了原來的12臺squid，性能竟然比之前更好。

Varnish 的作者Poul-Henning Kamp是FreeBSD的內核開發者之1，他認為現在的計算機比起1975年已復雜許多。在1975年時，貯存媒介只有兩種：內存與硬盤。但現在計算機系統的內存除主存外，還包括了cpu內的L1、L2，乃至有L3快取。硬盤上也有自己的快取裝置，因此squid cache自行處理物件替換的架構不可能得知這些情況而做到最好化，但操作系統可以得知這些情況，所以這部分的工作應當交給操作系統處理，這就是 Varnish cache設計架構。
【以上摘自http://www.oschina.net/p/varnish】

開始

空話說完了，接下來就是怎樣使用它了。本文描寫的內容基于Varnish 4.1.2版本。

安裝Varnish：

（1）源碼安裝
下面這個是Varnish的官網：https://www.varnish-cache.org/
github的地址：https://github.com/varnishcache/varnish-cache
下載下來源碼包，進行編譯安裝。
（2）命令安裝
在官網上可以看到說明：
https://www.varnish-cache.org/releases/install_ubuntu.html#install-ubuntu

apt-get install apt-transport-https
curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
echo "deb https://repo.varnish-cache.org/ubuntu/ trusty varnish⑷.1" \
     >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish

可以看到，通過命令行直接安裝比較方便。

配置Varnish

這1點還是值得注意的，我在配置的時候，發現網上大部份文檔都是之前的版本，但是在4.x版本以后，配置方式產生了變化，也折騰了些時間，所以，希望能幫助你跳過坑。

Varnish運行參數配置

默許的端口是監聽6081，你怎樣知道？
看這個文件：

vi /etc/default/varnish

tfxiaozi@server:~$ cat /etc/default/varnish
# Configuration file for Varnish Cache.
#
# /etc/init.d/varnish expects the variables $DAEMON_OPTS, $NFILES and $MEMLOCK
# to be set from this shell script fragment.
#
# Note: If systemd is installed, this file is obsolete and ignored.  You will
# need to copy /lib/systemd/system/varnish.service to /etc/systemd/system/ and
# edit that file.

# Should we start varnishd at boot?  Set to "no" to disable.
START=yes

# Maximum number of open files (for ulimit -n)
NFILES=131072

# Maximum locked memory size (for ulimit -l)
# Used for locking the shared memory log in memory.  If you increase log size,
# you need to increase this number as well
MEMLOCK=82000
DAEMON_OPTS="-a :6081 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,24G"
tfxiaozi@server:~$

沒錯啦，就是這里，Varnish運行時占用了6081端口，6082用作管理。

-a 如果希望使用80端口直接訪問，修改成80便可  
-s 使用的內存大小
-S 認證的密鑰文件 varnishadm -T localhost:6082 -S secret
-f 指定了Varnish運行起來時需要加載的配置文件，用來管理后端服務器

Varnish 密鑰文件

在上面的配置中可以看到，密鑰文件在/etc/varnish下。

訪問http://varnish_VPS_public_IP:6081，前面是Varnish安裝的機器ip，此時會出現503訪問毛病，緣由是沒有配置后端要cache的服務器。

配置后端web服務器

這里以后端FastDFS存儲集群機器為例進行配置，目的是讓Varnish對FastDFS的Storage節點進行緩存，以下降直接訪問Storage節點釀成的壓力。
Storage1: 10.10.10.81;
Storage2: 10.10.10.82.
給出詳細的配置文件：

root@server:/etc/varnish# cat default.vcl 
#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and https://www.varnish-cache.org/trac/wiki/VCLExamples for more examples.

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

import std;
import directors;
# Default backend definition. Set this to point to your content server.
#backend default {
#    .host = "127.0.0.1";
#    .port = "8080";
#}

#probe healthcheck {
#   .url = "/";
#   .interval = 6s;
#   .timeout = 1s;
#   .window = 5;
#   .threshold = 3;
#}

backend fdfs_g1_s1 {
     .host = "10.10.10.81";
     .port = "10000";
#     .probe = healthcheck;     #健康狀態檢測
}

backend fdfs_g1_s2 {
     .host = "10.10.10.82";
     .port = "10000";
#    .probe = healthcheck;     #健康狀態檢測
}

acl purgers {
    "localhost";
    "127.0.0.1";
}

sub vcl_init {
    new vdir = directors.round_robin();
    vdir.add_backend(fdfs_g1_s1);
    vdir.add_backend(fdfs_g1_s2);
    return (ok);
}


sub vcl_recv {
     # Happens before we check if we have this in cache already.
     #
     # Typically you clean up the request here, removing cookies you don't need,
     # rewriting the request, etc.
     set req.backend_hint = vdir.backend();
     if (req.http.x-forwarded-for) {
         set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
     } else {
         set req.http.X-Forwarded-For = client.ip;
     }    

     if (req.method == "PURGE") {    # PURGE要求的處理
        if (!client.ip ~ purgers) {
            return(synth(405,"Method not allowed"));
        }
        #清算緩存
        return (purge);
     }
     # 制止緩存的文件
     if (req.url ~ "\.(php|jsp|do)($|\?|#)") {
        return (pass);
     }

     if (req.method == "PRI") {
        #/* We do not support SPDY or HTTP/2.0 */
        return (synth(405));
     }
     if (req.method != "GET" &&
       req.method != "HEAD" &&
       req.method != "PUT" &&
       req.method != "POST" &&
       req.method != "TRACE" &&
       req.method != "OPTIONS" &&
       req.method != "DELETE") {
         #/* Non-RFC2616 or CONNECT which is weird. */
         return (pipe);
     }

     if (req.method != "GET" && req.method != "HEAD") {
         #/* We only deal with GET and HEAD by default */
         return (pass);
     }
     if (req.http.Authorization || req.http.Cookie) {
         #/* Not cacheable by default */
         return (pass);
     }
     return (hash);
}

sub vcl_backend_response {
    # Happens after we have read the response headers from the backend.
    #
    # Here you clean the response headers, removing silly Set-Cookie headers
    # and other mistakes your backend does.
    if (bereq.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
        unset beresp.http.set-cookie;
    }    
    # Large static files are delivered directly to the end-user without
    # waiting for Varnish to fully read the file first.
    # Varnish 4 fully supports Streaming, so use streaming here to avoid locking.
    if (bereq.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|rar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") {
       unset beresp.http.set-cookie;
       set beresp.do_stream = true;  # Check memory usage it'll grow in fetch_chunksize blocks (128k by default) if the backend doesn't send a Content-Length header, so only enable it for big objects
       set beresp.do_gzip   = false;   # Don't try to compress it for storage
    }
    # Set 2min cache if unset for static files
    if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") {
      set beresp.ttl = 120s; # Important, you shouldn't rely on this, SET YOUR HEADERS in the backend
      set beresp.uncacheable = true;
      return (deliver);
    }
    # Allow stale content, in case the backend goes down.
    # make Varnish keep all objects for 6 hours beyond their TTL
    set beresp.grace = 6h;

    return (deliver);
}


sub vcl_backend_fetch {
    return (fetch);
}

sub vcl_pipe {
     # By default Connection: close is set on all piped requests, to stop
     # connection reuse from sending future requests directly to the
     # (potentially) wrong backend. If you do want this to happen, you can undo
     # it here.
     # unset bereq.http.connection;
     return (pipe);
}

sub vcl_pass {
     return (fetch);
}

sub vcl_hash {
     hash_data(req.url);
     if (req.http.host) {
         hash_data(req.http.host);
     } else {
         hash_data(server.ip);
     }
     return (lookup);
}

sub vcl_purge {
     return (synth(200, "Purged"));
}

sub vcl_hit {
     if (obj.ttl >= 0s) {
         #// A pure unadultered hit, deliver it
         return (deliver);
     }
     if (obj.ttl + obj.grace > 0s) {
         #// Object is in grace, deliver it
         #// Automatically triggers a background fetch
         return (deliver);
     }
     #// fetch & deliver once we get the result
     return (miss);
}

sub vcl_miss {
     return (fetch);
}

sub vcl_deliver {
    set resp.http.x-hits = obj.hits;
    if (obj.hits > 0) {
        set resp.http.X-Cache = "Hit varnish cache";
    }else {
        set resp.http.X-Cache = "Miss varnish cache";
    }
    return (deliver);
} 

sub vcl_synth {
     set resp.http.Content-Type = "text/html; charset=utf⑻";
     set resp.http.Retry-After = "5";
     synthetic( {"<!DOCTYPE html>
 <html>
   <head>
     <title>"} + resp.status + " " + resp.reason + {"</title>
   </head>
   <body>
     <h1>Error "} + resp.status + " " + resp.reason + {"</h1>
     <p>"} + resp.reason + {"</p>
     <h3>Guru Meditation:</h3>
     <p>XID: "} + req.xid + {"</p>
     <hr>
     <p>Varnish cache server</p>
   </body>
 </html>
 "} );
     return (deliver);
}


sub vcl_backend_error {
     set beresp.http.Content-Type = "text/html; charset=utf⑻";
     set beresp.http.Retry-After = "5";
     synthetic( {"<!DOCTYPE html>
 <html>
   <head>
     <title>"} + beresp.status + " " + beresp.reason + {"</title>
   </head>
   <body>
     <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
     <p>"} + beresp.reason + {"</p>
     <h3>Guru Meditation:</h3>
     <p>XID: "} + bereq.xid + {"</p>
     <hr>
     <p>Varnish cache server</p>
   </body>
 </html>
 "} );
     return (deliver);
}

sub vcl_fini {
     return (ok);
}

默許情況下，如果只有1臺機器做web server，使用backend default進行配置便可。如果是多臺機器提供相同的內容服務，那末可以將多個web server進行組管理，通過robin輪詢或random提供服務，具體要注意的細節：
①自定義后臺server

backend fdfs_g1_s1 {
     .host = "10.10.10.81";
     .port = "10000";
#     .probe = healthcheck;     #健康狀態檢測
}

backend fdfs_g1_s2 {
     .host = "10.10.10.82";
     .port = "10000";
#    .probe = healthcheck;     #健康狀態檢測
}

②初始化

sub vcl_init {
    new vdir = directors.round_robin();
    vdir.add_backend(fdfs_g1_s1);
    vdir.add_backend(fdfs_g1_s2);

    return (ok);
}

初始化操作放到init中便可。
③然后，需要在vcl_recv中進行設置當前使用的backend，

set req.backend_hint = vdir.backend();

做完這3步，varnish就配置好了，固然還可以根據需要配置節點健康檢查。

啟動Varnish

履行以下命令：

service varnish start

可以通過netstat -unltp | grep varnish 查看當前端口是不是被varnish占用：

root@server:/etc/varnish# netstat -unltp | grep varnish
tcp        0      0 0.0.0.0:6081            0.0.0.0:*               LISTEN      26565/varnishd  
tcp        0      0 127.0.0.1:6082          0.0.0.0:*               LISTEN      26565/varnishd  
tcp6       0      0 :::6081                 :::*                    LISTEN      26565/varnishd  
tcp6       0      0 ::1:6082                :::*                    LISTEN      26565/varnishd

測試

既然配置好了，那末就改測試1下，看看效果怎樣樣了。
Storage1: 10.10.10.81;
Storage2: 10.10.10.82;
我這里varnish裝在192.168.165.21機器上，配置了雙網卡。
這里寫圖片描述
可以看到，當訪問6081端口時，出現了nginx的界面，這是由于在FastDFS配置了Storage使用Nginx提供http服務，也就是說當前配置到后端服務器是OK了。

下面來測試1下緩存到底有無起作用？
第1次：

Request URL:http://192.168.165.21:6081/group1/M00/00/00/CgoK0ldybhOAYFAfAAAOelbujmQ479.png
Request Method:GET
Status Code:200 OK
Remote Address:192.168.165.21:6081
Response Headers
view source
Accept-Ranges:bytes
Age:0
Connection:keep-alive
Content-Length:3706
Content-Type:image/png
Date:Tue, 28 Jun 2016 12:31:56 GMT
Last-Modified:Tue, 28 Jun 2016 12:31:15 GMT
Server:nginx/1.8.1
Via:1.1 varnish-v4
X-Cache:Miss varnish cache
x-hits:0
X-Varnish:98311
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, sdch
Accept-Language:zh-CN,zh;q=0.8
Connection:keep-alive
DNT:1
Host:192.168.165.21:6081
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36

可以看到，第1次沒有再緩存中命中：X-Cache:Miss varnish cache

第2次：

Request URL:http://192.168.165.21:6081/group1/M00/00/00/CgoK0ldybhOAYFAfAAAOelbujmQ479.png
Request Method:GET
Status Code:200 OK
Remote Address:192.168.165.21:6081
Response Headers
view source
Accept-Ranges:bytes
Age:16
Connection:keep-alive
Content-Length:3706
Content-Type:image/png
Date:Tue, 28 Jun 2016 12:31:56 GMT
Last-Modified:Tue, 28 Jun 2016 12:31:15 GMT
Server:nginx/1.8.1
Via:1.1 varnish-v4
X-Cache:Hit varnish cache
x-hits:1
X-Varnish:13 98312
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, sdch
Accept-Language:zh-CN,zh;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
DNT:1
Host:192.168.165.21:6081
If-Modified-Since:Tue, 28 Jun 2016 12:31:15 GMT
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36

可以看到，第2次命中：X-Cache:Hit varnish cache。

清算緩存

配置文件指定了PURGE和IP 就能夠用curl 清算緩存

root@server:/etc/varnish# curl -X PURGE http://127.0.0.1:6081/group1/M00/00/00/CgoK0ldybhOAYFAfAAAOelbujmQ479.png     
<!DOCTYPE html>
 <html>
   <head>
     <title>200 Purged</title>
   </head>
   <body>
     <h1>Error 200 Purged</h1>
     <p>Purged</p>
     <h3>Guru Meditation:</h3>
     <p>XID: 98320</p>
     <hr>
     <p>Varnish cache server</p>
   </body>
 </html>
 root@server:/etc/varnish#

1些命令

①varnishstat
可以查看緩存命中率
緩存命中率的高低，直接反應Varnish的運行狀態，以下通過varnishstat命令查看狀態信息：
這里寫圖片描述
Hitrate n 第1個數字范圍0⑴0，第2個數字范圍0⑴00，第3個數字范圍0⑴000。分別表示過去N秒內的
avg(n) 里的內容是命中率，需要乘以100轉換成百分比。
②varnishlog
可以查看實光陰志。

附錄

1篇英文文檔
https://www.digitalocean.com/community/tutorials/how-to-configure-varnish-cache⑷-0-with-ssl-termination-on-ubuntu⑴4-04
github上1篇關于4.0的詳細配置模板：
https://github.com/mattiasgeniar/varnish⑷.0-configuration-templates/blob/master/default.vcl
1篇關于之前低版本的配置，里面講了關于Varnish性能優化和varnishadm 命令。
http://xiaodong88.blog.51cto.com/1492564/1304988

生活不易，碼農辛苦
如果您覺得本網站對您的學習有所幫助,可以手機掃描二維碼進行捐贈
程序員人生