NFS, the slow performance Network File System

September 29th, 2008

As described, we are using 2 web servers and 1 application server. What’s the background? It’s easy to explain. Every good designed and developed website needs a template engine for visualization. Our template engine is smarty. Smarty has the caching feature and I like it to minimize the database connections and optimize the performance. I cache some pages (profile, news) until the next update (e.g. new comment) was done.

The problem is if the PHP script has deleted the cache file on the first web server (tbweb01), the cache file on the second web server (tbweb02) isn’t deleted. To avoid any delay times, we stored our source code on the application server and we are using NFS to share the files. Normally it works fine, but on some peak times the NFS is too slow.

Information like “logged in user” and so on are saved on the session. Usually the user opens the website and the browser always uses the same ip address. It does not matter which server it is. If the user uses a proxy, the proxy chooses one of the round robin ip addresses by each click. So on one server the user has a correct session with the information, on the other server he hasn’t. If we store the sessions on the NFS we make sure that every server uses the same session. But the performance…

Now we have to search for a good solution!
I found two interesting articles. One about a session_set_save_handler and the other about a cache_handler_func. And a new idea was born. Use NFS only to read the files and store the information in the database.

session_set_save_handler:
This default PHP function allows you to use your own way for the session handling. I think it’s really better to store everything in a mysql table and avoid the NFS storage. On the article they didn’t open and close a database connection. That’s bad… Normally your page is already using a connection, but sometimes you don’t have to open one.

It’s a little bit tricky, but not a problem for experienced developer. I wrote a class to handle my database connection(s). It was very important to use one class for more connections, but it was much more important that the class doesn’t connect with the same user and the same selected database twice times and make sure that the class closes every connection correct. I called my class “classDataBase” and overloaded the most used mysql functions. To make sure that the handling works, we’ve to use a static array for the connection and remember the used index in an attribute. Also we remember the count of objects for each connection. On calling the close function, we don’t want to close always the database connection. If more than one object uses this connection, we only decrease the count variable. To be sure that the PHP script closes all opened connections correctly, the class registers automatically on the initializing of the first object a static DBCloseAll() function for shutdown.

classDataBase.php

/******************************************
 * class created by Exi (http://www.technobase.fm/member/1)
 * please find some information about on
 * http://blog.rarecore.eu/nfs-the-unperformed-file-system.html
 ******************************************/
 
class classDataBase {
 
    public static $connections;
    var $connection;
    var $Idx = 0;
    var $DBName;
 
    // error handling
    var $errNum; // error number
    var $errTxt; // error message
 
    // constructor - set default values
    function classDataBase() {
        if(!is_array(classDataBase::$connections)) {
            classDataBase::$connections = array();
            register_shutdown_function(array('classDataBase', 'DBCloseAll'));
        }
 
        $this-&gt;errNum = 0;
        $this-&gt;errTxt = "";
        $this-&gt;connectionID = false;
    }
 
    function Init($cDBName, $cDBHost, $cDBUser, $cDBPass) {
        $found = false;
        $retVal = false;
 
        if($this-&gt;connection) {
            $this-&gt;DBClose();
        }
 
        for($i=0; $iconnection = $a['connection'];
                        classDataBase::$connections[$i]['count']++;
                        $this-&gt;Idx = $i;
                        $found = true;
                        $retVal = true;
                        break;
                    }
            }
        }
 
        if(!$found) {
            $a = array();
 
            $retVal = $this-&gt;DBConnecting($cDBName, $cDBHost, $cDBUser, $cDBPass);
 
            if(!$retVal) {
                //Lost connection to MySQL server at 'reading authorization packet', system error: 0
                if($this-&gt;errNum != 2013) {
                    $mail  = "Error on connect to the Databasen";
                    $mail .= "Error Text: ".$this-&gt;errTxt."n";
                    $mail .= "Error Number: ".$this-&gt;errNum."n";
 
                    $this-&gt;ErrorMail($mail);
                }
 
                echo "<strong>Error on connect to the Database</strong>";
                exit();
            }    
 
            $a['dbname'] = $cDBName;
            $a['dbhost'] = $cDBHost;
            $a['dbuser'] = $cDBUser;
            $a['connection'] = $this-&gt;connection;
            $a['count'] = 1;
            array_push(classDataBase::$connections, $a);
        }
 
        $this-&gt;DBName = $cDBName;
 
        return $retVal;
    }
 
    function DBConnecting($cDBName, $cDBHost, $cDBUser, $cDBPass) {
        $retVal = true;
 
        // connect to database
        $this-&gt;connection = @mysql_connect($cDBHost, $cDBUser, $cDBPass);
 
        // throw error if connection failed ...
        if (!$this-&gt;connection) {
            $this-&gt;errNum = @mysql_errno();
            $this-&gt;errTxt = @mysql_error();
            $retVal = false;
        } else if(!@mysql_select_db($cDBName, $this-&gt;connection)) {
            $this-&gt;errNum = @mysql_errno();
            $this-&gt;errTxt = @mysql_error();
            $retVal = false;
        }
 
        return $retVal;
    }
 
    //function to execute statements
    function execSQL($SQLQuery, $mail = true) {
        $XSQL = false;
 
        if($this-&gt;connection) {
            $XSQL = mysql_query($SQLQuery, $this-&gt;connection);
 
            // Check for error
            if (!$XSQL) {
                $this-&gt;errNum = @mysql_errno($this-&gt;connection);
                $this-&gt;errTxt = @mysql_error($this-&gt;connection);   
 
                if($mail) {
                    $this-&gt;LogDBError($SQLQuery);
                }
            }
        }
 
        return $XSQL;
    }
 
    function query($SQLQuery) {
        return $this-&gt;execSQL($SQLQuery);
    }
 
    function queryCatch($SQLQuery) {
        return $this-&gt;execSQL($SQLQuery, false);
    }
 
    function insertId() {
        return mysql_insert_id($this-&gt;connection);
    }
 
    function affectedRows() {
        return mysql_affected_rows($this-&gt;connection);
    }
 
    //function to fetch the query
    function fetchArray($SQLQuery) {
        $XSQL = @mysql_fetch_array($SQLQuery);
 
        if(!$XSQL) {
            return false;
        } else {
            return $XSQL;
        }
    }
 
    function numRows($SQLQuery) {
        $XSQL = @mysql_num_rows($SQLQuery);
 
        if(!$XSQL) {
            return 0;
        } else {
            return $XSQL;
        }
    }
 
    //function to send a mail on error to the programer
    function ErrorMail($Content) {
        // send an email to the programmer?
        echo '<strong>change the ErrorMail function!!!</strong>'.$Content;
    }
 
    function LogDBError($Statement = "") {
        /*
        // log the database errors?!?
        $qry = 'INSERT INTO dberror
                        SET dberr_number = ''.$this-&gt;errNum.'',
                            dberr_text = ''.mysql_escape_string($this-&gt;errTxt).'',
                            dberr_statement = ''.mysql_escape_string($Statement).'',
                            dberr_site = ''.$_SERVER['HTTP_HOST'].$_SERVER['REQUEST_URI'].'',
                            dberr_ip = ''.$_SERVER['REMOTE_ADDR'].'',
                            dberr_time = ''.time().'',
                            dberr_sent = 0';
 
        @mysql_db_query($this-&gt;DBName, $qry, $this-&gt;connection);*/
 
        echo nl2br('
              dberr_number = ''.$this-&gt;errNum.'',
              dberr_text = ''.$this-&gt;errTxt.'''
             );
 
        echo '<strong>change the LogDBError function!!!'</strong>.$Statement;
    }
 
    function ListTables() {
        $result = @mysql_list_tables($this-&gt;DBName, $this-&gt;connection);
 
        $table_names = array();
 
        for($x=0; $x &lt; $this-&gt;numRows($result); $x++) {
            $table_names[$x] = @mysql_tablename($result,$x);
        }
 
        return $table_names;
    }
 
    function DBClose() {
        if($this-&gt;connection) {
            if(classDataBase::$connections[$this-&gt;Idx]['count'] == 1) {
                @mysql_close($this-&gt;connection);
            }          
 
            classDataBase::$connections[$this-&gt;Idx]['count']--;
            $this-&gt;connection = false;
        }
    }
 
    function DBCloseAll() {
        if(class_exists('mysql_session_handler')) {
            session_write_close();
        }
 
        for($i=0; $i
 
After creating our database class we want to write our own session handling. At first we've to create a simple MySQL Table. I called my table cache_session. I want to prefix my example tables with cache_.

CREATE TABLE `cache_sessions` (
  `session_id` varchar(100) NOT NULL default '',
  `session_data` text NOT NULL,
  `session_expires` int(11) NOT NULL default '0',
  PRIMARY KEY  (`session_id`)
) TYPE=innodb;

What is my session handler class doing? It’s a static class to overwrite the session handling. On construct I get the maxfiletime from the php configuration, I register my methods to read, write and destroy the session and create a database connection.

On read / write the session, the class use MySQL to store the information.

mysql_session_handler.php

/******************************************
 * class created by Exi (http://www.technobase.fm/member/1)
 * please find some information about on
 * http://blog.rarecore.eu/nfs-the-unperformed-network-file-system
 ******************************************/
include_once 'classDataBase.php';
if(!class_exists('mysql_session_handler')) {
    class mysql_session_handler {
        static $life_time;
        static $mysql;
 
        // construct
        function mysql_session_handler() {
            // read the maxlifetime setting from PHP
            mysql_session_handler::$life_time = get_cfg_var('session.gc_maxlifetime');    
 
            // register this object as the session handler
            session_set_save_handler(
                array(&amp;$this, 'open'),
                array(&amp;$this, 'close'),
                array(&amp;$this, 'read'),
                array(&amp;$this, 'write'),
                array(&amp;$this, 'destroy'),
                array(&amp;$this, 'gc')
            );  
 
            if(mysql_session_handler::$mysql == null) {
                // create a new database connection or use the already opened
                mysql_session_handler::$mysql = new classDataBase();
                mysql_session_handler::$mysql-&gt;Init('db', 'ip', 'user', 'password');
            }
        }
 
        // open session -&gt; just to have. no action
        function open($save_path, $session_name) {
            global $sess_save_path;
            $sess_save_path = $save_path;
            return true;
        }
 
        // read session data
        function read($id) {
            $data = '';
 
            $qry = mysql_session_handler::$mysql-&gt;query("select session_data
                                                           from cache_sessions
                                                          where session_id = '".mysql_escape_string($id)."'
                                                            and session_expires &gt;= ".time());
 
            while($fetch = mysql_session_handler::$mysql-&gt;fetchArray($qry)) {
                $data = $fetch[0];
            }
 
            return $data;
        }
 
        // insert / update session information
        function write($id, $data) {
            mysql_session_handler::$mysql-&gt;query("replace into cache_sessions
                                                              (session_id,
                                                               session_data,
                                                               session_expires)
                                                       values ('".mysql_escape_string($id)."',
                                                               '".mysql_escape_string($data)."',
                                                                ".(time() + mysql_session_handler::$life_time).")");
 
            return true;
        }
 
        // destroy a session
        function destroy($id) {
            mysql_session_handler::$mysql-&gt;query("delete from cache_sessions where session_id = '".mysql_escape_string($id)."'");
            return true;
        }
 
        // clean old session
        function gc() {
            // do not clean by an page call, use a cronjob to delete all expired entries every minute!
            /*mysql_session_handler::$mysql-&gt;query('delete from cache_sessions where session_expires &lt; '.time());*/
 
            return true;
        }
 
        // close session
        function close() {
            return true;
        }
    }   
 
    $SessionHandler = new mysql_session_handler();
}

Now, I want to do some tests…

include 'mysql_session_handler.php';
session_start();
if(!isset($_SESSION['a'])) {
    $_SESSION['a'] = 1;
    echo 'session "a" not set!';
} elseif($_SESSION['a'] == 10) {
    session_destroy();
    echo 'session destroyed';
 
} else {
    $_SESSION['a']++;
    echo 'session "a" set!';
}

looks good – works!

The next step is to store the smarty cache in the database. I always write my own classes to overwrite existing open source projects. So I could implement features / settings and software updates doesn’t touch me… I created a classSmarty and opened a database connection using the classDataBase class.

classSmarty.php

/******************************************
 * class created by Exi (http://www.technobase.fm/member/1)
 * please find some information about on
 * http://blog.rarecore.eu/nfs-the-unperformed-file-system.html
 ******************************************/
 
include_once 'Smarty.class.php';
include_once 'classDataBase.php';
include_once 'mysql_cache_handler.php';
 
class classSmarty extends Smarty {
    var $mysql;
 
    function classSmarty() {
        parent::Smarty(); 
 
        $this-&gt;template_dir  = '_smarty/templates';
        $this-&gt;compile_dir   = '_smarty/templates_c';
        $this-&gt;cache_dir     = '_smarty/cache';
        $this-&gt;config_dir    = '_smarty/config';
 
        $this-&gt;compile_check = true;
        $this-&gt;debugging     = false;
        $this-&gt;caching       = true;
 
        $this-&gt;mysql = new classDataBase();
        $this-&gt;mysql-&gt;Init('db', 'host', 'user', 'pass');
        $this-&gt;cache_handler_func = 'mysql_cache_handler';
    }
}

As descripted in the smarty forum I created a function to handle the caching by mysql.

CREATE TABLE `cache_smarty` (
  `smarty_cacheId` varchar(32) NOT NULL,
  `smarty_template` varchar(255) NOT NULL,
  `smarty_group` varchar(255) NOT NULL,
  `smarty_content` mediumtext NOT NULL,
  `smarty_expire` int(11) NOT NULL,
  PRIMARY KEY  (`smarty_cacheId`),
  KEY `smarty_template` (`smarty_template`),
  KEY `smarty_expire` (`smarty_expire`)
) ENGINE=InnoDB ;

mysql_cache_handler.php

/******************************************
 * class created by Exi (http://www.technobase.fm/member/1)
 * please find some information about on
 * http://blog.rarecore.eu/nfs-the-unperformed-file-system.html
 ******************************************/
 
function mysql_cache_handler($action, &amp;$smarty_obj, &amp;$cache_content, $tpl_file = null, $cache_group = null, $compile_id = null, $exp_time = 0) {
    $db_table = 'cache_smarty';
 
    // create unique cache id
    $cache_id = md5($tpl_file.$cache_group.$compile_id); 
 
    switch ($action) {
        case 'read':
            // read cache from database
            $results = $smarty_obj-&gt;mysql-&gt;query("select smarty_content from $db_table where smarty_cacheId = '$cache_id' and smarty_expire &gt; ".time()); 
 
            if(!$results) {
                $smarty_obj-&gt;_trigger_error_msg('cache_handler: query failed.');
            } 
 
            $row = $smarty_obj-&gt;mysql-&gt;fetchArray($results);
            $cache_content = $row[0]; 
 
            $return = $results;
            break; 
 
        case 'write':
            // save cache to database
            $results = $smarty_obj-&gt;mysql-&gt;query("replace into $db_table (
                                                           smarty_cacheId,
                                                           smarty_template,
                                                           smarty_group,
                                                           smarty_content,
                                                           smarty_expire
                                                         ) values (
                                                           '$cache_id',
                                                           '$tpl_file',
                                                           '$cache_group',
                                                           '".addslashes($cache_content)."',
                                                           '".($exp_time == 0 ? time() + 3600 : $exp_time)."' )"); // no expire =&gt; expire next hour
 
            if(!$results) {
                $smarty_obj-&gt;_trigger_error_msg('cache_handler: query failed.');
            } 
 
            $return = $results;
            break; 
 
        case 'clear':
            // clear cache info
            if(empty($cache_group) &amp;&amp; empty($compile_id) &amp;&amp; empty($tpl_file)) {
                $results = $smarty_obj-&gt;mysql-&gt;query("delete from $db_table"); 
 
            } else {
                /*
                if(!$smarty_obj-&gt;mysql-&gt;query("delete from $db_table where smarty_expire &lt; ".time())) {
                    $smarty_obj-&gt;_trigger_error_msg('cache_handler: clear expired entries query failed.');
                }
                */
 
                if(strpos($cache_group, '|')) {
                   if(!empty($tpl_file)) {
                      $results = $smarty_obj-&gt;mysql-&gt;query("delete from $db_table where smarty_template = '$tpl_file' AND smarty_group LIKE '$cache_group%'"); 
 
                   } else {
                      $results = $smarty_obj-&gt;mysql-&gt;query("delete from $db_table where smarty_group LIKE '$cache_group%'"); 
 
                   }
                } else {
                   $results = $smarty_obj-&gt;mysql-&gt;query("delete from $db_table where smarty_cacheId = '$cache_id'"); 
 
                }
            } 
 
            if(!$results) {
                $smarty_obj-&gt;_trigger_error_msg('cache_handler: query failed.');
            } 
 
            $return = $results;
            break; 
 
        default:
            // error, unknown action
            $smarty_obj-&gt;_trigger_error_msg("cache_handler: unknown action "$action"");
            $return = false;
            break;
    } 
 
    return $return;
}

Now, I want to do some tests again…

include_once 'classSmarty.php';
 
$smarty = new classSmarty();
$smarty-&gt;caching = true;
$smarty-&gt;cache_lifetime = 10;
 
$template = 'smarty.html';
$cache_id = 'smarty|1337';
 
if(!$smarty-&gt;is_cached($template, $cache_id)) {
    $smarty-&gt;assign('time', date('H:i:s', time()));
    $smarty-&gt;assign('a', rand(1, 9999));
    $smarty-&gt;assign('b', rand(1, 9999));
    $smarty-&gt;assign('c', rand(1, 9999));
}
 
$smarty-&gt;display($template, $cache_id);

and works again!

And last but not least we want to delete all expired entries. Create a cronjob which is running every minute to do this.

I hope the blog helps you!

Tags: cache_handler_func, hardbase, housetime, network file system, NFS, PHP, session_set_save_handler, Smarty, technobase, trancebase, weareone

You can leave a response, or trackback from your own site.

6 Responses to “NFS, the slow performance Network File System”

Spacefish Says:
October 8th, 2008 at 9:29 pm
Hey there, why don´t you mirror the files on both webserver and sync them with rsync? or do you know memcached? it´s we use it to accelerate smarty cache. It is decentralized (you could build a memcached cluster) and is EXTREME!! fast!
http://www.danga.com/memcached/ <- it´s open source ;-) you could avoid a lot of DB Querys with it also by caching results from the DB for the news maybe. If someone posts a comment or post new news, you flush the cachekey for that specific page. It can accelerate Webserver a lot. You could also cache parts of HTML in a website, so you don´t have to rerender it on every request. You could cache the commentboxes or the menu or whatever ^^ results in lesser disk reads and lesser CPU usage. We save aprox. 70% mysql querys with it.
Exi Says:
October 10th, 2008 at 7:56 am
Hi,

the best on my solution is that you could choose an extra database server for the cache (smarty & session) and use 2 or more different servers for the cache.

Save the result in the database (smarty) is much fast as saving the result of a query in the ram. the webserver don’t have to create the part of the site again, it’s already done. He only have to print the string.

Greatz
joe Says:
October 27th, 2008 at 9:15 pm
No.
Saving the session and smarty cache in the DB is not as fast as saving it in the main memory. By looking at your page I see that NFS/caching is probably not your real problem. It is Apache. Try the lighttpd or litespeed webservers, they are significantly faster than Apache. By using one of those + memcached, you should be able to serve about 1000 users/second if you use a second server that handles the database.
By the way: I don’t understand your argument, that saving something in the DB can be faster than saving it in the memory – that is against all laws of computer science since the DB is just another layer of abstraction which does require more code than saving something to the memory.
Ruth Says:
March 27th, 2009 at 6:01 am
I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.

Ruth

http://laptopmessengerbag.info
web 2.0 development company Says:
November 9th, 2009 at 10:00 am
thanks for this & here is many relevant content written for developers to read
simply like it most . .
Robert Says:
February 15th, 2010 at 3:58 pm
NFS isn’t an option for scaleout at all. In general, there is no need to write the session handler for MySQL for yourself, as there is a much faster solution: http://websupport.sk/~stanojr/projects/session_mysql/

There is also a session handler for ndb, but memcache should also do the job for the moment.

The problem of sharing cache files between application servers can also be solved without NFS using an asynchron cache purging system. But you should not cache the complete site, if you offer the possebility of commenting or any other dynamic features. If you have something dynamically on the page think about another method, such as blockwise caching using memcache, apc, or in the easiest case by storing serialized objects on the hard drive.

Robert

NFS, the slow performance Network File System

6 Responses to “NFS, the slow performance Network File System”

Leave a Reply