Adventures in IT

This technique was developed in the Ruby community and is a great way to approach caching partial views. In Ruby rendering views is more expensive than PHP, but this technique is worth understanding as it could be applied to data models and not just views. In the Ruby world Russian Doll caching is synonymous with key-based expiration caching.  I think it's useful to rather view the approach as being the blend of two ideas.  That's why I introduce key-based expiration separately. Personally I think Russian Dolls are a bit of a counter-intuitive analogy.  Real life Russian Dolls each contain one additional doll, but the power of this technique rests on the fact that "dolls" can contain many other "dolls".  I find the easiest way to think about it is to say that if a child node is invalidated then its siblings and their children are not affected.  When the parent is regenerated those sibling nodes do not need to be rendered again. Cache Invalidation ...

Memcached saved my users a minute per query I'm working on a business intelligence tool that requires as an intermediary calculation a list of UK postcodes that are within a radius of a user supplied postcode. It currently takes about 7 seconds to query my Postgres database to get this list out.  Unfortunately I need to do this several times as part of a goal seeking function so I need to greatly improve this lookup speed. I'm already using the Postgres earthdistance module and have properly indexed my table so I realized that I needed to look for a caching solution. Memcached places limits on the size of the value you can store.  The default setup is 1meg and I'm reluctant to change this because it adds to the deployment burden.  My result sets were sometimes up to 4 megs large - searching on a 20 mile radius in London yields a lot of postcodes! My idea was to split the large piece of data into several smaller pieces and to place an index referencing the pie...

I'm using Varnish to cache a fairly busy property site.  Varnish works like a bomb for normal users and has greatly improved our page load speed. For bots that are scraping the site, presumably to add the property listings to their own site, though the cache is next to useless since the bots are sequentially trawling through the whole site. I decided to use fail2ban to block IP's who hit the site too often. The first step in doing so was to enable a disk based access log for Varnish so that fail2ban will have something to work with. This means setting up varnishncsa.  Add this to your /etc/rc.local file: varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid This starts up varnishncsa in daemon mode and appends Varnish access attempts to /var/log/varnish/access.log Now edit or create /etc/logrotate.d/varnish and make an entry to rotate this access log: /var/log/varnish/*log { create 640 http log compress ...

Postgres has two very useful extensions - earthdistance and postgis .  PostGIS is much more accurate but I found earthdistance to be very easy to use and accurate enough for my purpose (finding UK postcodes within a radius of a point). To install it first find your Postgres version and then install the appropriate package.  On my Debian Mint dev box it looks like the below snippet. My production machine is an Amazon RDS and you can skip this step in that environment. psql -V sudo apt-get install postgresql-contrib postgresql-contrib-9.3 sudo service postgresql restart Having done that you should launch psql and run these two commands.  Make sure that you install cube first because it is a requirement of earthdistance. CREATE EXTENSION cube; CREATE EXTENSION earthdistance; Now that the extensions are installed you have access to all of the functions they provide. If you want to check that they're working you can run SELECT earth(); as a quick wa...

I'm busy on a project which involves importing fairly large datasets of about ~3.3GB at a time.  I have to read a CSV file, process each line, and generate a number of database records from the results of that process. Users are expected to be able to rerun batches and there is overlap between different datasets.  For example: the dataset of "last year" overlaps with the dataset of "all time".  This means that we need an elegant way to handle duplicate updates. Searching if a record exists (by PK) is fine until the row count in the table gets significant.  At just over 2 million records it was taking my development machine 30 seconds to process 10,000 records.  This number steadily increased as the row count increased. I had to find a better way to do this and happened across the option of using a database rule to ignore duplicates.  While using the rule there is a marked improvement in the performance as I no longer need to search the database fo...

PHP Framework popularity as at 2013 - Sitepoint After having a very good experience with using Varnish to cache a Wordpress site we decided to look at caching Laravel. Laravel always generates cookies regardless of whether a person is logged in or not.  This interferes with Varnish which by default will pass all requests with a cookie to the backend and skip the cache. In our particular case our site supported the ability for users to login and would then present them with custom content.  This means that cookies are not restricted to a particular path so we can't discard cookies based on the request as we did for Wordpress when discarding everything except /wp-admin/* requests. My solution was to use a package called session-monster ( Packagist  ) which sets a response header if the data in the Laravel session can be ignored.  Varnish can detect this header and prevent the cookie from being set since we don't really need it.  This together with t...