Adventures in IT

LXD logo from Ubuntu blog Selling a software package that is deployed to a customer data-centre can be challenging due to the diversity found in the physical infrastructure. It is expensive to make (and test) adjustments to the software that allow it to run in a non-standard manner. Maintaining a swarm of snowflakes is not a scalable business practice. More deployment variations means more documentation and more pairs of hands needed to manage them. Uniformity and automation are what keep our prices competitive. Opposing the engineering team's desire for uniformity is the practical need to fit our solution into the customers data-centre and budget. We can't define a uniform physical infrastructure that all of our customers must adhere to. Well, I suppose we could, but we would only have a very small number of customers who are willing and able to fit their data-centre around us. We are therefore on the horns of a dilemma. Specifying standard physical hard...

I've recently been playing with Docker on Windows in conjunction with Linux on Windows .  I'm really amazed at how cool the stuff coming out of Microsoft is under Satya Nadella.   When I was using Docker Toolbox on Windows my Dockerfiles would build correctly but as soon as I tried to run them in a Virtualbox host they would fail with the error " Something wicked happened resolving 'archive.ubuntu.com:http' (-5 - No address associated with hostname) " Of course this error wasn't distribution specific and none of the distros I tried were working. The stack that I am using is Windows Home hosting Ubuntu on VirtualBox which is running Docker.  I'm using bash on Linux for Windows because it's easier to do stuff like ssh but it's not relevant to this setup. I tried setting the DNS in the Docker setup by using RUN steps to update /etc/resolv.  This felt a bit hacky and didn't work anyway. In the end the fix was to go to my Window...

Image: https://boinc.berkeley.edu, fair use BOINC is a program curated by the University of Berkeley that allows people around the world to contribute to science projects.   It works by using spare cycles from your computer to perform calculations that help do things like folding proteins to find candidates for cancer treatment, mapping the milky way galaxy, searching for pulsar stars, and improving our understanding of climate change and its effects. It runs as a background process and is easily configured to only run in certain conditions - like when you haven't used your computer for 10 minutes for example. It comes with a nifty GUI manager and for most people using it on their desktop this post is not going to be at all relevant.  This post deals with the case where a person is running it on a server without the GUI manager. Anyway, the easiest solution I found to restarting BOINC on a headless server was to use supervisord .  It's pretty muc...

My Vagrant 1.7.2 machine bugged out and when I tried to `vagrant up` it spawned a new box instead of bringing up my existing machine. Naturally this was a problem because I had made some manual changes to the config that I hadn't had a chance to persist to my puppet config files yet. To fix the problem I found used the command ` VBoxManage list vms ` in the directory where my Vagrantfile is.  This provided me a list of the machine images it could find. I then went and edited the file at .vagrant/machines/default/virtualbox/id and replaced the UUID that was in there with the one that the VBoxManage command had output. Now when I run 'vagrant up' it spins up the correct VM.  Happy days.

Image: Pixabay Although I'm currently playing with Elixir and its HTTP servers like Cowboy at the moment Nginx is still my go-to server for production PHP. If you haven't already swapped your web-server from Apache then you really should consider installing Nginx on a test server and running some stress tests on it.  I wrote about stress testing in my book on scaling PHP . Redirecting non-www traffic to www in nginx is best accomplished by using the "return" verb.  You could use a rewrite but the Nginx manual suggests that a return is better in the section on " Taxing Rewrites ". Server blocks are cheap in Nginx and I find it's simplest to have two redirects for the person who arrives on the non-secure non-canonical form of my link.  I wouldn't expect many people to reach this link because obviously every link that I create will be properly formatted so being redirected twice will only affect a small minority of people. Anyway, here's...

I'm busy migrating an existing server to the cloud and need to replicate the SFTP setup.  They're using a password to authenticate a user and then uploading data files for a web service to consume. YMMV - My use case is pretty specific to this legacy application so you'll need to give consideration to the directories you use. It took a surprising amount of reading to find a consistent set of instructions so I thought I should document the setup from start to finish. Firstly, I set up the group and user that I will be needing: groupadd sftponly useradd -G sftponly username passwd username Then I made a backup copy of and then edited /etc/ssh/sshd_config Right at the end of the file add the following: Match group sftponly ChrootDirectory /usr/share/nginx/html/website_directory/chroot X11Forwarding no AllowTcpForwarding no ForceCommand internal-sftp -d /uploads For some reason if this block appears before the UsePAM s...

The cost saving from reserving an EC2 instance is quite dramatic.  This morning I moved two web servers to reserved instances and am dropping my hosting cost by 64% for those servers. There isn't actually any effort required in moving your on demand EC2 instance to a reserved instance.  The only actual change is a billing change, you don't need to do anything in your instance configuration. The only important thing to remember is that you're reserving an instance in a particular availability zone.  The billing effect will only apply to instances launched in the same availability zone. Amazon will apply the discount of having a reserved instance to your invoice automatically.  They provide quite extensive documentation on reserved instances on their site ( here ).

Reverse Proxy diagram from Wiki Commons We're currently hosting client sites on a Rackspace server and using their Load Balancer feature as a way to terminate SSL and not have to have multisite certificates. We only attach one node to the Load Balancer so we're paying for more than we're using.  My proof of concept is to use Nginx to terminate SSL certificates and proxy to the Apache server.  This will save us £ 225 per load balancer, and since we're using ten of them that's quite a significant saving. My first step was to spin up a free tier EC2 instance running Ubuntu 14.04 LTS.  I guess you can replace this with your favourite cloud or on-the-metal server. Then I installed my packages. These are the ones I remember so YMMV. sudo apt-get install nginx apache2 fail2ban php5-fpm mcrypt php5-mcrypt openssl php5-cli php5 libapache2-mod-php My network diagram is slightly different from the picture for this post in that the web server is hosted on the ...

I'm using Varnish to cache a fairly busy property site.  Varnish works like a bomb for normal users and has greatly improved our page load speed. For bots that are scraping the site, presumably to add the property listings to their own site, though the cache is next to useless since the bots are sequentially trawling through the whole site. I decided to use fail2ban to block IP's who hit the site too often. The first step in doing so was to enable a disk based access log for Varnish so that fail2ban will have something to work with. This means setting up varnishncsa.  Add this to your /etc/rc.local file: varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid This starts up varnishncsa in daemon mode and appends Varnish access attempts to /var/log/varnish/access.log Now edit or create /etc/logrotate.d/varnish and make an entry to rotate this access log: /var/log/varnish/*log { create 640 http log compress ...

I'm using Varnish to cache the frontend of a site that a client is busy promoting.  It does a great job of reducing requests to my backend but is prone to random crashes.  I normally get about two weeks of uptime on this particular server, which is significantly lower than other places that I've deployed Varnish. I just don't have enough information to work with to try and solve why the random crash is occurring.  The system log shows that a child process doesn't respond to CLI and so is killed.  The child never seems to be able to be brought up again. My /var/log/messages file looks like this: 08:31:45 varnishd[7888]: Child (16669) not responding to CLI, killing it. 08:31:45 varnishd[7888]: Child (16669) died signal=3 08:31:45 varnishd[7888]: child (25675) Started 08:31:45 varnishd[7888]: Child (25675) said Child starts 08:31:45 varnishd[7888]: Child (25675) said SMF.s0 mmap'ed 1073741824 bytes of 1073741824 08:32:19 varnishd[7888]: Child (2...

Hiphop has a built-in admin server that has a lot of useful functions.  I found out about it on an old post on the Hiphop blog . Since those times Hiphop has moved towards using an ini file instead of a config.hdf file. On a standard prebuilt HHVM on Ubuntu you should  find the ini file in /etc/hhvm/php.ini Facebook maintains a list of all the ini settings on Github . It is into this file that we add two lines to enable the admin server: hhvm.admin_server.password = SecretPassword hhvm.admin_server.port = 8888 I then added a server to Nginx by creating this file: /etc/nginx/conf.d/admin.conf (by default Nginx includes all conf files in that directory): server { # hhvm admin listen 8889; location ~ { fastcgi_pass 127.0.0.1:8888; include fastcgi_params; } } Now I can run curl 'http://localhost:8889' from my shell on the box to get a list of commands. Because I host this project with Amazon and have not set up a ...

On my desktop machine I run my package upgrades every day.  The other day my Hiphop version got updated to 3.6.0 and suddenly my Postgres support died. Running Hiphop gave a symbol not found error in the postgres.so file ( undefined symbol: _ZTIN4HPHP11PDOResourceE\n ) exactly like the issue reported on the driver repository ( here ). I tried to recompile the postgres driver against Hiphop 3.6.0 but hit a number of problems, mostly to do with  hhvm-pgsql-master/pdo_pgsql_statement.cpp it seems. The fix for the incompatibility was unfortunately rolling back to my previous version of Hiphop.  To do this on Mint/Ubuntu just do this: Run cat /etc/*-release to get your release information Download the appropriate package for your distro from http://dl.hhvm.com/ubuntu/pool/main/h/hhvm/ Remove your 3.6.0 installation of hhvm: sudo apt-get remove hhvm Install the package you downloaded : sudo dpkg -i <deb package> After that everything should be ...

While debugging and setting up Puppet I am still running the agent and master from CLI in --no-daemonize mode.  I kept getting an error on my agent - ""Exiting; no certificate found and waitforcert is disabled". The fix was quite simple and a little embarrassing.  Firstly I forgot to run my puppet master with root privileges which meant that it was unable to write incoming certificate requests to disk.  That's the embarrassing part and after I looked at my shell prompt and noticed this issue fixing it was quite simple. Firstly I got the puppet ssl path by running the command   puppet agent --configprint ssldir Then I removed that directory so that my agent no longer had any certificates or requests. On my master side I cleaned the old certificate by running  puppet cert clean --all  (this would remove all my agent certificates but for now I have just the one so its quicker than tagging it). I started my agent up with the command  puppet age...

PostGres is rapidly gaining traction as a solid relational database manager.  It provides transaction reliability (ACID), standards compliance, and has a better reputation for handling large datasets than MariaDB / mySQL. Luckily installing it for Hiphop is painless and there is an externally provided extension for it. There was not a prebuilt package available for the PocketRent/hhvm-pgsql  extension for my version of Hiphop so I built it following the advice in the project readme.  Their instructions worked first time. I did not have success with using the "ini" method of including the extension so had to create a Hiphop configuration file. DynamicExtensionPath = /path/to/hhvm/extensions DynamicExtensions { * = pgsql.so } I placed the snippet they provide (above) into a file called config.hdf in my Hiphop location.  I'm using Mint on my dev box so this was /etc/hhvm/config.hdf for me. Then I edited /etc/default/hhvm and set Hiphop up to...

I've just set up a new nginx host and was having problems with visiting it. Whenever I loaded the page it would download a copy of my index file. In my case the problem was two-fold.  Firstly I had mistyped my server name so it was falling back to an alternative catchall. Secondly Chromium has a "won't fix" bug whereby redirects are cached.  Interestingly the cache also persisted into Incognito mode. Clearing my cache with the "Clear Browsing History" menu function didn't work to clear out the redirect cache.  This gave the symptom of stopping nginx, stopping varnish, and even stopping hhvm but still having Chromium download the file when I tried to visit it. What did work was to affix a get variable to the link.  So instead of visiting http://mysite.local I tried http://mysite.local?foo=1 which invalidated the cache. This wasn't an entirely satisfactory solution so the next thing I tried was to use the developer tools (ctrl shift I) a...

Apache on Centos ships with mod_deflate installed and enabled by default.  To check this you can grep your config file and make sure the line which loads it is not commented out. cat /etc/httpd/conf/httpd.conf | grep LoadModule deflate_module When Apache loads it reads all the config files (ending in .conf) in /etc/httpd/conf.d so we'll add configuration options for mod_deflate into this directory. Lets use a file called deflate.conf to specify the config: <IfModule mod_deflate.c> AddOutputFilterByType DEFLATE text/plain AddOutputFilterByType DEFLATE text/html AddOutputFilterByType DEFLATE text/xml AddOutputFilterByType DEFLATE text/css AddOutputFilterByType DEFLATE text/javascript AddOutputFilterByType DEFLATE image/svg+xml AddOutputFilterByType DEFLATE image/x-icon AddOutputFilterByType DEFLATE application/xml AddOutputFilterByType DEFLATE application/xhtml+xml AddOutputFilterByType DEFLATE application/rss+xml AddOutput...

PHP Framework popularity as at 2013 - Sitepoint After having a very good experience with using Varnish to cache a Wordpress site we decided to look at caching Laravel. Laravel always generates cookies regardless of whether a person is logged in or not.  This interferes with Varnish which by default will pass all requests with a cookie to the backend and skip the cache. In our particular case our site supported the ability for users to login and would then present them with custom content.  This means that cookies are not restricted to a particular path so we can't discard cookies based on the request as we did for Wordpress when discarding everything except /wp-admin/* requests. My solution was to use a package called session-monster ( Packagist  ) which sets a response header if the data in the Laravel session can be ignored.  Varnish can detect this header and prevent the cookie from being set since we don't really need it.  This together with t...

I recently was asked to investigate speeding up one of the Wordpress sites of a fairly large government organization in Britain.  A large part of my investigation focused on the server stack because I felt that we could get more out of the hardware that was provisioned for us. I decided to set up a stack on my development machine to see how it would work and if it was feasible.  I settled on nginX with hiphop and a Varnish frontend cache.  I realize that nginX would be just fine as the cache and server but in this particular case it would not be possible to replace Apache with nginx on the live server.  I also wanted to experiment with ESI and it looked better documented in Varnish than nginx. Installing HHVM is very easy: wget -O - http://dl.hhvm.com/conf/hhvm.gpg.key | apt-key add - echo deb http://dl.hhvm.com/ubuntu saucy main | tee /etc/apt/sources.list.d/hhvm.list apt-get update apt-get install hhvm Installing nginx is also very easy:...

I got this error even though I had an smsbox defined.  Unfortunately I had forgotten to create the smsbox-route group, so this is a very quick fix: #--------------------------------------------- ## SMS ROUTING ##--------------------------------------------- group = smsbox-route smsbox-id = mainbox smsc-id = ztemodem-smsc-group sim-buffering = false ... continues ... Just make sure the id's matchup, so my smsbox group begins with this: #--------------------------------------------- ## SMS BOX ##--------------------------------------------- group = smsbox smsbox-id = mainbox ... continues ... And my SMSC looks like this: #--------------------------------------------- ## GSM MODEM SMSC ##--------------------------------------------- group = smsc smsc-id = ztemodem-smsc-group ... continues ... This successfully cleared the "smsbox_list empty" error and allowed messages to be delivered properly. Kannel had until then been able to send messages and receive delivery...

Our office (who wants me to mention how awesome they are) had a need for a means to send/receive automated sms messages relating to online banking. We decided to give a Raspberry Pi the job and so I have had the fun of setting it all up. First off I installed Rasbian (  http://www.raspbian.org/  ) which is a Debian fork and is sufficiently easy to do following the guide on eLinux ( here  ) which is linked to from the Raspberry Pi project website downloads page. Next came a standard LAMP stack install.  I needed to have a web-server running because we will be running a management package on the box to track messages etc. Next came Kannel.  I used the default package available from the Rasbian repositories so there was no need for fiddling. My kannel.conf file was the next hurdle.  For the most part this is pretty easy to set up from the examples given on Kannel's site and elsewhere on the web.  I want to just mention the modem part, which was pe...