Encrypted EBS and OpsWorks

I was trying to attach an Encrypted EBS volume to an instance managed with OpsWorks. But after registering the EBS volume to OpsWorks stack and setting up the mount point to /srv/backups the instance refused to start with a message about not all volumes being mounted.

I suspected immediately it was a problem with the encryption and after googling around found this useful answer on AWS forum: https://forums.aws.amazon.com/thread.jspa?messageID=697774

So I went back to IAM and created a new Encryption key from the interface and gave both aws-opsworks-service-role and aws-opsworks-ec2-role access to that key. The weird thing is that the EBS was encrypted with default aws/ebs key, but somehow OpsWorks still managed to start the instance and mount the encrypted EBS.

Why I don’t like Kaspersky Endpoint Security

kaspersky endpoint security

I’ve been using Kaspersky Endpoint Security 10 for (almost) one year. What started as a great experience soon turned out to a nightmare as I began to use (or better tried to use) certain features.

  1. When you create a task to remotely deploy KES10 to a workstation it will pack both 32bit and 64bit. This is about 400MB. Good luck deploying that over Internet
  2. Even if the destination workstation is running 64bit OS, Kaspersky will install the 32bit version.
  3. Trying to deploy to multiple machines at the same time fails most of the times
  4. The KES10 installer fails to remove existing AV solutions like Microsoft Security Essentials, so the installation will fail. I had to manually remove MSE before running the install of KES remotely
  5. Deploying to a workstation takes a long time even if it’s in the same building
  6. I click check connection on a workstation. It says it’s available. I try to install something (a security update). I force synchronization and then check the status. It says the workstation is unavailable
  7. Kaspersky Security Center (the management console) will (wrongly) report that computers have not been scanned for a long time, even if this is not true. This is a known bug, and even using the patch provided by Kaspersky support didn’t fix it
  8. Network agent service will stop on some workstations. Why? Who knows.
  9. Contacting support? Get ready to wait half an hour before someone will take your call
  10. Trying to patch software vulnerabilities? Technically possible from Kaspersky Security Center. Reality is that the patch will fail with cryptic messages, even if you accept license agreements and stuff

I think there are more, but these are just a few that I can remember now. When we’ve purchased the licenses for KES10 and installed the management console I was expecting it will help me with workstation protection and management. Turned out I am spending more time trying to figure out why a workstation’s status is critical, why it hasn’t been connected to the administration server for several days, why the AV is not running, how to set policy so that programs will work correctly.

I am looking for other AV solutions for business as Kaspersky Endpoint Security 10 failed to deliver. If you have any suggestions for AV solutions for business that you are using please leave a comment.

Error code 1603 when installing Office 2010 on Windows XP

From time to time less technical friends ask me to reinstall their Windows and applications. Usually this takes a couple of hours, but the last time I did this things went completely wrong with Microsoft Office 2010 Pro. The OS was a Windows XP SP3, all security updates with MSE as antivirus solution. After starting the setup the installer failed with a very helpful message “something went wrong”. Thanks a lot Microsoft!

Tried with Run as Administrator. Same thing, failed within seconds after I’ve clicked install. I quickly begin searching the net for Office install issues. Found that the log for the installer is located in the temp directory. To get there simply go to My Computer and in the address bar type %temp%. Look for files called setup*. After reading through the log files found that I had an ErrorCode 1603. Nice thing, very helpful. Search again for 1603, read countless threads on Microsoft site. Tons of bullshits, unrelated answers, you know the usual Microsoft support experience: you did something wrong, uninstall and install again and hope it will magically work this time. It didn’t work! I tried setting the security permissions on files to allow SYSTEM to modify anything on C:, ran as Administrator, got another copy of Office 2010, installed Windows Search 4.0.

The solution I’ve found from a friend of mine:
1) Delete everything in %temp%
2) Go to %appdata% and delete everything from Microsoft/Office
3) Delete everything from C:\Program Files\Microsoft Office
4) C:\Documents and Settings\All Users\Application Data delete everything related to Office
5) regedit and delete the keys for Office from HKEY_LOCAL_MACHINE\Software and HKEY_CURRENT_USER\Software
6) Finally start the installer as Administrator

Hopefully it will work for you, it was a very frustrating experience for me!

Rackspace vs Amazon impressions

I’ve been a customer of Amazon Web Services since they have been in beta. I’ve also worked with Rackspace now and then. Lately I’ve been working extensively with Rackspace infrastructure so I have a better understanding of their products. Bellow there are a few things that bother me at Rackspace:


The thing you are going to use 99% of the time with any of the providers. Amazon offers normal instances, high cpu instances, high memory instances, huge memory and cpu instances. On Rackspace you will have a very limited offer of instances. No high cpu or high memory. They do have plans to add them in nearby future. Until then, if you have a CPU bottleneck then it’s tough luck: scale up the instance and pay double.

And since we are talking about pricing for instances, Amazon is cheaper on the long run due to Reserved Instances (you pay an upfront fee and lower price/hour for the instance after). Amazon has been cutting down prices several times. Rackspace? Never.

Block storage

Called EBS on Amazon or CBS on Rackspace, it’s the preferred way of adding extra space to running instances without upgrading them. Beware that on Rackspace it takes an horrible amount of time to do a snapshot. I had to contact RS support several times because of this.

MySQL instances

Called RDS on Amazon or Cloud Database on Rackspace, they are basically optimized instances that run MySQL, but you are not given SSH access to them. Here Rackspace has the worst offer:
no scheduled backups (RS recommends mysqldump … lol try to do mysqldump for 10GB+ of data)
no replication (really? I couldn’t replicate my database)
no hotspare (Amazon MultiA-Z)

The good thing about Cloud Database is that they are fast. Really fast. Also RS promised to address all the above things in the nearby future (yeah).

Load Balancing

Rackspace does not support LB inside private network. Come on RackSpace, do your homework. This thing has been working for AWS since ancient history.


There are many other things that need to be discovered, probably not pleasant ones. So far I am disappointed with Rackspace and what it has to offer and I would recommend to any customer to use Amazon instead.

Installing memcached with repcached patch for HA memcache cluster


Repcached is an interesting patch for memcached which allows replication between 2 memcached nodes (servers). The purpose of this article is to setup 2 memcached servers that replicate each others.

Note: This article is specifically written for Ubuntu 12.04 and memcached version 1.4.13. It may or it may not work for other versions.


2x 512MB RackSpace instances called node01(IP and node02(IP

Patch for memcached v1.4.13:


Prepare to build the package:

apt-get build-dep memcached
apt-get source memcached
cd memcached-1.4.13
wget https://github.com/usecide/repcached/blob/master/repcached-2.3.1-1.4.13.patch
patch -p1 -i repcached-2.3.1-1.4.13.patch

Now edit the file debian/rules and look for config.status and add –enable-replication like this:

config.status: configure
        CFLAGS="$(CFLAGS)" ./configure --host=$(DEB_HOST_GNU_TYPE) \
                                       --build=$(DEB_BUILD_GNU_TYPE) \
                                       --prefix=/usr \
                                       --mandir=\$${prefix}/share/man \
                                       --enable-replication \

Now build the package:

dpkg-buildpackage -us -uc -nc
cd ..

You should see a package named memcached_1.4.13-0ubuntu2_amd64.deb. Copy this file on both of your memcached servers and install it using this command:

dpkg -i memcached_1.4.13-0ubuntu2_amd64.deb


You are almost done now. Kill all running memcached processes on both nodes:

killall -9 memcached
ps aux | grep memcached

On node01 do the following things:

cp /etc/memcached.conf /etc/memcached_server1.conf

Edit /etc/memcached_server1.conf and replace the line:




and add at the end:


Start memcached:

service memcached start

On node02 do something similar:

cp /etc/memcached.conf /etc/memcached_server2.conf

Edit /etc/memcached_server2.conf and replace the line:




and add at the end:


Start memcached:

service memcached start


From node01:

telnet 11211:
Escape character is '^]'.
get hello
set hello 0 0 4
get hello
VALUE hello 0 4

From node02:

telnet 11211:
Escape character is '^]'.
get hello
VALUE hello 0 4

How to setup Galera 3 node cluster on Ubuntu 12.04

Galera is a multi-master replication solution for MySQL, which provides an interesting alternative to the standard master-master MySQL replication we are all so used with. One main advantage of Galera is the ability of doing sync replication, thus reducing the risk of data inconsistency between masters.

Setup on RackSpace Cloud

3x 512MB RAM instances, with 20GB storage space
1x Load Balancer for MySQL, RoundRobin algorithm, Health check enabled
1x 512MB RAM instance for testing
OS: Ubuntu 12.04 LTS 64bit


Quickly setup a Galera cluster and run some benchmarks using sysbench.

Note: For the sake of simplicity I will refer to the Galera instances as node01, node02 and node03. The test instance will be referred as test01.

Common settings on all nodes

On every node execute:

  1. An apt-get update and upgrade to bring the instances up to date.
  2. Install required packages
    apt-get install libaio1 libssl0.9.8 mysql-client libdbd-mysql-perl libdbi-perl
  3. Download Galera wsrep provider
    wget https://launchpad.net/galera/2.x/23.2.4/+download/galera-23.2.4-amd64.deb
    dpkg -i galera-23.2.4-amd64.deb
  4. Download MySQL server with wsrep patch
    wget https://launchpad.net/codership-mysql/5.5/5.5.28-23.7/+download/mysql-server-wsrep-5.5.28-23.7-amd64.deb
    dpkg -i mysql-server-wsrep-5.5.28-23.7-amd64.deb
  5. I had some issues and I had to create /var/log/mysql
    mkdir -pv /var/log/mysql
    chown mysql:mysql -R /var/log/mysql
  6. Secure the mysql installation and assign a good password to root user:
    service mysql restart
  7. Create an user for galera nodes to use for connect/replication
    mysql -p
    mysql> grant all privileges on *.* to galera@'%' identified by 'password';
    Query OK, 0 rows affected (0.00 sec)
    mysql> flush privileges;
    Query OK, 0 rows affected (0.00 sec)
    mysql> set global max_connect_errors = 10000;
    Query OK, 0 rows affected (0.01 sec)
  8. Edit /etc/hosts and make sure you add all the nodes and their corresponding IPs

Galera setup for each node

Edit the /etc/mysql/conf.d/wsrep.cnf and change the values for the following variables:

Configuration for node01:


Configuration for node02:


Configuration for node03:


Testing the setup

Now restart mysql on all the nodes and check out if cluster is working:

service mysql restart
mysql -p
mysql> show status like 'wsrep%';
| Variable_name | Value |
| wsrep_cluster_size | 3 |
| wsrep_ready | ON |

One more thing before you are done:
Edit node01 wsrep_cluster_address=”gcomm://node3:4567″ and restart mysql server.

Benchmarks were performed from test01 instance using sysbench 0.5 OLTP read-only complex test:

sysbench OLTP (ro) Galera cluster transactions vs threads
Threads Transactions/s
1 15
2 25
4 49
8 103
16 205
32 390
64 506
128 653


sysbench OLTP (ro) Galera cluster avg response time
Threads Avg response time Min response time Aprox 95%
1 66 42 131
2 79 53 135
4 80 42 153
8 77 42 136
16 77 43 143
32 81 42 142
64 125 48 322
128 194 45 427


Benchmark Galera cluster vs MySQL master-master on RackSpace


Before starting this I would like to point out that I have compared 2 instances(master-master) vs 3 instances(galera cluster) so the test is not correct/accurate. It’s more of a “what if I switch from master-master replication to 3 nodes galera”.

MySQL Master-Master replication:

2x 512 MB instances with 20GB of storage, Ubuntu 12.04 64bit, mysql-server 5.5 was used with no optimization changes to my.cnf, except the required changes for master-master replication.
1x LoadBalancer, RoundRobin algorithm

Galera 3 nodes cluster:

3x 512 MB instances with 20GB of storage, Ubuntu 12.04 64bit, mysql-server 5.5 from galera was used, with no changes to my.cnf, only required node changes were made wsrep.cnf.
1x LoadBalancer, RoundRobin algorithm

Test instance:

1x 512MB instance with 20GB of storage, Ubuntu 12.04 64bit running sysbench

sysbench --test=oltp --mysql-host=loadbalancer_ip --mysql-user=root --mysql-password=password--oltp-table-size=1000000 prepare

The tests were performed on a database of about 256MB size, InnoDB table(s). No optimization changes were made to default my.cnf files, except the required to setup replication.

sysbench OLTP transactions per second
Test Master-Master Single node Galera cluster
1 thread,3m 10.97 17.11 12
16 threads,1m, rw 154 140 0
16 threads,1m, r only 217 158.7 206
32 threads,1m, r only 325 160.79 375


As you can see from the table and graph I had some issues performing sysbench for Galera cluster in rw mode for 16 threads. From what I have found on Internet it’s an issue with sysbench 0.4.12 so I will attempt to rerun the tests with a newer version.

Installing Scalr 3.5 Open Source on Ubuntu 12.04

This is an update on an older post of mine, one of my first articles regarding cloud computing. Much has changed since since 2008 when I have wrote this article “How to install Scalr on Ubuntu 8.10 EC2 Instance“.

For example the Ubuntu has evolved to 12.04 LTS (I am using LTS 64bit for this howto) and Scalr is now version 3.5. One thing didn’t change: it’s still a royal PITA to get Scalr open source working. Hopefully this howto will help you to install Scalr on your server. It doesn’t cover operating Scalr and other things, which I will address in future posts, if there is enough interest.

After you have installed Ubuntu 12.04 64bit server edition to your server or virtual machine the way you like it it’s time to start the update process:

apt-get update && apt-get upgrade

Now you are ready to run tasksel and select the following roles for your server: OpenSSH, DNS server, LAMP server

You will need to install some dev packages before going anything else:

apt-get install libcurl4-gnutls-dev make librrd-dev

Now it’s time for PHP5 related extensions:

apt-get install php5-curl php-gettext php-net-socket php5-mcrypt php-xml-serializer libssh2-php php-soap php5-snmp php5-rrd
pecl install pecl_http
echo "extension=http.so" >/etc/php5/conf.d/pecl_http.ini
pecl install rrd
echo "extension=rrd.so" >/etc/php5/conf.d/rrd.ini

Time to get Scalr code:

cd /tmp
wget http://bit.ly/scalr35
tar zxvf scalr35
cd scalr-3.5.r7704
cp -r app /var/www/
chown -R www-data:www-data /var/www/app

Create new database and import sql from sql/scalr:

mysql -p
mysql> CREATE DATABASE scalr CHARACTER SET latin1 COLLATE latin1_swedish_ci;
mysql> grant all privileges on scalr.* to scalr@localhost identified by 'password';
mysql> flush privileges;
mysql> quit
mysql -p scalr <sql/scalr.sql

While doing that import I’ve got a nice error:
ERROR 1054 (42S22) at line 2222: Unknown column ‘architecture’ in ‘field list’
1) Drop database
2) Search sql/scalr.sql for “CREATE TABLE IF NOT EXISTS `role_images`” and add after platform:

`architecture` varchar(25) DEFAULT NULL,
`os_family` varchar(25) DEFAULT NULL,
`os_name` varchar(25) DEFAULT NULL,
`os_version` varchar(25) DEFAULT NULL,
`agent_version` varchar(25) DEFAULT NULL,


Configuration of Scalr is quite simple:

cd /var/www/app/etc
cp config.ini-sample config.ini
edit config.ini

Cron jobs required by Scalr? Just type crontab -e and add the following lines:

*/2 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --Poller
* * * * * /usr/bin/php -q /var/www/app/cron/cron.php --Scheduler2
*/10 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --MySQLMaintenance
* * * * * /usr/bin/php -q /var/www/app/cron/cron.php --DNSManagerPoll
17 5 * * * /usr/bin/php -q /var/www/app/cron/cron.php --RotateLogs
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --EBSManager
*/20 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --RolesQueue
*/5 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --DbMsrMaintenance
*/2 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --Scaling
*/5 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --DBQueueEvent
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --SzrMessaging
*/4 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --RDSMaintenance
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --BundleTasksManager
* * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --ScalarizrMessaging
* * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --MessagingQueue
*/2 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --DeployManager
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --UsageStatsPoller
* * * * * root /usr/bin/php -q /var/www/app/cron-ng/cron.php --SNMPStatsPoller

Time to add a Virtual Host:

cat <<EOF> /etc/apache2/sites-available/scalr
<VirtualHost *:80>
ServerName scalr.example.com
ServerAdmin webmaster@example.com
DocumentRoot "/var/www/app/www"

<Directory "/var/www/app/www">
Options Indexes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all

Enable required Apache modules and site and restart everything:

a2ensite scalr
a2enmod rewrite
service apache2 restart

DNS managed by bind9:

chmod g+w /etc/bind/named.conf
echo 'include "/var/named/etc/namedb/client_zones/zones.include";' >> /etc/bind/named.conf
mkdir -p /var/named/etc/namedb/client_zones
chown root.bind /var/named/etc/namedb/client_zones
chmod 2775 /var/named/etc/namedb/client_zones
echo ' ' > /var/named/etc/namedb/client_zones/zones.include
chown root.bind /var/named/etc/namedb/client_zones/zones.include
chmod g+w /var/named/etc/namedb/client_zones/zones.include

To get rid of nasty AppArmor warnings and errors edit /etc/apparmor.d/usr.sbin.named and add:

/var/named/etc/namedb/client_zones/zones.include rw,

And finish it by restarting AppArmor and bind9:

service apparmor restart
service bind9 restart

Open your browser and go to http://scalr.example.com. Default username/password: admin/admin.

If you have issues or you need more info please feel free to comment 🙂

NRPE errors regarding SSL handshake


CHECK_NRPE: Error - Could not complete SSL handshake.

If you are trying to work with Nagios and setup NRPE, when performing check_nrpe -H hostname you might get the above error. Usually it happens when you have added another IP to the list of allowed hosts and you have added a SPACE after the coma. Just remove the space after the coma so your /etc/nagios/nrpe.cfg contains a line like this:


Fix “Unknown user or password incorrect” for email addresses under ISPConfig

If you are trying to login using either webmail or an email client to your ISPConfig server and all you get is “Unknown user or password incorrect” then it might be an issue with your auth daemon. Check /var/log/mail.log for a line similar to this:

authdaemond: modules="(none)", daemons=0

If you got it then the fix is quite simple. Just add the following lines to /etc/courier/authdaemonrc:


Now restart courier auth daemon:

service courier-authdaemon restart

You should see something similar to this in you mail logs now:

authdaemond: stopping authdaemond children
authdaemond: modules="authmysql", daemons=5
authdaemond: Installing libauthmysql
authdaemond: Installation complete: authmysql