Get A Sys Admin

A sysadmin blog about Linux and cloud IaaS

Apache2 worker vs prefork for ISPConfig benchmark

I’ve been running ISPConfig latest version(3.0.4) on Amazon cloud t1.micro instance for some time to host several small sites, mostly WordPress. I’m quite happy with the performance of the instance. The OS is Ubuntu 10.04 LTS. Until recently I’ve used the default mpm which is prefork, but I decided to test out worker also. If you are wondering I use mod_fcgid for all the sites. That being said I performed several tests with ab (apache benchmark) to see which mpm can server most requests per second.

While I do not claim this is the best setup, I think worker is better suited for me. Some people said they had problems because of mpm worker. So far so good, but will update this post if there are any issues.

Test results:

prefork worker
Concurrency Level: 32
Time taken for tests: 7.834 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 4972
Total transferred: 84831033 bytes
HTML transferred: 83206915 bytes
Requests per second: 638.27 [#/sec] (mean)
Time per request: 50.136 [ms] (mean)
Time per request: 1.567 [ms] (mean, across all concurrent requests)
Transfer rate: 10575.21 [Kbytes/sec] received
Concurrency Level: 32
Time taken for tests: 7.096 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 4968
Total transferred: 84877824 bytes
HTML transferred: 83247322 bytes
Requests per second: 704.63 [#/sec] (mean)
Time per request: 45.414 [ms] (mean)
Time per request: 1.419 [ms] (mean, across all concurrent requests)
Transfer rate: 11681.17 [Kbytes/sec] received

Creating consistent backups for EBS with EXT4 and quota

What’s this about?
Data security and backups are very important aspects when you work with servers, especially if you are using cloud infrastructure. I am using AWS(Amazon Web Services) as my preferred IaaS, so the following how-to is tailored for Amazon EC2 instances using EBS as storage for the web sites files. On my instance I have Ubuntu 10.04 LTS installed and on top of it I run ISPConfig 3.0.4(latest version at the moment I write this article). Some of the programs required to run this setup were already installed, but it should be pretty obvious if you miss anything. If you need help you can either leave a comment or contact me via email.

The following setup will allow you to create an EBS using EXT4 as file system, with quota enabled on it(for ISPConfig) and weekly backups of the EBS. In case of instance failure you should be able to launch a new instance and attach the EBS, without losing any web sites files. In case of EBS failure you can recreate one from the most recent snapshot.

Create an EBS in the same zone as your instance and attach it to your instance as /dev/sdf. This can be easily done from AWS Management Console.

Install xfsprogs
sudo apt-get install xfsprogs

Create EXT4 filesystem on /dev/sdf
sudo mkfs.ext4 /dev/sdf

Now mount it temporarily
sudo mkdir /mnt/ebs
sudo mount /dev/sdf /mnt/ebs

Stop the apache2 web server and copy the files to /mnt/ebs
sudo service apache2 stop
cd /mnt/ebs
sudo cp -rp /var/www/* .

Prepare quota
touch quota.user quota.group
sudo chmod 600 quota.*

Add the entry to /etc/fstab
/dev/sdf /var/www ext4 noatime,nobootwait,usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv0 0 0

Unmount the EBS and remount it to /var/www
sudo umount /dev/sdf
sudo mount /dev/sdf /var/www -o noatime,usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv0

Enable quota
sudo quotacheck -avugm
sudo quotaon -avug

Start the apache2 web server and check that the web sites are working properly
sudo service apache2 start

Install ec2-consistent-snapshot script for weekly backups of EBS
sudo add-apt-repository ppa:alestic
sudo apt-get update
sudo apt-get install -y ec2-consistent-snapshot

Prepare first snapshot(I assume the cron will run as root user, hence I create the awssecret file in /root directory)
sudo touch /root/.awssecret
sudo chmod 600 /root/.awssecret

Edit .awssecret and add following lines, in this order, replacing ACCESS_KEY_ID and SECRET_ACCESS_KEY with your own, both can be found under Account->Security Credentials:
ACCESS_KEY_ID
SECRET_ACCESS_KEY

Test the snapshot creation with debug mode activated, replace VOLUME_ID with the right volume ID:
sudo ec2-consistent-snapshot --debug --description "snapshot $(date +\%Y-\%m-\%d-\%H:\%M:\%S)" --freeze-filesystem /var/www vol-VOLUME_ID

If everything went well you should be able to see your new snapshot in the AWS Management Console.

Finally add this to your root crontab (by running sudo crontab -e):
@weekly /usr/bin/ec2-consistent-snapshot --debug --description "snapshot $(date +'%Y-%m-%d %H:%M:%S')" --freeze-filesystem /var/www vol-VOLUME_ID>>/var/log/backup.log 2>&1
Make sure you put the correct VOLUME_ID!

This should be all, you now have all your web sites on EBS, quota is enabled and weekly backups enabled. I think I pretty much nailed everything you need in order to perform this setup, but if there are any issues feel free to leave a comment. Also I love getting feedback so if you found this article useful leave a comment also :)

Install ffmpeg on Ubuntu 10.04

Note: These are my instructions for installing ffmpeg on ubuntu linux 10.04 server (lts). Most of the code here can be found on ubuntu forums also and probably you will find more stuff there.

Install requisite packages
sudo apt-get update
sudo apt-get install build-essential git-core checkinstall texi2html libopencore-amrnb-dev libopencore-amrwb-dev libsdl1.2-dev libtheora-dev libvorbis-dev libx11-dev libxfixes-dev zlib1g-dev automake autoconf libxvidcore-dev

Install latest version of yasm
cd
git clone git://github.com/yasm/yasm.git
cd yasm
sh autogen.sh
make
sudo checkinstall --pkgname=yasm --pkgversion="1.1.0" --backup=no --deldoc=yes --default

Install x264
cd
git clone git://git.videolan.org/x264
cd x264
./configure
make
sudo checkinstall --pkgname=x264 --pkgversion "1:0.svn`date +%Y%m%d`-0.0ubuntu1" --backup=no --deldoc=yes --fstrans=no --install=yes --default

Install LAME for mp3 support
cd
sudo apt-get install nasm
wget http://downloads.sourceforge.net/project/lame/lame/3.99/lame-3.99.tar.gz
tar xzvf lame-3.99.tar.gz
cd lame-3.99
./configure
make
sudo checkinstall --pkgname=lame-ffmpeg --pkgversion="3.99" --backup=no --default --deldoc=yes

Install opencore-amr for amr support
cd
wget http://downloads.sourceforge.net/project/opencore-amr/vo-amrwbenc/vo-amrwbenc-0.1.1.tar.gz
tar zxvf vo-amrwbenc-0.1.1.tar.gz
cd vo-amrwbenc-0.1.1
./configure --disable-shared
make
sudo checkinstall --pkgname="libopencore-amr" --pkgversion="0.1.1" --backup=no --fstrans=no --install=yes --default

Install libtheora for ogg support
cd
wget http://downloads.xiph.org/releases/theora/libtheora-1.1.1.tar.bz2
tar jxvf libtheora-1.1.1.tar.bz2
cd libtheora-1.1.1
./configure --disable-shared
make
sudo checkinstall --pkgname=libtheora --pkgversion "1.1.1" --backup=no --fstrans=no --install=yes --default

Install faac
cd
sudo apt-get install unzip
wget http://downloads.sourceforge.net/faac/faac-1.28.tar.gz
tar zxvf faac-1.28.zip
cd faac-1.28
wget http://www.linuxfromscratch.org/patches/blfs/svn/faac-1.28-glibc_fixes-1.patch
patch -Np1 -i faac-1.28-glibc_fixes-1.patch
sed -i -e '/obj-type/d' -e '/Long Term/d' frontend/main.c
make
sudo checkinstall --pkgname=libfaac --pkgversion "1.28" --backup=no --fstrans=no --install=yes --default

Install FFmpeg

svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg
./configure --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-x11grab
make
sudo checkinstall --pkgname=ffmpeg --pkgversion "0.8.5" --backup=no --fstrans=no --install=yes --default

Amazon RDS SUPER privileges

#1419 – You do not have the SUPER privilege and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable

This error occurs sometimes on RDS instances when you try to use procedures. You will soon find out that grant super privilege for a user won’t work. So the only way to make things work is to set log_bin_trust_function_creators to 1.

RDS console available at https://console.aws.amazon.com/rds/ allows you to create a new group and modify its parameters. Log in to RDS console, go to “DB Parameters Groups” and click the “Create DB Parameter Group”. Set the following
- DB Parameter Group Family: mysql5.1
- DB Parameter Group Name: mygroup
- Description: mygroup

Confirm by clicking “Yes, create” button.

Here comes the ugly part, since you cannot edit from the console the parameters (for the moment, I hope they are going to change that). You will need to log to your instance using SSH and download RDS cli from here: http://aws.amazon.com/developertools/2928?_encoding=UTF8&jiveRedirect=1

To do so right click on “Download” button and copy link location. In the SSH window use wget to download and unzip it:
wget “http://s3.amazonaws.com/rds-downloads/RDSCli.zip”
unzip RDSCli.zip

If you don’t have unzip you can quickly get it using “apt-get install unzip”(for ubuntu) or “yum install unzip”(for centos). Of course you will need root privileges.

After successfully unpacking the RDSCli cd to that directory and set a few variables. Following is an example on Ubuntu 10.04:
cd RDSCli-1.4.006
export AWS_RDS_HOME=”/home/ubuntu/RDSCli-1.4.006″
export JAVA_HOME=”/usr/lib/jvm/java-6-sun”
cd bin
./rds –help

If rds –help outputs no errors then you have set it correctly. Congrats. One more command:
./rds-modify-db-parameter-group mygroup –parameters=”name=log_bin_trust_function_creators, value=on, method=immediate” –I=”YOUR_AWS_ACCESS_KEY_ID” –S=”YOUR_AWS_SECRET_ACCESS_KEY”

The AWS keys can be obtain from your AWS account Security Credentials->Access Credentials->Access Keys.

Go to AWS RDS console, “DB Instances”, select your instance and right click “Modify”. Set “DB Parameter group” to “mygroup” and check “Apply Immediately”. Confirm with “Yes, modify”.

You are done :)

Mysql benchmark: RDS vs EC2 performance

the setup: 1 m1.small ec2 instance vs 1 db.m1.small rds instance, tests are being run from the m1.small instance. The goal is to determine how the site will perform when moving the database from localhost to a remote instance.

I used sysbench for mysql benchmarks. On a linux server running ubuntu 10.04 you can simply install it with the following command(it’s obvious but just in case):
sudo apt-get install sysbench

The first tests performed were m1.small EC2 instance running mysql-server 5.1.41-3ubuntu12.8 VS RDS instance type db.m1.small running mysql server 5.1.50. The test database had been set to 10 000 records, number of threads = 1, test oltp.
sysbench --test=oltp --mysql-host=smalltest.us-east-1.rds.amazonaws.com --mysql-user=root --mysql-password=password --max-time=180 --max-requests=0 prepare
sysbench --test=oltp --mysql-host=smalltest.us-east-1.rds.amazonaws.com --mysql-user=root --mysql-password=password --max-time=180 --max-requests=0 run

The results

m1.small EC2 instance db.m1.small RDS instance
OLTP test statistics:
queries performed:
read: 263354
write: 94055
other: 37622
total: 395031
transactions: 18811 (104.50 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 357409 (1985.56 per sec.)
other operations: 37622 (209.01 per sec.)
Test execution summary:
total time: 180.0044s
total number of events: 18811
total time taken by event execution: 179.7827
per-request statistics:
min: 4.04ms
avg: 9.56ms
max: 616.04ms
approx. 95 percentile: 38.42ms
OLTP test statistics:
queries performed:
read: 188230
write: 67225
other: 26890
total: 282345
transactions: 13445 (74.67 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 255455 (1418.74 per sec.)
other operations: 26890 (149.34 per sec.)
Test execution summary:
total time: 180.0573s
total number of events: 13445
total time taken by event execution: 179.9174
per-request statistics:
min: 9.08ms
avg: 13.38ms
max: 904.58ms
approx. 95 percentile: 20.99ms

As you can see the EC2 can perform 40% more transactions than the RDS instance. Nothing unexpected so far.

Time to move on and increase the number of threads to 10

m1.small EC2 instance db.m1.small RDS instance
OLTP test statistics:
queries performed:
read: 264866
write: 94545
other: 37818
total: 397229
transactions: 18899 (104.97 per sec.)
deadlocks: 20 (0.11 per sec.)
read/write requests: 359411 (1996.22 per sec.)
other operations: 37818 (210.05 per sec.)

Test execution summary:
total time: 180.0462s
total number of events: 18899
total time taken by event execution: 1799.9289
per-request statistics:
min: 4.08ms
avg: 95.24ms
max: 2620.70ms
approx. 95 percentile: 445.91ms

OLTP test statistics:
queries performed:
read: 343812
write: 122772
other: 49109
total: 515693
transactions: 24551 (136.18 per sec.)
deadlocks: 7 (0.04 per sec.)
read/write requests: 466584 (2588.13 per sec.)
other operations: 49109 (272.41 per sec.)

Test execution summary:
total time: 180.2788s
total number of events: 24551
total time taken by event execution: 1801.8298
per-request statistics:
min: 13.41ms
avg: 73.39ms
max: 1126.02ms
approx. 95 percentile: 143.83ms

In this test the small RDS instance is faster than the EC2, 136 vs 105 transactions per second. I’ve also benchmarked a large RDS instance (the next one available after db.m1.small) and it got 185 transactions per second. Quite good, but the price is 4x higher.

The next test was performed vs a 10 million records, 16 threads. This time I only benchmarked a small and a large RDS instance. The large instance managed to do 228 transactions per second while the small one got a decent score of 127 transactions. One thing I noticed during this test is that the small instance started to use it’s swap, while the large one did not have this issue. This is probably due to the fact that 10M records db is aprox 2.5GB and the small RDS only has 1.7GB of RAM.

So if you are planing to grow and want an easy way to do it, switching your database to its own RDS is one of the first things you should consider. One of the immediate effects you will notice is that the CPU usage on the EC2 instance will be greatly reduced, leaving more power for the web server. You can easily increase the size and capacity of the RDS instance with just a few clicks. The backups are done automatically, which is great considering how many times I had to recover databases.

ISPConfig3 running on Ubuntu 9.10 on AWS

This is an idea on how to set ISPConfig 3 on Amazon EC2 for web hosting. It’s still work in progress and I await for any constructive feedback. Since English is not my first language there might be some grammatical errors.

Prerequisites:

  • AWS Account
  • Some Linux experience

Part I – Setting up AWS

I have done all the setup of instance and ebs from the Amazon Management Console. It has a very intuitive interface and it’s so easy to work with.

Optional: You can purchase reservation for an instance if you are planing to run it 24/7. To do so go to Instances -> Reserved Instances and click on Purchase Reserved Instance. For example a c1.medium instance reservation for 1 year costs 455$ one time fee and the hourly cost of the instance goes to 0.06$/hour. It’s your decision, do the math if it’s worth it or not to go with reserved instances.

  1. From the Management Console go to Volumes and Create Volume. I created a 40 GB volume in us-east-1a zone, choose an appropriate value for your needs.
  2. Next step which is also optional is to get an Elastic IP for your instance. Go to Elastic IPs and click Allocate New Address.
  3. You will need a keypair to access your instance. You can create one before launching the instance or create one when you also setup the new instance. Your choice. I created one before by going to Key Pairs -> Create Key Pair.

Part II – Launching the instance

Instance type used was c1.medium, the AMI for it was Ubuntu 9.10 Karmic 32bit ami-1515f67c from Canonical Images for Amazon EC2.

Go to Instances and press Launch Instance and make sure you set it to use ami-1515f67c. If you need to use a different zone or a 64 bit instance you can check the alestic page which is a wonderful resource for Ubuntu images for EC2. I would also suggest to join their EC2 Ubuntu Google Grup if you are serious about AWS + Ubuntu.

After the instance is up and running attach the Elastic IP and the Volume you have created to it. To attach the IP go to Elastic IPs and select Associate, for the EBS go to Volumes. I attached the EBS as /dev/sdb to the instance.

Later Edit: You will need to also modify the Security Groups and permit access on ports 22(SSH), 80(HTTP), 8080(ISPConfig 3), 20-21(FTP) -> still need more work to support PASV.

Part III – Install ISPConfig 3

Before proceeding with ISPConfig 3 setup connect to your instance and update the packages.

sudo su
apt-get update
apt-get upgrade

Note: You will have to connect to your instance using the ubuntu username and the key pair you generated for that instance. It won’t allow you to connect directly as root.

To install the ISPConfig 3 I followed the HowtoForge excellent tutorial with a small exception: I skipped the part of setting up the quota since I was planning to use the EBS as storage for sites and MySQL databases. The rest of the tutorial was just perfect.

Part IV – Setting up the EBS and ISPConfig to work with it

Most of the stuff and talk about EBS and MySQL snapshots is discussed on Amazon developer forums in the thread called Tutorial: Running MySQL on Amazon EC2 with EBS (Elastic Block Store). Make sure you read the discussion before or after you are done with this post.
apt-get install dmsetup
modprobe dm_mod
modprobe dm_mirror
modprobe dm_snapshot
mkdir /dev/.static/dev/mapper -pv

Create ext3 filesystem:

mkfs.ext3 /dev/sdb

Setup it with dmsetup

echo 0 `blockdev --getsize /dev/sdb` linear /dev/sdb 0 | dmsetup create ebs

Add the records to fstab

echo "/dev/mapper/ebs /ebs ext3 noatime,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 0 0" >> /etc/fstab

Setup quota

touch /ebs/aquota.user /ebs/aquota.group
chmod 600 /ebs/aquota.*
mount -o remount /ebs

Enable quota on ebs

quotacheck -avugm -F vfsv0 /ebs
quotaon -avug /ebs

For websites:

mkdir /ebs/sites
ln -s /ebs/sites /var/www/clients

Jailkit:

Set Jailkit chroot home: /ebs/home/[username]

mkdir /ebs/home

MySQL:

/etc/init.d/mysql stop
mkdir /ebs/mysql/data /ebs/mysql/log -pv
mv /var/lib/mysql /ebs/mysql/data
ln -s /ebs/mysql/data/mysql /var/lib/mysql
mv /var/log/mysql /ebs/mysql/log
ln -s /ebs/mysql/log/mysql /var/log/mysql

To make a snapshot of the database, go to the mysql console (msqyl -p)

FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;
SYSTEM dmsetup suspend /dev/mapper/ebs

Take your snapshot of the EBS

SYSTEM dmsetup resume /dev/mapper/ebs
UNLOCK TABLES;

Note:This is still work in progress, any advices on how to improve it are greatly appreciated and any constructive criticism also.

Updating to Ubuntu Intrepid Ibex

Last weekend I’ve decided to update my netbook, an MSI Wind U100 clone labeled Advent, to the latest Ubuntu version (Intrepid Ibex). While on theory everything should have been simple, the reality is I had a ton of problems.

So lets start with updating from Ubuntu 8.04 to Ibex. First you should backup your home directory to an USB stick. Also copy xorg.conf to a safe place in case things go wrong.

Start a terminal in graphic mode (press alt+f2 and type xterm). Become root:

sudo su

Type your password and when you get the root prompt (#) type:

update-manager -d

You should get a new window like this one

Press “Upgrade” and answer the questions. This was the easy part, after update-manager would have finished running I should have been a happy user running Ubuntu Intrepid, right? Wrong!

I went out to town to drink some beer since it was weekend and left the update-manager to do its job. When I got back home, surprise: I didn’t plug the netbook to a power source and battery was empty, netbook offline. Nice … Pressed the power on button and crossed fingers.

After 3-4 minutes the reality unveiled, upgrade didn’t finished OK so netbook wasn’t entering graphic mode and I’ve got a ton of failed services and errors. A quick look showed that the file system was mounted read-only. Time to repair all that mess as I didn’t want to reinstall the whole system because I’m a lazy person.

Recovering after disaster

Ctr+Alt+F1 to get to first text console. Login with your username and password. Become root:

sudo su

As root remount the file system so it can be used:

mount / -o remount

If you have more partitions do so for each of them (replace / with their mount point).
Switch to run level 3:

telinit 3

When you are running in run level 3 type:

dpkg –configure -a

This should restart/fix the upgrade process from where it stopped. It will ask you if you want to replace your custom config files with new ones. I answered No. When it’s done restart the netbook (use telinit 6). Make it shutdown even if you have to switch it off from power button.

When it’s online again you should have a working graphic mode. If not, try to replace your xorg.conf with your backup and restart gdm with:

/etc/init.d/gdm restart

Probably you won’t be having the wireless working so plug a network cable and manually reconfigure your NIC:

sudo ifconfig eth0 192.168.1.113/24
sudo route add default gw 192.168.1.1

In the above example 192.168.1.113 is netbook IP address, /24 is the netmask (equivalent to 255.255.255.0) and 192.168.1.1 is the gateway (the IP of router)

Run the following commands as root to fix all the missing dependencies:

apt-get update
apt-get -f install
apt-get upgrade

Restart the system and everything should be ok now.