Get A Sys Admin

A sysadmin blog about Linux and cloud IaaS

Tag Archives: mysql

Moved blog to EC2

I just finished moving my blog from Hostgator to my own EC2 t1.micro instance. While Hostgator shared hosting is cheap, running on shared servers is not so great. Had times when server was responding very slow.

I’ve decided to drop the “blog” prefix and keep it simple as http://getasysadmin.com. Hope there won’t be any problems with links.

In the process of moving to new server I changed the database from MyISAM to InnoDB, using the tips provided at step 4 from this post: 5 Essential Steps For Hosting A Scalable WordPress Blog Or Website. I also changed the theme of the blog. This one (wplook) is cleaner and better suited for my blog.

Amazon RDS SUPER privileges

#1419 – You do not have the SUPER privilege and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable

This error occurs sometimes on RDS instances when you try to use procedures. You will soon find out that grant super privilege for a user won’t work. So the only way to make things work is to set log_bin_trust_function_creators to 1.

RDS console available at https://console.aws.amazon.com/rds/ allows you to create a new group and modify its parameters. Log in to RDS console, go to “DB Parameters Groups” and click the “Create DB Parameter Group”. Set the following
- DB Parameter Group Family: mysql5.1
- DB Parameter Group Name: mygroup
- Description: mygroup

Confirm by clicking “Yes, create” button.

Here comes the ugly part, since you cannot edit from the console the parameters (for the moment, I hope they are going to change that). You will need to log to your instance using SSH and download RDS cli from here: http://aws.amazon.com/developertools/2928?_encoding=UTF8&jiveRedirect=1

To do so right click on “Download” button and copy link location. In the SSH window use wget to download and unzip it:
wget “http://s3.amazonaws.com/rds-downloads/RDSCli.zip”
unzip RDSCli.zip

If you don’t have unzip you can quickly get it using “apt-get install unzip”(for ubuntu) or “yum install unzip”(for centos). Of course you will need root privileges.

After successfully unpacking the RDSCli cd to that directory and set a few variables. Following is an example on Ubuntu 10.04:
cd RDSCli-1.4.006
export AWS_RDS_HOME=”/home/ubuntu/RDSCli-1.4.006″
export JAVA_HOME=”/usr/lib/jvm/java-6-sun”
cd bin
./rds –help

If rds –help outputs no errors then you have set it correctly. Congrats. One more command:
./rds-modify-db-parameter-group mygroup –parameters=”name=log_bin_trust_function_creators, value=on, method=immediate” –I=”YOUR_AWS_ACCESS_KEY_ID” –S=”YOUR_AWS_SECRET_ACCESS_KEY”

The AWS keys can be obtain from your AWS account Security Credentials->Access Credentials->Access Keys.

Go to AWS RDS console, “DB Instances”, select your instance and right click “Modify”. Set “DB Parameter group” to “mygroup” and check “Apply Immediately”. Confirm with “Yes, modify”.

You are done :)

Mysql benchmark: RDS vs EC2 performance

the setup: 1 m1.small ec2 instance vs 1 db.m1.small rds instance, tests are being run from the m1.small instance. The goal is to determine how the site will perform when moving the database from localhost to a remote instance.

I used sysbench for mysql benchmarks. On a linux server running ubuntu 10.04 you can simply install it with the following command(it’s obvious but just in case):
sudo apt-get install sysbench

The first tests performed were m1.small EC2 instance running mysql-server 5.1.41-3ubuntu12.8 VS RDS instance type db.m1.small running mysql server 5.1.50. The test database had been set to 10 000 records, number of threads = 1, test oltp.
sysbench --test=oltp --mysql-host=smalltest.us-east-1.rds.amazonaws.com --mysql-user=root --mysql-password=password --max-time=180 --max-requests=0 prepare
sysbench --test=oltp --mysql-host=smalltest.us-east-1.rds.amazonaws.com --mysql-user=root --mysql-password=password --max-time=180 --max-requests=0 run
The results

m1.small EC2 instance db.m1.small RDS instance
OLTP test statistics:
queries performed:
read: 263354
write: 94055
other: 37622
total: 395031
transactions: 18811 (104.50 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 357409 (1985.56 per sec.)
other operations: 37622 (209.01 per sec.)
Test execution summary:
total time: 180.0044s
total number of events: 18811
total time taken by event execution: 179.7827
per-request statistics:
min: 4.04ms
avg: 9.56ms
max: 616.04ms
approx. 95 percentile: 38.42ms		 OLTP test statistics:
queries performed:
read: 188230
write: 67225
other: 26890
total: 282345
transactions: 13445 (74.67 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 255455 (1418.74 per sec.)
other operations: 26890 (149.34 per sec.)
Test execution summary:
total time: 180.0573s
total number of events: 13445
total time taken by event execution: 179.9174
per-request statistics:
min: 9.08ms
avg: 13.38ms
max: 904.58ms
approx. 95 percentile: 20.99ms

As you can see the EC2 can perform 40% more transactions than the RDS instance. Nothing unexpected so far.

Time to move on and increase the number of threads to 10

m1.small EC2 instance db.m1.small RDS instance
OLTP test statistics:
queries performed:
read: 264866
write: 94545
other: 37818
total: 397229
transactions: 18899 (104.97 per sec.)
deadlocks: 20 (0.11 per sec.)
read/write requests: 359411 (1996.22 per sec.)
other operations: 37818 (210.05 per sec.)

Test execution summary:
total time: 180.0462s
total number of events: 18899
total time taken by event execution: 1799.9289
per-request statistics:
min: 4.08ms
avg: 95.24ms
max: 2620.70ms
approx. 95 percentile: 445.91ms

 OLTP test statistics:
queries performed:
read: 343812
write: 122772
other: 49109
total: 515693
transactions: 24551 (136.18 per sec.)
deadlocks: 7 (0.04 per sec.)
read/write requests: 466584 (2588.13 per sec.)
other operations: 49109 (272.41 per sec.)

Test execution summary:
total time: 180.2788s
total number of events: 24551
total time taken by event execution: 1801.8298
per-request statistics:
min: 13.41ms
avg: 73.39ms
max: 1126.02ms
approx. 95 percentile: 143.83ms

In this test the small RDS instance is faster than the EC2, 136 vs 105 transactions per second. I’ve also benchmarked a large RDS instance (the next one available after db.m1.small) and it got 185 transactions per second. Quite good, but the price is 4x higher.

The next test was performed vs a 10 million records, 16 threads. This time I only benchmarked a small and a large RDS instance. The large instance managed to do 228 transactions per second while the small one got a decent score of 127 transactions. One thing I noticed during this test is that the small instance started to use it’s swap, while the large one did not have this issue. This is probably due to the fact that 10M records db is aprox 2.5GB and the small RDS only has 1.7GB of RAM.

So if you are planing to grow and want an easy way to do it, switching your database to its own RDS is one of the first things you should consider. One of the immediate effects you will notice is that the CPU usage on the EC2 instance will be greatly reduced, leaving more power for the web server. You can easily increase the size and capacity of the RDS instance with just a few clicks. The backups are done automatically, which is great considering how many times I had to recover databases.

Mysql max_allowed_packet error

You are probably here because you tried to import a big database (several GB) and got the following error:
ERROR 1153 (08S01) at line 2533: Got a packet bigger than 'max_allowed_packet' bytes

If you have access to your mysql server and SUPER privileges things are easy, you just need to log in as superuser to mysql and type this:
mysql>set global max_allowed_packet=64*1024*1024;
and then import the database normally, just adding “–max_allowed_packet=64M” to the parameter list. Example:
$mysql --max_allowed_packet=64M database < database.sql
Everything is so easy. But if you are using Amazon RDS you are out of luck. You setup a user when you create the instance but of course it doesn’t have the SUPER privilege so if you try to execute the above command it will fail. Not even “grant super on *.* to myuberuser” will help you, no no. So after some googling and reading a lot of crap I found this blog which had the same error as mine. Yuppy! Thanks Henry!

The solution is to use DB Parameter Groups. Grab your mouse and start copy pasting fast.

Download Amazon RDS Command Line Toolkit
The latest version can be found here
wget http://s3.amazonaws.com/rds-downloads/RDSCli.zip
unzip RDSCli.zip
cd RDSCli-1.3.003 (this will surely change so make sure you cd to the right directory)
export AWS_RDS_HOME=`pwd`
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk (this may vary depending on your java location and may not have to set it)
cp credential-file-path.template credential-file
vi credential-file (set your aws credentials there, use whatever text editor you like)
export AWS_CREDENTIAL_FILE=${AWS_RDS_HOME}/credential-file
cd bin
./rds --help

If everything went well you should get some output. On Henry blog he says he suggests that you create a parameter group. Well the reality is you have to create it since Amazon won’t let you modify parameters inside the default group.
./rds-create-db-parameter-group mygroup -f MySQL5.1 -d "My group"
./rds-modify-db-parameter-group mygroup --parameters "name=max_allowed_packet,value=67108864,method=immediate"
./rds-modify-db-instance YOURINSTANCENAMEHERE --db-parameter-group-name mygroup
Go to Amazon management console and check that the new parameter group is created and applied to your instance. You can begin now the import as you would do normally just add “–max_allowed_packet=64M” to the list of your options.

Hope it helps!

MySQL benchmarks using Amazon EC2 instances

Here are some tests I’ve run on Amazon using AMIs provided by scalr for the mysql role. I’ve used the benchmark scripts supplied by MySQL located in /usr/share/mysql/sql-bench. I had to install a package before running the tests:
apt-get install libdbd-pg-perl

After that everything was simple:
root@ec2# mysql
mysql> create database test;
mysql> quit;
root@ec2# cd /usr/share/mysql/sql-bench
root@ec2# perl run-all-tests --dir='/root/'

For EBS tests I’ve done the following:
-created 1GB EBS volume in scalr
-attached it to the instance I was testing
-notice the device name (/dev/sdb for example)
root@ec2# apt-get install xfsprogs
root@ec2# mkfs.xfs /dev/sdb
root@ec2# mkdir /mnt/storage
root@ec2# cp -R /var/lib/mysql /mnt/storage/
root@ec2# chown mysql:mysql -R /mnt/storage/mysql

-edit /etc/mysql/my.cnf and change datadir from “/var/lib/mysql” to “/mnt/storage/mysql”
-restart mysql server and start the tests:

root@ec2# /etc/init.d/mysql restart
root@ec2# mysql
mysql> drop database test;
mysql> create database test;
mysql> quit;
root@ec2# cd /usr/share/mysql/sql-bench
root@ec2# perl run-all-tests --dir='/root/'

Instances types used and their codes:

m1.small(0.10$/hour) – Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

m1.large(0.40$/hour) – Large Instance 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform

c1.medium(0.20$/hour) – High-CPU Medium Instance 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of instance storage, 32-bit platform

c1.xlarge(0.80$/hour) – High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

EC2 Compute Unit (ECU) – One EC2 Compute Unit (ECU) provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.


seconds usr sys cpu tests
m1.small 1823 196.54 28.66 225.2 3425950
m1.small+ebs 1646 197.18 29.61 226.79 3425950
m1.large 1072 157.06 26.97 184.03 3425950
m1.large+ebs 1088 154.23 25.23 179.46 3425950
c1.medium 902 131.18 25.63 156.81 3425950
c1.medium+ebs 901 130.76 28.84 159.6 3425950
c1.xlarge 704 123.31 32.8 156.11 3425950
c1.xlarge+ebs 781 121.02 29.52 150.54 3425950

Bellow you can see a nice chart with how much time it took for each instance to finish the benchmark tests. Either I did something terribly wrong or EBS doesn’t improve MySQL performance.