Migrating an RDS instance into VPC

I really like Amazon's Relational Database Service - I've used it for a few years now, and especially like that I don't have to worry about managing backups and maintenance for patches on the server.  I've done several restores to a point in time, mostly to debug something that happened at that moment but could not be replicated later (because the db state had changed) - and that has proven to be a great feature.

I have used RDS long enough now though that Amazon's push to use VPC has got me wanting to move everything into VPC.  They don't have an easy way to migrate your RDS server into VPC, though.  Well, they do, but it involves taking your server offline (or switching it to read-only) while you dump the database and upload it into your new server.  This, to me, is unacceptable - a large database could experience hours of downtime using this method.  I'm used to making these transitions with zero downtime, and I intended to keep that up with my own migration to VPC.  Here are the steps I've taken.

It is important that you are using a CNAME for your DB connections.  While you may be able to go through and update each connection script, using a CNAME will enable you to make a switch very quickly.  To this end, I will admit that I have a pretty good control on the write activities - only one process generally is writing to my database.  So I am not at a risk of having one server write to the old database while another writes to the new as the CNAME update is propagating.  Still, I set the TTL to a very low value (60 seconds) while the change is being made, to ensure that the old DB server stops getting writes as soon as possible.

So, on to the process.  What we'll do is:

1. Update our master server's binlog retention time
2. Create a replication user on our master server
3. Create a slave server from our master
4. Stop the slave on the server
5. Snapshot the slave's database
6. Restore the snapshot into VPC
7. Set the snapshot to be a slave to the original master
8. Update the CNAME to point to our new database
9. Stop the new server's slave once the CNAME has propagated
10. Shut down the original master

I'm not going into the steps needed to prepare your VPC for a database server, which involve setting up a subnet group in the RDS console. If you haven't already, you might just go through the process of launching a fresh RDS instance in your VPC to tackle what needs to be done there. AWS does a fairly good job of helping you through this process, and what they don't tell you can easily be found in their documentation.

So, here we go.

Update our master server's binlog retention time

If you are not familiar with how MySQL master/slave replication works, it is fairly simple; as queries that update the data on the master server are executed, they are written to a log file which is sent to the slave databases who then execute the same queries on their database. You can see how far out of sync your slave server is from its master by executing the SHOW SLAVE STATUS command. You can see the current location on your master by executing the SHOW MASTER STATUS command. What we are going to do is snapshot our main database and then slave the new server to our original master until it catches up to it, then switch to use the new server. However, the default binlog retention time is very short (the binlog - binary log file - is the file that the master writes to for the slave. You don't want to keep this file around forever because it would grow to be huge, but you need it long enough so that your new slave can start reading it when we bring it up).  I have set mine to 24 hours to make sure I can attach to my log file with plenty of time to spare.

This is not a configuration you can set in your MySQL parameters, but you can call a special command embedded in the RDS MySQL:

call mysql.rds_set_configuration('binlog retention hours', 24);

Create a replication user on our master server

RDS will create a replication user for the slave that it creates, but we need to create one that we know the password to so we can connect with that user.  You can create a user in phpMyAdmin if you have that installed, or you can do the following from the command line:

CREATE USER 'replication'@'%' IDENTIFIED BY '[password]';
GRANT REPLICATION SLAVE ON *.* TO 'rep'@'%' IDENTIFIED BY '[password]' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0;

Be sure you keep the replication username and password handy, you'll need it in a little bit.

Create a slave server from our master

Now we're ready to create a read-only slave of our database server.  The reason we are going to do this is because we want to make sure we have a valid snapshot in a point in time and we want to know what that point is, which we'll be able to identify from our slave.

Name it something relative to your main server, but don't think too hard about it, we won't keep it around for long. This step may take a while depending on how big your database is, but we have to wait now until the slave server has been created and is available.

Stop the slave on the server

Now that we have a slave, we're going to stop it from propagating from the master. Normally we would use a "STOP SLAVE" command, but our RDS credentials won't allow us to do that. Fortunately, they have given us another RDS-specific command to stop the slave:

CALL mysql.rds_stop_replication;

This will stop our slave.  We want to check and see what the status of our slave is, and note some very important values that we'll use when we bring up our new server:

show slave status\G

The values we are looking for are Master_Log_File and Read_Master_Log_Pos.  Make a note of these values with your replication user info.

Snapshot the slave's database

Now we're ready to take our slave's database and prep it to become our new database. Right-click on the slave in the console and select Take DB Snapshot.

Name the snapshot something about moving your database into VPC, and then wait for it to be completed.

Restore the snapshot into VPC

Once you have a snapshot, it's time to bring up your new database server in the VPC. At the time of this writing, there appears to be some sort of a bug in the management console that is preventing a restore of a snapshot into a VPC, so I had to do this from the command line.  Perhaps you will be more fortunate. Here is the command line call I used:

rds-restore-db-instance-from-db-snapshot [new database name] -sn [subnet group] -s [snapshot name] -pub true

It is important to make this database publicly accessible, because otherwise it will not be able to access our master. To this end, be sure your master is publicly accessible as well (at least to the IP address of the new server - because otherwise we won't be able to attach our slave to it).   You might want to include additional options from the rds-restore-db-instance, such as the instance class or option group, but you can also change those via the GUI before switching to the new database.

Once again, wait a bit until this server completes its launch.  We're almost there!

Set the snapshot to be a slave to the original master

Now connect to your new database server that is in the VPC. We need to turn this server into a slave of our existing master server where our previous slave left off.  Once again, we don't have the rights to use the CHANGE MASTER command, but we do have a handy RDS command:

CALL mysql.rds_set_external_master ([host], 3306, [user], [password], [logfile], [log-pos], 0);

where "host" is the very original (still live) database, "user" and "password" are the credentials to your replication user, and "logfile" and "log-pos" are the values we got from our slave server.

Now we need to start our slave, again using an RDS command:

CALL mysql.rds_start_replication;

If we call SHOW SLAVE STATUS at this point, we should see information indicating that our slave is running ("Slave IO Running" and "Slave SQL Running" should both be "Yes").  If this is not the case, the most likely problem is that your two databases cannot see each other; make sure you have a network path between them.

Update the CNAME to point to our new database

Once our slave server has caught up to our master (see the "Seconds behind master" attribute in the SHOW SLAVE STATUS output) we can migrate our CNAME to point to our new database. If all your applications use the CNAME, they will all switch within the TTL.  If you have the potential for two systems to connect at different times, you may need to be more cautious.  Depending on how likely this is and how much trouble it could cause, you might need to treat this as a master-master setup by changing the auto_increment_increment and auto_increment_offset values for the two servers - but that is beyond the scope of this tutorial.

One step that you might want to do before actually changing the CNAME is to launch a copy of your site (if your site is on AWS then just make an Image of your site and launch it), and go into that server to set the database connection directly to your new server. This will give you a chance to verify that your new database server is in a security group that your web server(s) can access. If you are unable to connect from your web server, you will probably need to check your security groups across the board to figure out why.

At this time, you will want to wait until all your servers have propagated to your new database. If your TTL was 5 minutes, wait at least 5 minutes.  After a while, you should be able to check your slave status and see that the master log position is not moving even if there has been activity on your site. When you are convinced that your old database server is no longer being used, you are ready to go on to the next step.

Stop the new server's slave once the CNAME has propagated

Your new server no longer needs to be connected to the slave - so we can stop it:

CALL mysql.rds_stop_replication;

and reset our slave info:

CALL mysql.rds_reset_external_master;

Now your old server is done and your new server is active!

Shut down the original master

At this point your original master and slave are ready to be shut down. I always take a snapshot as AWS advises just in case something goes wrong with the new server that was missed, or if for some reason I accidentally shut down the wrong server ... 

So there you have it, you have successfully moved your RDS server from the classic network to your VPC!

Using an m3.medium AWS server for less than the cost of a t1.micro

I've used Amazon Web Services for quite some time. My online football management simulator, MyFootballNow, runs on AWS.  The sims spin up on spot instances to help keep costs down, because they need to run on at least a m1.small server to finish in an acceptable amount of time. I also have a test environment where I try to sim a full season every day, to see the long-term effects of various code changes in the game engine. I've been using a physical server for that, because at this stage I don't want to pay for a server at Amazon full-time just to run a test environment. This server has been showing signs of impending failure lately, though, and rather than purchase a new server I began to explore the least expensive way to have a decent server at AWS.

My first thought of course was to purchase a reserved instance.  With the ability to sell your reservation if you end up not needing it any longer, the risks are low. But it's still a good amount of money up front for a system that isn't generating any revenue yet.

The absolute lowest cost for a server at AWS can be through spot instances.  When I changed running the sims from an on-demand server to a spot instance server, my costs reduced dramatically. Could I leverage a spot server to run my test environment?

If you're not familiar with spot instances, Amazon makes their unused compute capacity available at a discounted price.  You set a maximum price you're willing to spend, and if the spot price is below your bid then your instance launches, and you pay whatever the current spot price is.  If the spot price rises above your bid, your instance is terminated.  You can't stop a spot instance, you can only terminate it.  But you can set up a persistent spot request that will re-launch your spot instance from the AMI once the price goes back below your bid.

The biggest problem with the persistent request for me was that the instance essentially resets itself every time it launches.  This wouldn't be horrible, but it would mean making sure that I created a new AMI every time I made a significant change to the environment, and if I was shut down then I'd have to roll back to my most recent AMI.  There had to be a better way.

And there was.  When you create a spot instance (or any instance, for that matter) you are presented with the following screen as you create your volumes:

Note the checkbox on the far right "Delete on Termination" - if you uncheck this, the volume will stick around even after the instance has been terminated.  With this orphaned volume, we can recreate our instance at the point it was shut down.

The price of an m3.medium spot instance right now is a few hundredths of a cent less per hour than an on-demand t1.micro, and quite a bit less than a heavy utilization m3.medium instance.  The catch is that your instance could be shut down if there is a spike in the spot price, so take caution using this technique on a production server that you don't want to disappear unexpectedly.  For a development server, it works fabulously, and if you set your spot price bid high enough you might even be able to sustain your instance as long as you like.

So, what I did is create a spot instance of type m3.medium with a $0.02 max bid (the current rate is $0.008 and hasn't gone up even to $0.009 since Amazon's last price reduction).  That means I should pay no more than $15 per month for this server, and if the spot price never goes above my bid then I could potentially have my m3.medium for about $5/month.  You can of course extrapolate this to any server type.

Back to the catch: if your server shuts down, it will leave its volume behind, but if you're like me and have tried to launch a new instance from a volume in this state you probably failed miserably.  I finally figured out that the reason for this failure is because I was using the default kernel, which didn't match my previous server.  So, find the kernel id for your spot instance:

and make a note of it.  I have it in the name of my Volume, i.e. "Dev-01-/dev/sda-aki-919dcaf8" - which will give me everything I need to know to launch a new instance from this volume.

To launch an instance with your volume after the system has been terminated, right-click on the volume and create a snapshot.

Now go to the snapshot list, find your snapshot you just created, and create an image from it:

When you create the image, you are given the opportunity to choose the kernel ID - it is important to use the same kernel ID you discovered in the above step.  This is why I always name my spot instance volumes with the kernel ID.

My spot instance actually has two volumes plus the instance store - the instance store is gone forever when the instance is terminated, I just use it for scratch space. To apply the second volume, you'll need to create a snapshot from it and then choose the snapshot when you set up the volumes.  Make sure you have it mounted to the same location if you have an /etc/fstab entry for it; again, that's why my volume naming convention contains the /dev/sda part.

After you've created the image, make a new spot request with that image.  Once you have logged in to the new instance and verified that everything is as it should be, you can delete the old volumes, delete the AMI, and delete the snapshot.

You might also combine this with my script to set a DNS entry on boot, which will keep your server's DNS record up to date.  The only gotcha here is that the host identification will change with the new image; depending on how you are connecting via SSH it should not be difficult to reset the identification.

Hope this helps and enjoy your no commitment discount m3.medium servers!

An automatic snapshot management script for AWS

I love using Amazon's servers. I grew up in this industry dealing with physical servers and they are a pain to deal with, especially if you ever need to migrate to a new piece of hardware or have a hard drive fail.  With Amazon's ability to snapshot your drives, you can quickly spin up a server that is identical to an existing one, or is from a snapshot from before you made that huge mistake that blew up your filesystem.

There is the rub: you need to have those snapshots, and if you are paranoid like me you need to have them taken regularly.  AWS doesn't really have a great facility to manage your snapshots.  What I wanted was a way to take a weekly snapshot of all of my drives, but only keep the snapshots for a month so as not to clutter my snapshot list.  This post is to share my script that I have written to manage these snapshots.

This is actually really a simple script.  I'm going to drop it in here, and then tell you how it works:

#!/bin/bash

export EC2_HOME=/opt/aws/apitools/ec2
export JAVA_HOME=/usr/lib/jvm/jre
export SNAPSHOT_LIST=/var/spool/snapshots

VOLUMES=$(/opt/aws/bin/ec2-describe-volumes | sed 's/ /-/g' | grep TAG | cut -f 3,5 --output-delimiter='|')
for line in ${VOLUMES//\\n/$cr}
do
VOLUME=`echo $line | cut -f 1 --delimiter='|'`
NAME=`echo $line | cut -f 2 --delimiter='|'`-`date "+%m-%d-%y"`
SNAP=`/opt/aws/bin/ec2-create-snapshot --description $NAME $VOLUME | cut -f 2`
echo $SNAP > $SNAPSHOT_LIST/$SNAP
echo $NAME snapshot to $SNAP
done

echo
echo

# Purge old snapshots
find $SNAPSHOT_LIST -ctime +30 -type f -execdir /usr/local/bin/delete_snapshot {} \;

First we're setting some variables needed by the amazon tools.  The third entry for SNAPSHOT_LIST is a folder that I'm going to use to keep track of my snapshots.  You'll need to create this folder and give access to the user you'll have run this script.

Next, I'm calling ec2-describe-volumes to retrieve all of the volumes in my EC2 area.  I'm replacing spaces with a hyphen, looking for the TAG line, and using cut to get the volume and volume name. One feature of this script is that it will only snapshot volumes that you have named - so if you have something temporary you can leave its volume unnamed.  You certainly could modify this script to do every volume, but you'd have to come up with some way to make the snapshots make sense as to where they came from.  Here, we are naming the snapshot after the volume name and stamping the date to the end of it.

Next, we execute ec2-create-snapshot to create the snapshot for our volume, and storing it in a file in our snapshot list folder.  We'll use this file list to see how old our snapshots are, which you see in the last line where we are finding any files in our snapshot list that are older than 30 days.  We're executing /usr/local/bin/delete_snapshot which is the second script in our system:

#!/bin/bash

# Receives a file that contains the snapshot-id that we want to delete

SNAP=`cat $1`
echo Deleting snapshot $SNAP
/opt/aws/bin/ec2-delete-snapshot $SNAP
rm $1

This one is pretty easy, we're grabbing the snapshot id from the file (which also happens to be the file name) and executing ec2-delete-snapshot to delete it.  Then we're removing the file.

I have this set to run every monday morning.  It does a very nice job of keeping one month's supply of snapshots should something catastrophic happen.

Have your AWS server set its own DNS on boot

One of the huge advantages of using Amazon's Web Services is the fact that you can turn virtual machines on and off, and only pay for the time that the instance is running.  This can be really cool if you need a staging server of some sort, and only need it to be running during the work day, or even just on the days you are working on a particular project.  One huge disadvantage to  this, though, is that you get a new IP address every time you boot an instance. Sure, you could use an elastic IP address, or you could manually set your DNS record each time you boot, but why pay for the IP address when you're not using it (Amazon charges you by the hour that it's not connected to an instance)? And why go through the trouble of setting your own DNS?  You have a server at your fingertips, let it grow up and take care of itself!

Unfortunately, there is not a Route 53 implementation in the command line tools that are automatically installed on the Amazon AMI.  There is, however, a great application by Barnaby Gray called cli53 which is on github here: https://github.com/barnybug/cli53.

Installation is painless via the python pip package management system.  Here is an install on a fresh Amazon AMI instance:

First, I'm installing pip (because it doesn't come automatically):

$ sudo yum install python-pip

Now we can install cli53:

$ sudo pip install cli53

That's it, it is now installed. In order for pip to run it needs to have either the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables set, or they need to be placed in a file located in ~/.boto. I'm using the boto method, so using your favorite text editor create a file that contains the following:

[Credentials]
aws_access_key_id = XXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

You can also set up your server in a role that has access to Route 53, and you won't have to manually set these credentials.

The next step is to install the script that will set your DNS entry for the server.  Amazon provides several URLs that you can request from your server that gives you information about your server.  We'll be using http://instance-data.ec2.internal/latest/meta-data/public-hostname which returns our public hostname entry.  We'll use this to add a CNAME entry to our Route53 DNS.

#!/bin/bash

# This script will set the DNS record for this server.  Run from rc.local to set on boot.
# Requires cli53 https://github.com/barnybug/cli53

# This is the domain we are updating
DOMAIN=example.com
# This is the subdomain
SUBDOMAIN=staging

# Obtain our public host name
PUBLIC_HOST_NAME="`wget -q -O - http://instance-data.ec2.internal/latest/meta-data/public-hostname || die \"wget public-hostname has failed: $?\"`"
test -n "$PUBLIC_HOST_NAME" || die 'cannot obtain public-hostname'

echo Setting $SUBDOMAIN.$DOMAIN to $PUBLIC_HOST_NAME
echo

cli53 rrcreate $DOMAIN $SUBDOMAIN CNAME $PUBLIC_HOST_NAME --ttl 300 --replace

Let me go over this line by line.  First we have two variables that we're setting: $DOMAIN and $SUBDOMAIN.  $DOMAIN is a domain that you have in Route53; it's the primary domain name, not the subdomain.  The $SUBDOMAIN is the actual subdomain element that you are wanting to set the CNAME for.  In my case, I'm using example.com and setting the subdomain staging, so my server will receive the DNS entry staging.example.com.

The next part queries http://instance-data.ec2.internal/latest/meta-data/public-hostname to get our public hostname.  This is the long public DNS entry that looks like ec2-X-X-X-X.compute-1.amazonaws.com, and we'll set our CNAME to this value.  We could just as easily get the IP address, but using the subdomain has the advantage of resolving to an internal IP address from another AWS instance, making communication between AWS instances available on the internal network.

Lastly, the cli53 command replaces the CNAME entry for this hostname. This will only work if you don't have an existing setting for this subdomain, or if the existing entry is already a CNAME; i.e. it  will fail if you have an A record.

Once you have the script ready, run it from the command line. You should see an output something like this:

Setting staging.example.com to ec2-X-X-X-X.compute-1.amazonaws.com

Success
ChangeInfo:
  Status: PENDING
  SubmittedAt: 2014-05-15T15:51:11.975Z
  Id: /change/C3LPBE20HC9ITN

If you now go into your Route53 console, you should see the new entry with a TTL of 5 minutes.

The next step is to get this script to execute every time the machine boots. Enter cron, which has a handly option to execute on boot.

Using your text editor, create a file at /etc/cron.d/set_route_53 with the following contents:

EC2_HOME=/opt/aws/apitools/ec2
JAVA_HOME=/usr/lib/jvm/jre

@reboot [user] /usr/local/bin/set_route_53

where [user] is the username you installed the .boto file in.  The environment variables at the top are required to have the environment variables accessible to the script, and those are variables that are used in the AWS command line utilities.

Now, reboot your machine.  If you have mail configured for the user running the cron job, you'll get an e-mail with the output of the command.  If you stop and then start your instance, you should see the Route53 entry update.  With the TTL set to 5 minutes, you should always be able to access your instance within 5 minutes of its boot being complete, and you don't have to lift another finger!

Sending E-mail through Mandrill in C++

Mandrill is MailChimp's awesome transactional e-mail service. Until recently, I have always used the server's native outgoing mail server to send mail from my systems, often linking those servers to deliver through Amazon's Simple E-mail Service. I have slowly been moving all of my systems over to Mandrill because of it's great feature set - you can easily generate templates with merge tags in them, you can see the e-mails that have been sent including a rendering of the most recent ones, and best of all, for my usage level it's free.

My most recent conversion was for our online football management game, MyFootballNow.com. In this system, there are two sources that send e-mails to users. Our PHP website sends various notifications and confirmations as users interact with the site. But the main engine that drives the simulation was built in C++, and it needs to send e-mail notifications to the appropriate players when it completes. The first generation of our game engine sent e-mails through the server's mail command, but that was complicated to build the HTML e-mail, and didn't offer much flexibility. I needed a way to send e-mails from my C++ application though Mandrill, but wasn't able to find any information about how that could be done. There are SDK's for several languages, but C++ was not among them. I decided to buckle down and figure out how to send an e-mail through Mandrill directly from my application. As it turns out, it was remarkably easy to do, and this entry is a short tutorial on how to do it.

First, I'll briefly talk about Mandrill. While not limited to sending from a template, that is where it really shines. You build your template once and can use it again and again. You can even build the template in MailChimp and transfer it over to Mandrill. You can make it a simple e-mail that has no dynamic content, or you can use merge variables to fill in parts of the message. You can even do complex logic such as if-then statements. I encourage you to check out MailChimp's support page for Mandrill at http://kb.mailchimp.com/article/mandrill-for-mailchimp-users to learn more; I'm not going to go too much more into how to create a template, but here's what my template looks like:

<p>Hello, *|NAME|*,</p>


<p>This is a friendly e-mail to let you know that
we can send an e-mail through Mandrill using
a C++ application.</p>

*|NAME|* in this e-mail is our merge variable, we will be inserting the recipient's name into the salutation.

Mandrill receives its command via an HTTP Post request, where the post data payload is a JSON-encoded string of information. The call we're going to be using is called "send-template." You can see all of the options for the JSON string you pass here. This tutorial will barely scratch the surface of what you can send, but you should be able to add anything else you need to your heart's content. You should also be able to apply these same techniques to the other API calls available.

So, the first thing we need to figure out how to do is to create a JSON string in C++. This could be easy enough if you want to build it from scratch, but that is error prone especially with nested JSON data. No, we need something that will do it for us. And I have just the tool for the job: the Boost library.

When I first started writing MyFootballNow.com, I fell in love with the Boost library. And I probably use less than 1% of what it offers. When I need to figure out how to do something new, the first place I check is Boost. For this application, we are going to use two of Boost's modules: the Property Tree and the JSON Parser.

If you need to install Boost, you can do it from the link above. Or, you probably can install it via your package manager. In CentOS and Amazon's AMI, it's just:

yum install boost-devel

While we're installing libraries we'll need, we need to make sure we have the CURL library so we can execute the HTTP Post request:

yum install curl-devel

Now we should have what we need. (This is of course assuming we already have the C++ compiler installed - before jumping into this tutorial make sure you can build and execute a simple "Hello World" application in C++)

One gotcha for using CURL is that you need to link the CURL libraries when you compile. Depending on your setup, you'll want to add -lcurl to your compile command. I normally use a Makefile which is generated by my IDE, but the compile command should look something like this:

g++ -o mandrill_test main.o -lcurl

Now for the heavy lifting. The first thing we need to do is get the include files lined up. You need to include the boost property tree and json parser files, as well as the curl library:

#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
#include <curl/curl.h>

Another housekeeping note, I'm using namespace std:

using namespace std;

Now to the body of our function. For this tutorial, we are creating a simple application that sends a message to a pre-determined e-mail address every time it runs. In a real application, of course, you would be collecting the information from elsewhere and probably calling a function that sends a message with the requested information.
The main variables we'll be using are the API Key, the Template Name, the e-mail address, and the recipient's name:

char mandrill_key[] = "[Your API Key Here]";
char template_name[] = "[The Template Name]";
char email[] = "test@example.com";
char name[] = "Joe Smith";

We also need some variables for our CURL:

CURL *curl;
CURLcode res;
struct curl_slist *headerlist = NULL;

and our property tree:

boost::property_tree::ptree req;
boost::property_tree::ptree template_content, template_content_value, to_data, to;
boost::property_tree::ptree message;
boost::property_tree::ptree global_merge_vars;
boost::property_tree::ptree merge_var;
stringstream ss;
char payload[5000];

This one doesn't look quite as straight forward; I've got a ton of property tree variables, why is that? Boost's property tree is built using key/value pairs. As you may know, a JSON string can have multiple nested arrays and objects. So we need to build the nested properties from the inside out, making the deepest element a child of the next-deepest, and so on. To keep things clear, I'm using a different ptree item for all pieces of information.

"ss" is of type std::stringstream; this is where we will be storing our JSON string. The Boost JSON parser dumps the encoded JSON to a file, not a variable. std::stringstream gives us a stream that acts like a file but we can then use as a variable.

The payload is where we're storing the final JSON string. I'm going to guess that I don't need this intermediary variable, but was unable to get it working otherwise ... so you're welcome to try to eliminate this one.

Now it's time to build the property list:

    // First item in our property list will be the mandrill key
    req.put("key", mandrill_key);
    // The template we will be using
    req.put("template_name", template_name);
    // template_content must exist but can be empty
    template_content_value.put("name", "Template");
    template_content_value.put("content", "Template Content");
    template_content.push_back(make_pair("", template_content_value));
    req.add_child("template_content", template_content);

    // To data
    to_data.put("email", email);
    to_data.put("name", name);
    to_data.put("type", "to");
    to.push_back(make_pair("", to_data));
    // Repeat above for as many recipients as you want for this e-mail
    message.add_child("to", to);

    // Merge variables
    message.put("merge", "true");
    merge_var.put("name", "name");
    merge_var.put("content", name);
    global_merge_vars.push_back(make_pair("", merge_var));
    // Repeat the above for all merge variables
    message.add_child("global_merge_vars", global_merge_vars);
    req.add_child("message", message);

Note that the top-level parent tree is "req" and we are ultimately appending everything to that tree. The command to attach an item to a tree is "put" - so the first line, 'req.put("key", mandrill_key);' is putting our API key into the tree with the key "key" ... (maybe that wasn't the clearest example)

Things change up a little bit when we get to the template_content item. Mandrill appears to require this even though we're not using it, so we're going to include it to quiet the error message. But I have something odd in the way that template_content_value is being applied to template_content: 'template_content.push_back(make_pair("", template_content_value));' Why not just do 'template_content.put("", template_content_value);'? The answer is that the put() command requires a key - and in this instance, we don't want a key, we want an array of objects without keys. The way to make this happen is to use std::make_pair() with an empty string, and push that back onto the template_content tree. We do this same thing as we add the "To" data and merge variables.

Next up, we need to create the JSON string. This is a simple call to write_json(), which as I mentioned above will write the JSON out to a file stream, in our case we're usinng the stringstream ss; then immediately we'll convert this into a null-terminated c string to use in our request:

    // Generate the json from our property tree
    write_json(ss, req);
    strcpy(payload, ss.str().c_str());

Now all we have left to do is to send the HTTP Post request to the Mandrill API endpoint. Here's the code for that:

    // Send the message
    curl = curl_easy_init();
    headerlist = curl_slist_append(headerlist, "Content-Type: application/json");
    curl_easy_setopt(curl, CURLOPT_URL, "https://mandrillapp.com/api/1.0/messages/send-template.json");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, payload);

    res = curl_easy_perform(curl);
    // Check for errors
    if (res != CURLE_OK) {
        // Handle error. Error string can be retrieved via curl_easy_strerror(res)
    }

    // Cleanup
    curl_easy_cleanup(curl);
    curl_slist_free_all(headerlist);

The first thing we do is initialize the CURL instance. Next, we apply a header to identify the content type as application/json. Then we tell the CURL instance the URL it will connect to, we attach the header, and attach the payload. After that we're ready to send, check for an error, and then clean up curl and headerlist.

When this is compiled and run, the response from Mandrill will be sent to stdout - you should see a verification of the e-mail being sent, or an error message as to why it didn't work.

I hope this short tutorial helps you if you are needing to send e-mails through Mandrill via C++! I will leave you with the full listing of my main.cpp that I used in this tutorial.

#include <cstdlib>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
#include <curl/curl.h>

using namespace std;

/*
 *
 */
int main(int argc, char** argv) {
    // Basic stuff
    char mandrill_key[] = "[Your Mandrill API Key]";
    char template_name[] = "[Template Name]";
    char email[] = "test@example.com";
    char name[] = "Joe Smith";

    // CURL Stuff
    CURL *curl;
    CURLcode res;
    struct curl_slist *headerlist = NULL;

    // Boost stuff
    boost::property_tree::ptree req;
    boost::property_tree::ptree template_content, template_content_value, to_data, to;
    boost::property_tree::ptree message;
    boost::property_tree::ptree global_merge_vars;
    boost::property_tree::ptree merge_var;
    stringstream ss;
    char payload[5000];

    // First item in our property list will be the mandrill key
    req.put("key", mandrill_key);
    // The template we will be using
    req.put("template_name", template_name);
    // template_content must exist but can be empty
    template_content_value.put("name", "Template");
    template_content_value.put("content", "Template Content");
    template_content.push_back(make_pair("", template_content_value));
    req.add_child("template_content", template_content);

    // To data
    to_data.put("email", email);
    to_data.put("name", name);
    to_data.put("type", "to");
    to.push_back(make_pair("", to_data));
    // Repeat above for as many recipients as you want for this e-mail
    message.add_child("to", to);

    // Merge variables
    message.put("merge", "true");
    merge_var.put("name", "name");
    merge_var.put("content", name);
    global_merge_vars.push_back(make_pair("", merge_var));
    // Repeat the above for all merge variables
    message.add_child("global_merge_vars", global_merge_vars);
    req.add_child("message", message);

    // Generate the json from our property tree
    write_json(ss, req);
    strcpy(payload, ss.str().c_str());

    // Send the message
    curl = curl_easy_init();
    headerlist = curl_slist_append(headerlist, "Content-Type: application/json");
    curl_easy_setopt(curl, CURLOPT_URL, "https://mandrillapp.com/api/1.0/messages/send-template.json");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, payload);

    res = curl_easy_perform(curl);
    // Check for errors
    if (res != CURLE_OK) {
        // Handle error. Error string can be retrieved via curl_easy_strerror(res)
    }
    // Cleanup
    curl_easy_cleanup(curl);
    curl_slist_free_all(headerlist);

    return 0;
}