[dibby]

A long time client of mine has half a dozen small shopping cart sites. They mostly do low volume but quite high value sales. He had been using cheap shared hosting and after a problem his sites had all been offline for a few days he asked me to look into a more serious setup.

I knew that while the sites were important to him, getting him to swallow a large hosting bill was never going to happen. I suggested 2 tiny $50 a month servers and renting part of a failover capable load balancer - but he wasn't really going for it.

Started hunting arround and then I found linode.com

They offer virtual private servers, nothing new, but the rather cool part is they have facilities for failover.

The initial setup was 2 $19 a month virtual servers + the purchase of extra IP, giving a monthly bill of $39

The virtual servers were setup on 2 seperate physical machines within the same datacenter, no point having the 2 virtuals running on 1 physical.

Each virtual server has apache, mysql, haproxy and heartbeat running.

Heartbeat is pretty simple, it runs on each machine and if it detects a machine is down... it transfers an IP address to the other machine. So, your domains point to an IP... if the machine that IP is mapped to goes down.. it remaps to another machine.

It currently takes heartbeat about 5 seconds to switch the IP over if there is a problem.

HAproxy is essentially a software load balancer, it splits requests up between the 2 apache instances, if one goes offline, it stops sending traffic to it. Haproxy can also be used to handle mysql connections in the same manner.

HAproxy takes about 2 seconds to notice if any apache/mysql servers go offline.

Mysql is ran in master/master replication, so any changes on 1 server occur on the other. However, to simplify things, the majority of the time server A mysql is the one doing all reads/writes. It only switches to B if A is down, this is handled automagically by HAProxy.

To handle file changes between servers, there is a simple cron job to duplicate changes to www data on server A to server B. The client essentially only ever deals directly with server A.

We've changed the setup a bit here and there, and we run some more stuff - but this setup was really quite painless even for a linux noob like me. I love the fact you can get a pretty decent tollerant system up so easy.

A couple of other cool things about linode: your traffic allowance is poolable i.e. 200gigs/month per $19 account, but if you use 300 on server A, and 100 on server B - it doesn't matter.

Traffic between servers is free (if in same data center) - so replication/heartbeat traffic is free in this instance.

They have an API for all server actions.

I just thought for $39 a month... it's a brilliant possibility.

Edit: I forgot to explain the extra IP - thats the IP that gets swapped between machines if one goes down.

[gonzogone]

That's a GREAT solution Dibby! and for $39/mo?!

Not sure I've got a need anyplace for it right now, but I'll for sure be keeping it in mind

The virtual servers were setup on 2 seperate physical machines within the same datacenter, no point having the 2 virtuals running on 1 physical.

I'd expect you to catch that fine point, but there are many who would miss it... Things like this are great for anybody who is not all that geekie to read. It doesn't take a HUGE amount of money to build a great solution - what it does usually take is somebody who knows their stuff.

[VoyGeorge]

It would be interesting to know more about Heartbeat. Machines going down is a fairly remote reason for a server being unavailable. It's more likely to be due to a jammed-up MySQL database server, hung Apache, or something like that.

You'd always want to be connected to two separate machines on two separate switches, and ideally two separate power strips. I think of the times my sites have gone down it's been the fault of a power strip or switch, and not the machine itself, though that has happened (HD was going, or whatever).

For the price I guess it's okay, but it *could* be extra charges per month for failure insurance for a not-so-likely failure.

Interesting concept, though.

[gonzogone]

feels like it's been ages since I worked with servers at that level VG, but things like Heartbeat can work from both the hardware level and software. Specific hardware information can be measured dynamically with appropriate alerts sent out various ways, such as SNMP. On the software side, all levels of things can be monitored, down to specific port services. Between the two of those, it is not too difficult to catch a failure.

Link two or more systems together to check for each others heartbeat and it can work very well, even if the powerstrip fails. No pulse is sent, so the other system detects the failure and adapts.

I'd think that if you're in a good data center, this type of setup would be a major step up from most configurations and you wouldn't have to worry too much about what the servers are plugged into. Different servers, different switches and ideally a different router as well.

[dibby]

It would be interesting to know more about Heartbeat. Machines going down is a fairly remote reason for a server being unavailable. It's more likely to be due to a jammed-up MySQL database server, hung Apache, or something like that.

In that instance, the machine is not down is it? Merely an apache or mysql instance - haproxy deals with that issue.

You'd always want to be connected to two separate machines on two separate switches, and ideally two separate power strips. I think of the times my sites have gone down it's been the fault of a power strip or switch, and not the machine itself, though that has happened (HD was going, or whatever).

Think of heartbeat being a failover for machine availability. Whatever the reason for the machine being unavailable.. so your power strip example is fine, heartbeat works perfectly.

Not only can you do it on seperate power strips and switches, you can do it on seperate data centers in different continents :-) (you just pay for the replication data, but it's pretty small potatoes)

For the price I guess it's okay, but it *could* be extra charges per month for failure insurance for a not-so-likely failure.

Interesting concept, though.

..I went back and reread your question.

Heartbeat is purely IP level availability checking, so any issue that nocks the machine offline at the IP level (power strip, networking issue) will allow heartbeat to failover the active IP address to a new host.

HAProxy deals with things at a software sever level e.g. hung apache, mysql, etc

You are perfectly right, heartbeat without haproxy would be fairly pointless. Combine the two and you have monitoring and failover at essentially hardware/network level and at the software server level.

HAProxy would handle apache or mysql having issues, not heartbeat.

Do you still see gaps?

[VoyGeorge]

As long as there is a way to actually test a live site (e.g., test for a specific string on a page or result from an SQL query) that part is okay.

Since (somewhat) common failure points are the switch and power strip, having both of your machines on either or both of the same leaves you nothing to fail over to. So you'd want separation there.

Of course having the machines in two separate data centers would avoid these problems altogether, and adds failover protection for bandwidth connectivity, overall power supply at the DC, and what-not. Does this host also provide for this, even at a higher cost? Would be a good thing to investigate if so.

[dibby]

Originally Posted by VoyGeorge

As long as there is a way to actually test a live site (e.g., test for a specific string on a page or result from an SQL query) that part is okay..

Each apache instance is monitored for http headers, each nginx instance is monitored for http headers and each mysql instance has simple queries executed against a database.

Since (somewhat) common failure points are the switch and power strip, having both of your machines on either or both of the same leaves you nothing to fail over to. So you'd want separation there.
[

I'll need to ask the host if they share either :-)

Of course having the machines in two separate data centers would avoid these problems altogether, and adds failover protection for bandwidth connectivity, overall power supply at the DC, and what-not. Does this host also provide for this, even at a higher cost? Would be a good thing to investigate if so.

The host provides this for the same cost, theres no price difference in which data center you have your instance.

[gonzogone]