Cheap way to a fully redundant cluster!

Over the past 18 months, cloud computing has become an increasingly important component of the individual business; large and startup alike. More and more companies are turning to the cloud for their hosting, storage, and application needs. As companies switch their infrastructure over, the high-availability of these services becomes an ever important, but often overlooked, piece of the critical infrastructure puzzle.

In this article we will discuss how to achieve High-Availability on a tight budget which is surprisingly easy and inexpensive for most organizations to achieve. Using just a few of the open source utilities out on the market today, companies are able to achieve total redundancy and high-availability by utilizing Gigenet's Los Angeles (LA) Cloud Servers and Chicago (CHI) Cloud Servers. We will be looking at one way businesses can leverage their cloud by providing high-availability services—affording respective clientèle the top-notch service they expect, and the individual company the fail-safe infrastructure they deserve.

The 4 technologies that we are going to discuss to achieve this desired effect of high-availability services in the infrastructure are as follows:

HAproxy (For Load Balancing)
BIND + GeoDNS (For less Latency and High-Availability)
MySQL (For Database back-end and replication)
Apache / Nginx (For serving the content)

-

Making all of these opensource technologies work together is extremely simple as the step by step guides will show in future articles. For now lets take a glimpse at each of the aforementioned technologies!

-

HAProxy – The Reliable, High Performance TCP/HTTP Load Balancer
http://haproxy.1wt.eu/

“HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and risk-less, while still offering the possibility not to expose fragile web servers to the Net.”


ISC BIND DNS + MaxMind GeoDNS patch
http://www.isc.org/software/bind
http://www.maxmind.com/app/geoip_resources

“BIND is by far the most widely used DNS software on the Internet. It provides a robust and stable platform on top of which organizations can build distributed computing systems with the knowledge that those systems are fully compliant with published DNS standards.”

In addition, BIND is in use on 10 out of the 13 root name servers; the top-level DNS servers that directly answer requests for records in the root zone... these 13 servers form a crucial part of the Internet because they are the first step in resolving domain and host names into IP addresses—the method by which Internet hosts use to communicate.

The MaxMind GeoDNS patch is a complementary, and free, software patch for BIND that enables your company to add geographical filter support to the existing views in BIND.

One very typical use of this is directing visitor traffic to the nearest geographically available web-server – no longer will it be necessary to direct this critical traffic in a hap-hazard way when there's clearly a better route available...

Combine BIND and GeoDNS with the power of the GigeNET Cloud and you've got a rock-solid infrastructure that will field requests to your clientèle with the lowest latency possible!


MySQL
http://www.mysql.com/

“MySQL is the world's most popular open source database software, with over 100 million copies of its software downloaded or distributed throughout it's history. With its superior speed, reliability, and ease of use, MySQL has become the preferred choice for Web, Web 2.0, SaaS, ISV, Telecom companies and forward-thinking corporate IT Managers because it eliminates the major problems associated with downtime, maintenance and administration for modern, online applications.

Many of the world's largest and fastest-growing organizations use MySQL to save time and money powering their high-volume Web sites, critical business systems, and packaged software — including industry leaders such as Yahoo!, Alcatel-Lucent, Google, Nokia, YouTube, Wikipedia, and Booking.com.”

MySQL includes not only the features that expert DBAs expect—such as Stored Procedures, Triggers, Cursors, Distributed Transactional Support, multiple storage engines, SSL support, Query Caching, and Hot Backup—but ACID compliance, multiple asynchronous replication modes, and even synchronous Clustering!


Apache and Nginx – High Performance Web Servers
http://httpd.apache.org/
http://nginx.org/


NGinx is known to have better performance than Apache does but either one of these two free technologies will work. Its really about choosing what you are more comfortable with.

"Nginx is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. Igor Sysoev started development of Nginx in 2002, with the first public release in 2004. Nginx now hosts nearly 7.65% (22.8M) of all domains worldwide.

Nginx is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.

Nginx is one of a handful of servers written to address the C10K problem. Unlike traditional servers, Nginx doesn't rely on threads to handle requests. Instead it uses a much more scalable event-driven (asynchronous) architecture. This architecture uses small, but more importantly, predictable amounts of memory under load.
Even if you don't expect to handle thousands of simultaneous requests, you can still benefit from Nginx's high-performance and small memory footprint. Nginx scales in all directions: from the smallest VPS all the way up to clusters of servers.

Nginx powers several high-visibility sites, such as WordPress, Hulu, Github, Ohloh, SourceForge, WhitePages and TorrentReactor. "

Apache is known for its ease of use and is essentially the worlds most popular web server. If you run in to any problems with Apache you are more likely to find the solution much quicker than you would with, say, NGinx.

"Apache Synapse is a lightweight and high-performance Enterprise Service Bus (ESB). Powered by a fast and asynchronous mediation engine, Apache Synapse provides exceptional support for XML, Web Services and REST. In addition to XML and SOAP, Apache Synapse supports several other content interchange formats, such as plain text, binary, Hessian and JSON. The wide range of transport adapters available for Synapse, enables it to communicate over many application and transport layer protocols. As of now, Apache Synapse has support for protocols like HTTP/S, Mail (POP3, IMAP, SMTP), JMS, TCP, UDP, VFS, SMS, XMPP and FIX.

Apache Synapse is free and open source software distributed under the Apache Software License 2.0 The latest version of Synaspe is v2.0 . This release comes with a horde of new features, bug fixes, performance and stability improvements."


Tie the Technologies Together

In order to really achieve high-availability as promised in this article you will have to know how to fit the pieces of the puzzle together. While this article might not go in to technical details on a step by step guide we have included some references to get you started next to each line item.

Essentially you will need to separate the Dynamic files (Files and Directories that are subject to change) from the Static files (Files and Directories that will remain consistent in both locations). Once this is achieved you will be able to keep both machines up-to-date with rsync or a similar tool. Basically for the files you will have two directories on each machine; one will be for staging and the other production. You will NFS mount the production directory from one server to the staging area on the other server and vice versa so that data is always stored locally. For more information on this I would refer you to pages like this one:

http://serverfault.com/questions/119961/is-there-a-way-to-mirror-two-severs-on-ubuntu
http://www.howtomonster.com/2007/08/08/how-to-sync-data-between-2-servers-automatically/
http://superuser.com/questions/126884/best-way-to-automatically-synchronize-files-between-linux-and-windows

To sync Mysql you will need to refer to articles such as this:

http://www.neocodesoftware.com/replication/
http://www.howtoforge.com/mysql_master_master_replication

For GEODns I would recommend going to a provider that specializes in this so that if one server is down the DNS server knows not to send traffic to it at all. This is a form of load balancing but its more of a high-availability feature in this article. This is the feature that will prevent traffic from being sent to a server or load balancer that is not operational or online. Some providers to look at for this service (GigeNET will soon have this):

http://www.autofailover.com
http://www.ultradns.com/Services/DNS-Traffic-Services/Directional-DNS
http://dyn.com/enterprise-dns/dynect-platform/dynect-traffic-management-gslb

For HAProxy you will need at least one HAProxy server per physical location. This could be on the same server that the MySQL Database, Dynamic and Static Files, Apache/NGINX, and BIND are located on. The reason why you need HAProxy is so that you can add servers in each physical location (as much as needed) quite easily. Expanding Horizontally is a very important feature in any cluster solution. GigeNET's Cloud Servers enable you to scale both vertically (by growing the Cloud Server resources) or horizontally (by building new Cloud Servers based on a template or from backup).

http://www.howtoforge.com/setting-up-a-high-availability-load-balancer-with-haproxy-wackamole-spread-on-debian-etch
http://linuxadminzone.com/how-to-install-setup-and-config-haproxy-loadbalancer-for-content-switching/

GigeNETCloud V3.0.6 Release in LA! Chicago customers will soon be migrated to new portal!

We are excited to announce the latest version of our cloud portal version 3.0.6! This is only available in the LAX cloud at the moment but we will be replacing the Chicago Legacy portal (V2) with the new GCloud Portal (V3) in a few short weeks! The new system is on a completely different database for users so your Chicago login will not work in LA and your LA login will not work in Chicago. You will need to sign up as a new user if you would like to obtain an account in the other location. When we replace the Legacy portal we will automatically migrate all users in to one portal (V3).

We are also please to announce the direct connectivity between our Chicago and LA datacenters allowing your cloud servers to utilize the private network to communicate with each other!

Another major feature released is the highly requested API. The API is now fully functional and can be setup through the LAX portal and once everyone is migrated to the new portal the API will still function the same.

On this new release there has been many enhancements in stability and features. We have attached a full changelog to the bottom of this communication.

On the prior release there was only a small choice of Operating Systems. We have now added many more operating systems listed below.

List of available operating systems in LAX Version 3.0.6:

- Arch Linux 64/32 (06-2011)
- CentOS 5.6 64/32
- ClearOS 5.1 64bit
- Debian 5.08 64/32
- Debian 6.01 64/32
- Ubuntu 10.0.4 64/32
- Ubuntu 8.04 64/32
- VirtualMin 64bit
- Windows 2003 R2 enterprise 64bit
- Windows 2008 R2 enterprise 64bit

* If you need CPanel simply install one of CPanel’s supported operating systems above and place a ticket to have the latest CPanel installed. Pricing for CPanel on the cloud is $10.00/month.

The Newest Release in production is 3.0.6 coming from 3.0.2 so here is a list of changes we made to the system since 3.0.2:

CCS 3.0.3
—–
- Feature: Add Forgotten Password Functionality to Client Login
- Feature: VMDep Rebuild Initial Architecture and Coding #93
- Feature: Show last individual to update support ticket #90
- Feature: Virtualmin 64-bit conversion #177
- Feature: CentOS 64-bit conversion #176
- Feature: VDI Resize for Deployment #175
- Fix: Task Checker Spawns Endless Loops #183
- Fix: Disk/fs resize to proper size #173
- Fix: Can create VMs as small as 128MB RAM and 4GB Disk #105
- Fix: Cores are not reduced after deployment on Windows 2003 #74
- Fix: remove 100 char limit on news comments #157
- Fix: Remove “Close Ticket” as the default for response posts #149
- Fix: News System Has CSS Overlow #136
- Fix: VM Deletion Issues Graceful Shutdown #56

CCS 3.0.4
—–
- Feature: Windows 2003 Support #184
- Feature: Debian Support 64-bit #16
- Feature: Debian Support 32-bit #15
- Feature: Adding customer services #154
- Feature: Reformat VM Creation Page #180
- Feature: Daily Status Digest Email #179
- Feature: Manage Service per client #178
- Feature: Min and max credit purchase editing #174
- Feature: Ability to set client discount #131
- Fix: HTTP Code 418 Should not be Used for Error Handling #182

CCS 3.0.5
—–
- Feature: Windows 2008 Support #45
- Feature: Ubuntu 10 64-bit Support #20
- Feature: Ubuntu 10 32-bit Support #19
- Feature: Ubuntu 8 64-bit Support #18
- Feature: Ubuntu 8 32-bit Support #17
- Feature: Arch Linux Support #201
- Feature: Debian 6 64-bit Support #194
- Feature: Debian 6 32-bit Support #193
- Feature: ClearOS 5.1 Support #200
- Feature: Identify credit card IIN issuer #204
- Feature: cman should use Debian Live instead of GRML #158
- Feature: Refund queing #192
- Feature: Projected annual revenue on admin dashboard #190
- Feature: Quickbooks integration ported to OpenLSS #33,#32
- Feature: MTD payments breakdown for dashboard #190
- Fix: Payments are recorded as checks in Quickbooks #31
- Fix: Changing the way we do Startup Scripts in Linux #195
- Fix: Clients link of refunds list does not work #191
- Fix: Inability to add negative credits #107

CCS 3.0.6
—–
- Feature: Password reset and network Recon on Linux OS’s #207
- Feature: Port reseller portal to V3 #186
- Feature: Windows 2008 modify #239
- Feature: Windows 2003 Password Reset #235
- Feature: Windows 2003 Network Recon #234
- Feature: Windows 2008 Password Reset #232
- Feature: Windows 2008 Network Recon #231
- Feature: New startup harness for Arch Linux #229
- Feature: Change startup to new harness CentOS, Virtualmin, ClearOS #199
- Feature: Change startup to new harness Ubuntu 10 #198
- Feature: Change startup to new harness Ubuntu 8 #197
- Feature: Change startup to new harness Debian 5 #196
- Feature: Deprecation of script_windows for deployments #226
- Feature: Windows 2003 using ntfscp for file copying #225
- Fix: Ledger cleanup in admin panel #228
- Fix: Remove mass storage drivers and entries from Windows 2003 #227
- Fix: Client portal must pass W3C validation #206
- Fix: URL classes should output W3C friendly URLs #205
- Fix: Ticket response typing not working correctly #152

For those interested, the reseller panel that is available now in V2 will be available next week for V3 in LA.

Tilera Announces the Release of its 'Ultimate' Cloud Computing Processor

US processor designer Tilera recently announced the release of its new cloud computing processor which is seen as a major competitor to the Intel x/86 / x64 general purpose processor. The company claims that it is the ultimate Cloud Computing processor. Tilera's computing processor, TILE-Gx™ 3000 processor family is specifically designed for today’s most common cloud computing applications. Co-developed with some of the world’s largest Internet brands, the TILE-Gx 3000 processors are optimized for cloud datacenters. The TILE Gx 3000 family, is based around a 64-bit core, and comes in 36, 64 and 100-core versions. The company is expecting that it will consume 80% lower Wattage than Intel’s recently announced 'Sandy Bridge Xeon,' which is claimed to have a 10x performance per Watt advantage over the Intel device.

Tilera’s Director of Marketing, Ihab Bishara, stated “that the company has been working with many of the leading Cloud Computing businesses on the development of the 3000 processor family.” The processors target scale-out data centers, particularly high throughput-oriented applications. These include web applications that require high throughput processing and low latency, database applications and the increasingly important in-memory databases, which require high-memory throughput and storage but in turn offer much faster and richer performance. Each core includes 32 kilobytes (kB) of L1 I-cache, 32 kB of L1 D-cache and 256 kB L2 cache, with up to 32 megabytes L3 coherent cache across the device. Processor utilization is optimized using advanced memory stripping that utilizes up to 4 integrated 72-bit DDR3 memory controllers that support up to one terabyte (TB) total capacity. The TILE-Gx 3000 series integrates smart NIC hardware for pre-processing, load balancing, and buffer management of incoming traffic. As well, the TILE architecture is supported with the latest Linux release (2.6.36) enabling users to get their Linux-based applications up and running easily.

Tilera highlights the processor's features by comparing it to Intel's processors. For instance, a single 36-core device can replace a single-socket Sandy Bridge 8-core server, while a 64-core device will replace a dual-socket 8-core Sandy Bridge-based server. In addition, the 100-core device will replace a quad-socket 8-core Sandy Bridge server. Bishara notes that all "provide better performance while consuming much less power."

Bishara noted “the suitability of the processors for tasks such as analytics and data mining applications.” Full support is provided for the standard languages used on building cloud applications, including ANSI C/C++, Java, PHP, Perl and Python. The advanced programming frameworks supported include Erlang, TBB, and open MP, while the standard management protocols available include IPMI 2.0, SNMP, Syslog, Telnet, SSH, TFTP, FTP and SCP. The first of the TILE-Gx family of processors, the 36-cores device, will be sampling in July of 2011, with the 64 and 100-core devices available early in 2012.

There is a global industry trend towards providing IT infrastructure as a 'service.' Auckland company, Softsource, who employs a staff of 42, is planning to heavily invest in 'cloud computing' by building a $10 million data center in Albany, NY, so that it can rent computer servers and storage to businesses around the country on a "pay-as-you-go" basis. All of the computer equipment for the data center has been supplied by Hewlett-Packard.

General manager Pablo Garcia-Curtis said, “a distinguishing feature between the facility and larger centers recently commissioned by IBM and Datacom was that customers would be able to increase or decrease the amount of infrastructure they rented within 15 to 30 minutes through a self-service online portal.” For instance, “businesses will have the ability increase capacity to process accounts at the end of the financial year, without having equipment sitting idle during the rest of the year.”

In March, Hewlett-Packard announced a plan to build a $60 million data center in the Waikato town of Tuakau. Early customers of the data center include DB Breweries, Genesis Energy, Navico and Specsavers. Mr. Garcia-Curtis said, “Softsource might install wind turbines and solar panels to increase its 'green' credentials.”

According to the study entitled 'The State of the Data Center' that was conducted by AFCOM, a global association of data centers, data centers have expanded rapidly in recent years. The study found, “approximately 44.2 percent of data centers occupy more floor space than they did three years ago. An additional 49.4 percent are either currently expanding or have plans to do so in the near future.” The study also revealed that cloud computing has played an important role in the expansion of data centers worldwide

Jill Yaoz, CEO of AFCOM, said that “change and technological innovations are now an accepted part of day-to-day life, making it important for data center managers to be able to adapt to the new technologies and directions emerging in the industry.” She noted, “one of the most interesting changes our survey illustrates is the continued transition to the cloud. When we last did this survey in October 2009, very few data centers were even interested in the cloud, let alone actually adapting it. Now, however, we see that data center managers are more familiar with the risks and concepts, and cloud computing is quickly becoming a new standard of operation."

Abiquo Receives Best Cloud Management Award from Cloud Computing World Series Awards

At the Cloud Computing World Forum 2011 that took place at the Olympia Conference Center in London, England, Abiquo, the leading Enterprise Cloud Management software provider, received the Cloud Computing World Series award for Best Cloud Management Solution, beating out Hewlett-Packard and VMware. The annual Cloud Computing World Series Awards recognizes the best Cloud products and services. An independent panel of judges, including press editors, industry analysts, management consultants, and practicing IT architects and leaders, selects the winners. This award adds to the many honors Abiquo has received this year. It recently won the Citrix Synergy Award for Best Partner Solution for Virtual Datacenters. It also was recognized among "Top 100" vendors by CRN and AlwaysOn.

The judges at the Cloud Computing World Forum 2011 had to select from hundreds of nominations in each category. Wendy Perilli, vice president, global marketing for Abiquo, said, “This is the second year in a row that we were recognized for this prestigious international award, but to win in the Cloud Management category this year is very gratifying. The short listed nominees for the awards, along with other assessments of the market like the recent Forrester private cloud report, suggests the top three in the market are firming up. We are very pleased to be cited consistently in that group, and in this case to come out on top."

Abiquo was a finalist in two award categories - Best Cloud Management Solution and Best Virtualization Product or Service. Abiquo received the award for Best Cloud Management Solution because the product is the most complete enterprise-class cloud management solution that is available today. With Abiquo's enterprise cloud management software, organizations can manage a globally deployed computing infrastructure, comprising unlimited physical and Cloud resources including private, public and hybrid Clouds through a "single pane of glass." Abiquo's solution offers such features as support for all six major hypervisors, the ability to convert virtual machines from one hypervisor to another in a single drag-and-drop operation, and comprehensive business policy, to ensure corporate goals such as utilization and energy saving, as well as security, compliance and governance are consistently met.

Abiquo is a leading Enterprise Cloud Management software company. Abiquo assists organizations of all sizes to improve business agility, mitigate risk, and reduce costs. With Abiquo, organizations can use business policy to manage an entire, globally deployed computing infrastructure, comprising unlimited physical and Cloud resources including private, public and hybrid Clouds through a "single pane of glass".

SOURCE: Abiquo