Best of breed

AWS is the best of Breed Iaas/Paas in our opinion, though there are some issues with using AWS in practice [stated below].  This case study is intended to provide some insight into how to migrate to AWS in the real world, and what issues you will encounter.  It is from an existing migration and implementation, that overall was successful in achieving, cost, IT and business objectives.  It was not pain-free however.  No IT projects are pain free. 

 

Use Case:  Distribution Services

Distribution/Wholesale firm moving business intelligence onto a Private Cloud platform, VPC connection, using AWS and the IaaS and PaaS deployment platforms.

 

ARCHITECTURE OVERVIEW

Information on clients to be deployed securely in the Cloud.  Standard 3 Tier architecture.  Purpose is to be web-enable the CRM system, which contains rich content, and multiple reports per client. It must also be highly secure.  Clients may login with appropriate IAM security.  Group rules and security were also used.

 

Within AWS the architecture utilised Amazon’s Elastic Load Balancers combined with Auto Scaling Groups to balance the traffic across multiple instances and automate the creation/termination of instances across multiple Availability Zones. A number of components were also split out from the current infrastructure to make it more modular and fault-tolerant. This provides redundancy across multiple geographic locations, while also having the ability to scale horizontally with ease. At the data-layer, AWS RDS was deployed, rather than trying to manage MySQL on custom instances.

 

DEPLOYMENT & ELASTIC BEANSTALK

Initially there was some interest in using AWS Elastic Beanstalk to manage the deployment, ELBs [load balancing], ASGs [auto scaling elasticity], logging & monitoring. Elastic Beanstalk is quite restrictive; namely when you deploy an application version it could result in the staged termination of the instances running that application, and their replacement with new instances running the updated codebase.

 

Usually you want to deploy without terminating instances in certain parts of the product. Most clients use shell scripts for setting up the base AMIs; user-data scripts that are automatically run whenever a new instance comes online and fabric scripts to automate deployment across multiple instances.

 

VERSION CONTROLLING YOUR INFRASTRUCTURE

By doing the above you will control part the versioning of the infrastructure. You can create a git repo with all of the relevant setup & configuration scripts, along with any relevant documentation etc. After a while you will have some detailed information on how to recreate our infrastructure from scratch.  Combining this with CloudFormation allows you to take this to another level, where you can template and automate the creation of almost all AWS services.

 

THE DATA LAYER & AMAZON AURORA

RDS was used for the data-layer.  AWS Aurora is a MySQL-compatible RDBMS which was designed from the ground-up to run on AWS, and has a number of benefits over MySQL running on RDS.  AWS claims that Aurora will give you a 5x benefit in throughput vs MySQL on RDS, but this is untrue.  It is however more durable and redundant. 

 

However, Aurora patching occurs every month, and this will result in a restart of your instance. There is no workaround and hopefully will be fixed soon.

 

GETTING OUR APPLICATION READY FOR AWS

Migrating from a traditional architecture, means you have some work to get your application “cloud ready”. This will typically involve making sure that the application can run on stateless instances, and a switch in thinking to assume that local storage is ephemeral. This means shifting any local file storage to S3, using Elasticache for sessions & caching, and updating our queue, among other changes.

 

THE MIGRATION PROCESS

Given the large volume of rich data and extremely large number of clients, the migration of data had to proceed in steps. To allow this you can register some extra SSL certs, make updates to the admin application to manage & track the state of clients and export/import data, and add extra logic to the application to handle redirects.

 

You can migrate some groups of data, or a limited number of users, test to make sure the data has integrity and that the process of data egress and ingress is handled properly. 

 

THE BENEFITS IN MIGRATING TO AWS

A number of benefits exist in migrating to AWS, these are some of the key ones:

 

Uptime
Significant improvement, which can monitored with 3rd party tools and confirmed.

 

No single point of failure

 

Ability to innovate
Being on AWS opens up an array of new services and technologies that are now significantly more accessible to the average IT group. Whether it’s looking at new storage engines like Redshift or services like Amazon Machine Learning or Lambda, the time to implement – and therefore innovate – is significantly reduced.

 

Integration to other systems
Being on AWS opens up new possibilities to integrate including best of breed tools & services to connect to other systems.

 

Scale
Elasticity, load balancing and the capability to scale automatically is a big bonus. 

 

PROBLEMS YOU MIGHT RUN INTO

 

Elastic Beanstalk

This just isn’t flexible in how it handles deployments.

 

Insert performance on write heavy applications

There could be a performance issue when doing a large volume of inserts.  Most apps import data from other sources.  Parsing information in a high valuable – for example importing around 3m transactions – took 7 times longer on Aurora.  There is a parameter within Aurora “innodb_flush_log_at_trx_commit”. Turning this off dropped the time on Aurora to around twice what it was on the old infrastructure. 

 

Query indexing

There might be some differences in the way indexes are chosen when you migrate to Aurora.  An answer is to use FORCE INDEX in a few specific places within the migrated MySQL data.

 

EC2 Swap Space

Swap space is not enabled by default on EC2 instances running Amazon Linux. You have to update the base-AMIs to include swap space.

 

Backup Policies

Backup options for RDS are a little confusing.  At a base-level, you can just turn on automatic nightly snapshots and you have instant cover for a 35-day period, with incremental rollback to any point in time. While this is very easy to implement and a great start, there are a few limitations to it.

 

 

 

 

 

You can implement your own backup solution on top of the automated snapshots. This involves taking nightly SQL dumps for all clients, encrypting them and storing them on S3 with specific retention periods set.

 

COST COMPARISON WITH PREVIOUS INFRASTRUCTURE

AWS is complicated and it can be notoriously difficult to predict your costs accurately.

 

Costs are approximately the same for many migrations and applications. However, you will likely have a vastly superior infrastructure, with all single points of failure eliminated. This will include significantly more power and the ability to scale.

 

KEY RECOMMENDATIONS