Speed Through installation of Flume-NG on Amazon AWS

By | June 23, 2012

This is a thunder-speed (:P) through installation of Flume NG (1.x) on Amazon AWS. I tried to make it as fast as possible. Note that for a final product, more attention should be spent on security issues.

Tools used:

  • puttygen
  • putty
  • scp
  • pssh
  • cloudera manager 3.7
  • flume-ng

Operating System:

  • Ubuntu (on my machine)
  • Suse on AWS

Quick Steps

Instaling the CDH3 and setting up the HDFS

  1. Go to Security groups and create a new group
  2. In the inbound add ssh, http, icmp (for ping). Apply Rule Changes. If its only for test purposes you can just add all tcp, udp and icmp
  3. Go to Key pairs, create and download pem file
  4. Now that you have the pem file, go to its directory and do this: (adds the keys to ./ssh/)
    1. puttygen key.ppk -L > ~/.ssh/id_rsa.pub
    2. puttygen /path/to/puttykey.ppk -O private-openssh -o ~/.ssh/id_rsa
  5. Create, for example, 10 Suse Medium instances in AWS Management Console (1 CDH Manager, 1 HDFS NameNode/DataNode, 2 HDFS DataNodes, 3 Agent, 3 Agent that work as collectors):
    1. Choose Suse, next
    2. 10 instances, Medium, next
    3. Next
    4. Use the created key pair, next
    5. Choose previously created security group
  6. Choose an AWS instance, rename it to CDHManager. Rigth click on it, connect -> copy public DNS
  7. Dowload Cloudera Manager Free Edition and copy its bin file into the the machine:
    1. scp cloudera-manager-installer.bin root@publicdns:/root/cloudera-manager-installer.bin
  8. SSH the machine and perform an ls, cloudera bin file should be there. Do the following:
    1. ssh root@publicdns
    2. ls <- check that
  9. Install it:
    1. chmod u+x cloudera-manager-installer.bin
    2. sudo ./cloudera-manager-installer.bin
    3. next, next, yes, next, yes…wait till installation finishes
  10. Go to your web browser, paste the public dns and port 7180 like this: publicDns:7180. Note that it can’t access it. Its because our security group doesn’t allow connections on this port :
    1. Go to security groups. Add custom tcp rule with port 7180. Apply rule changes.
    2. Reopen webpage. Username: admin, pass: admin
    3. Install only free. Continue.
    4. Proceed without registering.
  11. Go to the My instances in AWS and select all except the CDHManager. Notice that below all the public dns appear listed, copy them at once and paste on the webpage. Take out the not needed parts such as “i-2jh3h53hk3h:”. (Sometimes, some nodes might be not accessible, just delete/restart them, create another and put the public dns in the list)
  12. Find instances and install CDH 3 on them with default values. continue.
  13. Choose root, all accept same public key, select your public key and for private select the pem key. Install…
  14. Continue, continue, cluster CDH3
  15. if an error occur it is due to not open ports (generally icmp or other). Common error : “The inspector failed to run on all hosts.”.
  16. Add service hdfs, for example 3 datanodes, one of them can be a name node as well.

Installing flume ng

  1. On linux install pssh: sudo apt-get install pssh.
  2. Create an host.txt file with all the public dns.
  3. Install putty on ubuntu.
    1. sudo apt-get install putty.
  4. Install flume like a boss:
    1. parallel-ssh -O StrictHostKeyChecking=no -v -i -l root -h hosts.txt “sudo zypper –non-interactive install flume-ng”
  5. Make it boot from startup
    1. parallel-ssh -O StrictHostKeyChecking=no -v -i -l root -h hosts.txt “sudo zypper –non-interactive install flume-ng-agent”

For the next part I will be describing my experiment, and not really going into details on explaining what could be done with it. For that purpose I already referred to my colleague website http://www.otnira.com/2012/05/28/weekend-with-flume-part-1/. I will be showing my configuration and how I will hack Flume into performing dynamic routing. It will probably not be the best way to do it, since I will be only looking forward reliability results and not an end product.

(pic: computerhotline@flickr)

Leave a Reply