AWS-Big Data Demo(Word Count)
In The AWS Management Console,click Services then click-> S3,
Click Create Bucket
In the Bucket Name field,type "emr-agoutam"
Select Region
Click Create
Creating and Launching a MapReduce Cluster
In the AWS Management Console, -> Click Services and Click ->EMR
Create cluster ->Go to Advanced Options
On Clciking Advanced Options ,Advanced Cluster Configuration page, Click Configure sample application
From the Select Sample application drop-down list,Select Wordcount
In the Output location field,enter :s3://emr-agoutam/output/
Check box logging enabled should be selected
In the Logging section, enter s3://emr-agoutam/logs/
Leave Debugging enabled.
In the Tags section Type Owner in the key column.
In the Value column ,type Development
In the Software Configuration section ,Amazon is selected as a default .
Choose EMR 4.0.0.(Hadoop 2.6)
Configure Master ,core and Task Node as Mentioned Below
Master m3.xlarge count=1
Core m3.xlarge count =2
Task m3.xlarge count =0
EC2 key Pair ,select Proceed without an EC2 Key pair.
Under Security and Access /IAM Roles,select the default radio button .
Towards the bottom of the page,select Yes option for Auto-terminate
Auto terminate :
Determines what the cluster does after its last step,
Yes means the cluster auto-terminates after the last step completes.
No means the cluster runs untill you manually terminate it.
Review the Configuration and ,click Create cluster
Monitor the Cluster
Click Cluster List. This shows a list of clusters and their status .
Click the arrow to the left of cluster name ("word count")
Click Monitoring Tab Below to see Cluster running Process
Once the EMR job is completed ,A status of Terminated .All Steps completed
Status changes to terminated All steps completed
In The AWS Management Console,click Services then click-> S3,
Click In The AWS Management Console,click Services then click-> S3,
Click in front of Bucket "emr-agoutam" and click on folder name output to see results
Click in front of Bucket "emr-agoutam" and click on folder name logs to see LOGS
No comments:
Post a Comment