MapReduce 編程 系列四 MapReduce例子程序運行
來源：程序員人生 發布時間：2014-10-02 08:00:00 閱讀次數：3364次
MapReduce程序編譯是可以在普通的Java環境下進行，現在來到真實的環境上運行。
首先，將日志文件放到HDFS目錄下
$ hdfs dfs -put *.csv /user/chenshu/share/logs/
14/09/27 17:03:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicab
le
$hdfs dfs -ls /user/chenshu/share/logs
14/09/27 17:03:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicab
le
Found 10 items
-rw-r--r-- 3 chenshu chenshu 539026 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-05-10.0.csv
-rw-r--r-- 3 chenshu chenshu 33212 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-05-12.0.csv
-rw-r--r-- 3 chenshu chenshu 1117191 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-05-13.0.csv
-rw-r--r-- 3 chenshu chenshu 2642634 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-05-18.0.csv
-rw-r--r-- 3 chenshu chenshu 4676438 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-06-01.1.csv
-rw-r--r-- 3 chenshu chenshu 633015 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-06-27.0.csv
-rw-r--r-- 3 chenshu chenshu 4749439 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-07-01.1.csv
-rw-r--r-- 3 chenshu chenshu 1551312 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-08-01.2.csv
-rw-r--r-- 3 chenshu chenshu 2957316 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-08-11.0.csv
-rw-r--r-- 3 chenshu chenshu 4032863 2014-09-27 17:03 /user/chenshu/share/logs/sign_2014-09-01.1.csv
同時還要確保output目錄不存在，該目錄會被MapReduce程序創建，用于存放輸出結果：
$ hdfs dfs -rm -r /user/chenshu/share/output
然后運行程序，觀察輸出結果
[chenshu@hadoopMaster example1]$ hadoop jar target/mr1_example1-1.0-SNAPSHOT.jar org.freebird.LogJob /user/chenshu/share/logs /user/chenshu/share
/output
14/09/27 17:57:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicab
le
14/09/27 17:57:42 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/09/27 17:57:42 INFO input.FileInputFormat: Total input paths to process : 10
14/09/27 17:57:46 INFO mapred.JobClient: Running job: job_201404261903_548924
14/09/27 17:57:47 INFO mapred.JobClient: map 0% reduce 0%
14/09/27 17:58:11 INFO mapred.JobClient: map 40% reduce 0%
14/09/27 17:58:12 INFO mapred.JobClient: map 80% reduce 0%
14/09/27 17:58:20 INFO mapred.JobClient: map 100% reduce 0%
14/09/27 17:58:25 INFO mapred.JobClient: map 100% reduce 8%
...
14/09/27 17:59:12 INFO mapred.JobClient: Reduce input records=163643
14/09/27 17:59:12 INFO mapred.JobClient: Reduce input groups=162631
14/09/27 17:59:12 INFO mapred.JobClient: Combine output records=163643
14/09/27 17:59:12 INFO mapred.JobClient: Physical memory (bytes) snapshot=6787436544
14/09/27 17:59:12 INFO mapred.JobClient: Reduce output records=162631
14/09/27 17:59:12 INFO mapred.JobClient: Virtual memory (bytes) snapshot=43714277376
14/09/27 17:59:12 INFO mapred.JobClient: Map output records=354282
現在將output目錄都復制到本地磁盤，查看結果：
hdfs dfs -get /user/chenshu/share/output/
然后查看本地output目錄：
[chenshu@hadoopMaster output]$ ll -alh
total 4.3M
drwxrwxr-x 3 chenshu chenshu 4.0K Sep 27 18:02 .
drwxrwxr-x 6 chenshu chenshu 4.0K Sep 27 18:03 ..
drwxrwxr-x 3 chenshu chenshu 4.0K Sep 27 18:02 _logs
-rw-r--r-- 1 chenshu chenshu 286K Sep 27 18:02 part-r-00000
-rw-r--r-- 1 chenshu chenshu 290K Sep 27 18:02 part-r-00001
-rw-r--r-- 1 chenshu chenshu 284K Sep 27 18:02 part-r-00002
-rw-r--r-- 1 chenshu chenshu 284K Sep 27 18:02 part-r-00003
-rw-r--r-- 1 chenshu chenshu 289K Sep 27 18:02 part-r-00004
-rw-r--r-- 1 chenshu chenshu 284K Sep 27 18:02 part-r-00005
-rw-r--r-- 1 chenshu chenshu 290K Sep 27 18:02 part-r-00006
-rw-r--r-- 1 chenshu chenshu 284K Sep 27 18:02 part-r-00007
-rw-r--r-- 1 chenshu chenshu 285K Sep 27 18:02 part-r-00008
-rw-r--r-- 1 chenshu chenshu 286K Sep 27 18:02 part-r-00009
-rw-r--r-- 1 chenshu chenshu 290K Sep 27 18:02 part-r-00010
-rw-r--r-- 1 chenshu chenshu 288K Sep 27 18:02 part-r-00011
-rw-r--r-- 1 chenshu chenshu 287K Sep 27 18:02 part-r-00012
-rw-r--r-- 1 chenshu chenshu 281K Sep 27 18:02 part-r-00013
-rw-r--r-- 1 chenshu chenshu 286K Sep 27 18:02 part-r-00014
-rw-r--r-- 1 chenshu chenshu 0 Sep 27 18:02 _SUCCESS
打開其中一個part-r文件，可以看到結果如下：
536dbba04700aab274729ce9 2
536dbba14700aab274729cf9 2
536dbba14700aab274729cff 2
536dbba14700aab274729d02 2
536dbba14700aab274729d11 2
536dbba14700aab274729d20 2
536dbba14700aab274729d89 2
536dbba14700aab274729d8f 2
536dbba14700aab274729d98 3
536dbba14700aab274729d9e 3
536dbba14700aab274729de9 2
536dbba14700aab274729def 2
536dbba14700aab274729df8 2
536dbba14700aab274729dfe 2
536dbba14700aab274729e8e 2
536dbba14700aab274729ed9 3
536dbba14700aab274729eee 2
程序運行成功了。
所有代碼在gitlab.com上。
git@gitlab.com:hadoop/share.git
生活不易，碼農辛苦
如果您覺得本網站對您的學習有所幫助,可以手機掃描二維碼進行捐贈
