MapReduce with Hadoop Streaming in bash – Part 3

Share Button

In our first MapReduce with Hadoop Streaming in bash article, we took a collection of Stephen Crane poems and used a MapReduce job to calculate ‘term frequency’–meaning we counted the number of times each word in the collection appeared in the collection. In the second part, we calculated ‘document frequency’ by counting the number of documents each word appears in using results from the first job.

For this final part, we will use the term frequency and document frequency to build the final Term Frequency/Inverse Document Frequency (TF-IDF) score. To do this, we need to fill our results into the TF-IDF algorithm.

TF-IDF

This algorithm shows that TF-IDF equals the Term Frequency times the natural logarithm of total documents divided by document frequency. So for each term/file combination, we need to calculate the TF-IDF based on the values provided by our last MapReduce job. Some people prefer to use base 10 logarithm to dampen down the results–I’ll cover this in the Mapper section.

Setup

So the first thing I’m going to do is get the output from the last job. You should be used to this by now.

[training@localhost steve]$ hadoop fs -get crane_out2/part-00000

The next thing I’m going to do is cheat. See, the algorithm requires the total number of documents that we’ve been analyzing. Sure, I could write a MapReduce job that looks through our latest output and emits a list of unique files; however, this is very inefficient and a waste of resources. Using a simple ‘ls’ command with a glob is much more efficient and makes better use of our (pseudo)cluster. To figure out our total document count, we’ll do just that:

[training@localhost steve]$ hadoop fs -ls crane | tail -n +2 | wc -l
8

For our testing we’ll just explicitly set this as a variable. In the Hadoop job we’ll pass it in as a parameter.

On to the Mapper

So let’s go ahead and do our final calculation using the Mapper.

[training@localhost steve]$ cat maptfidf.sh 
#!/bin/bash

while read term file tf df; do
  TFIDF=$(echo $N $df $tf | awk '{print $3 * log($1/$2)}')
  printf "%s\t%s\t%s\n" "$term" "$file" "$TFIDF"
done

Simpler than you thought? Let’s look at what we did.

  1. Read each line into the variables term, file, tf, and df. These represent (from our last job) a unique term/document combination, the number of times the term appeared in that document (tf), and the number of documents the term appears in (df).
  2. Calculate TFIDF using awk. We do this by passing total documents ($N, calculated with the variable I mentioned in the setup), document frequency, and term frequency. TF times log(total/DF) is the final answer.
  3. Print the final output as Term, File, and TF-IDF. Term and File make up the unique key for each line of output.

This is actually our final result. This is exactly what we’ve been trying to calculate and the product of our three jobs. As I mentioned in the intro to this article, some people prefer to use log10() instead of natural logarithm (which uses the constant e) to dampen the results like this:

TF-IDF 10

If that’s the case, you can replace the awk line in the Mapper with this one:

  TFIDF=$(echo $N $df $tf | awk '{print $3 * (log($1/$2)/log(10))}')

Let’s test it our in the shell, first setting the ‘N’ variable required for the algorithm:

[training@localhost steve]$ export N=`hadoop fs -ls crane | tail -n +2 | wc -l`
[training@localhost steve]$ cat part-00000 | ./maptfidf.sh | head -6
a	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0

Not too descriptive with all those 0’s, but if you know how TF-IDF works you know it is working. The lower the value, the less relevant that word is. The letter ‘a’ appears in all 8 documents, making it a very irrelevant word. Usually words like ‘a’ or ‘and’ or ‘the’ would have been filtered out in the beginning via a stoplist.

What Reducer?

Screen Shot 2013-02-10 at 5.28.01 PMSince the Mapper produced our final output, we actually don’t need to worry about a reducer. We could specify no reducer (-reducer NONE in the options) but instead we’ll use something called the IdentityReducer. An IdentityReducer means that we want the reducer to take its input and just output it naturally with no calculation. This accomplishes two things: 1) the data is sorted/shuffled when it’s sent to the reducer, so with a single reducer it should come out sorted, and 2) we will get a single output file instead of one per mapper which is easier to work with later.

So you get a break this time. No reducer. Sweet!

Our Hadoop Command

[training@localhost steve]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar \
-D stream.num.map.output.key.fields=2 -D N=`hadoop fs \
-ls crane | tail -n +2 | wc -l` -input crane_out2 \
-output tfidf -mapper /home/training/steve/maptfidf.sh \
-reducer org.apache.hadoop.mapred.lib.IdentityReducer

Just as in the previous articles, the backslashes are just there to show this is a multiline command. If you put it all on one line you don’t need them.

So a few things to note here. First, we set the stream.num.map.output.key.fields variable to 2. Even though we don’t have a formal reducer, we still want to tell the job the key field count so it will sort properly. Second, we set a new variable (-D is required for each one) called ‘N’ to the result of an ‘ls’ command in Hadoop against our original document folder. This variable will be expressed as bash variable inside our shell script and denotes the total document count. The third thing is the -reducer setting. To use the identity reducer, set it to org.apache.hadoop.mapred.lib.IdentityReducer.

Running this command gives us the final job output and the save to the ‘tfidf’ folder under HDFS:

[training@localhost steve]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar -D stream.num.map.output.key.fields=2 -D N=8 -input crane_out2 -output tfidf -mapper /home/training/steve/maptfidf.sh -reducer org.apache.hadoop.mapred.lib.IdentityReducer
packageJobJar: [/tmp/hadoop-training/hadoop-unjar6684831878608134041/] [] /tmp/streamjob5486308040698764550.jar tmpDir=null
13/10/01 07:33:26 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/10/01 07:33:26 WARN snappy.LoadSnappy: Snappy native library is available
13/10/01 07:33:26 INFO snappy.LoadSnappy: Snappy native library loaded
13/10/01 07:33:26 INFO mapred.FileInputFormat: Total input paths to process : 1
13/10/01 07:33:26 INFO mapred.JobClient: Running job: job_201309292255_0066
13/10/01 07:33:27 INFO mapred.JobClient:  map 0% reduce 0%
13/10/01 07:33:32 INFO mapred.JobClient:  map 100% reduce 0%
13/10/01 07:33:35 INFO mapred.JobClient:  map 100% reduce 100%
13/10/01 07:33:36 INFO mapred.JobClient: Job complete: job_201309292255_0066
13/10/01 07:33:36 INFO mapred.JobClient: Counters: 33
13/10/01 07:33:36 INFO mapred.JobClient:   File System Counters
13/10/01 07:33:36 INFO mapred.JobClient:     FILE: Number of bytes read=24244
13/10/01 07:33:36 INFO mapred.JobClient:     FILE: Number of bytes written=421188
13/10/01 07:33:36 INFO mapred.JobClient:     FILE: Number of read operations=0
13/10/01 07:33:36 INFO mapred.JobClient:     FILE: Number of large read operations=0
13/10/01 07:33:36 INFO mapred.JobClient:     FILE: Number of write operations=0
13/10/01 07:33:36 INFO mapred.JobClient:     HDFS: Number of bytes read=22438
13/10/01 07:33:36 INFO mapred.JobClient:     HDFS: Number of bytes written=23586
13/10/01 07:33:36 INFO mapred.JobClient:     HDFS: Number of read operations=3
13/10/01 07:33:36 INFO mapred.JobClient:     HDFS: Number of large read operations=0
13/10/01 07:33:36 INFO mapred.JobClient:     HDFS: Number of write operations=2
13/10/01 07:33:36 INFO mapred.JobClient:   Job Counters 
13/10/01 07:33:36 INFO mapred.JobClient:     Launched map tasks=1
13/10/01 07:33:36 INFO mapred.JobClient:     Launched reduce tasks=1
13/10/01 07:33:36 INFO mapred.JobClient:     Data-local map tasks=1
13/10/01 07:33:36 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=5617
13/10/01 07:33:36 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=3000
13/10/01 07:33:36 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/10/01 07:33:36 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/10/01 07:33:36 INFO mapred.JobClient:   Map-Reduce Framework
13/10/01 07:33:36 INFO mapred.JobClient:     Map input records=326
13/10/01 07:33:36 INFO mapred.JobClient:     Map output records=326
13/10/01 07:33:36 INFO mapred.JobClient:     Map output bytes=23586
13/10/01 07:33:36 INFO mapred.JobClient:     Input split bytes=108
13/10/01 07:33:36 INFO mapred.JobClient:     Combine input records=0
13/10/01 07:33:36 INFO mapred.JobClient:     Combine output records=0
13/10/01 07:33:36 INFO mapred.JobClient:     Reduce input groups=326
13/10/01 07:33:36 INFO mapred.JobClient:     Reduce shuffle bytes=24244
13/10/01 07:33:36 INFO mapred.JobClient:     Reduce input records=326
13/10/01 07:33:36 INFO mapred.JobClient:     Reduce output records=326
13/10/01 07:33:36 INFO mapred.JobClient:     Spilled Records=652
13/10/01 07:33:36 INFO mapred.JobClient:     CPU time spent (ms)=840
13/10/01 07:33:36 INFO mapred.JobClient:     Physical memory (bytes) snapshot=199655424
13/10/01 07:33:36 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=776904704
13/10/01 07:33:36 INFO mapred.JobClient:     Total committed heap usage (bytes)=176492544
13/10/01 07:33:36 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
13/10/01 07:33:36 INFO mapred.JobClient:     BYTES_READ=22330
13/10/01 07:33:36 INFO streaming.StreamJob: Output directory: tfidf

The Final Results

After three MapReduce jobs, we’re finally ready to see our word/document and associated TF-IDF. Score! (ba dum tss)

[training@localhost steve]$ hadoop fs -cat tfidf/part-00000
a	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0
a	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0
accosted	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
achieved	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.07944
addressed	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
again	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.07944
ages	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
agony	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
ah	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	1.38629
ah	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	1.38629
already	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
am	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
and	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	0.267063
and	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	0.400594
and	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	0.400594
and	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	0.133531
and	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0.267063
and	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0.267063
and	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0.133531
another's	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
are	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
as	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
at	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
aye	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.38629
aye	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
ball	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	8.31777
bawled	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
been	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
before	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
began	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
believed	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
black	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	1.38629
black	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	1.38629
blind	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
book	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	4.15888
boys	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
breath	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	4.15888
but	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	1.38629
but	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	1.38629
by	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.38629
by	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	5.54518
called	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
calling	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	4.15888
can	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
cavern	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
child	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	4.15888
chronicle	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
clay	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.07944
climbed	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.07944
collection	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	4.15888
concentrating	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
court	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
created	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	2.07944
crevice	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
cried	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	1.38629
cried	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	2.77259
crowd	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
crowned	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
cuddle	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
curious	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
dead	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
death	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
deathslime	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
denial	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
desert	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	6.23832
dire	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
disturbed	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
earth	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.07944
echoes	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
error	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
eternal	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
even	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
eventually	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.38629
eventually	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	1.38629
ever	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	4.15888
every	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
exist	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	2.07944
fact	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	2.07944
families	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
feckless	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
fenceless	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
fireside	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
fleetly	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
for	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	0.980829
for	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	0.980829
for	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0.980829
fortress	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
freedom	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
from	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	0.693147
from	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	1.38629
from	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0.693147
from	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0.693147
futile	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
game	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
garment	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	4.15888
god	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	13.8629
god	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	1.38629
gold	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	8.31777
grown	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
had	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
halfinjustices	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
hand	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
hands	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
has	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	2.07944
have	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
have	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	4.15888
he	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.38629
he	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	4.15888
he	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.77259
he	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0.693147
heat	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	2.07944
heavens	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.07944
held	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	4.15888
hem	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	4.15888
highest	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
him	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.77259
him	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
his	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	1.38629
his	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
hold	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
honest	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
horizon	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	1.38629
horizon	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	1.38629
however	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	2.07944
i	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0.470004
i	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.82002
i	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	1.88001
i	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.35002
i	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	1.41001
in	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0.287682
in	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	0.287682
in	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	0.287682
in	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	0.287682
in	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	0.287682
in	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0.287682
into	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
is	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	0.575364
is	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.01377
is	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	0.287682
is	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0.287682
is	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0.575364
is	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0.575364
it	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.43841
it	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	0.287682
it	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	0.287682
it	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0.287682
it	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0.575364
it	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0.575364
its	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.77259
its	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	4.15888
joys	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
kindly	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
know	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
let	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
lie	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
life's	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
lived	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
lo	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.07944
lone	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
long	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
looked	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.07944
looks	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
loud	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
mad	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
man	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0.980829
man	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.96166
man	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	1.96166
market	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
me	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0.693147
me	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
me	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0.693147
me	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0.693147
melons	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
men	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	4.15888
merciful	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
met	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
mighty	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
mile	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	4.15888
million	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
mocked	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
much	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	4.15888
never	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	1.38629
never	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.77259
newspaper	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	10.3972
night	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
no	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	1.38629
no	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	2.77259
not	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	1.38629
not	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
now	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	4.15888
obligation	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	2.07944
of	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0.287682
of	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.15073
of	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	1.43841
of	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	0.863046
of	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	0.575364
of	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0.575364
often	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
on	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
one	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
opened	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
opinion	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
part	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	4.15888
phantom	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	4.15888
place	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	2.07944
plains	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
player	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
pursued	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
pursuing	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
ran	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
read	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
remote	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
replied	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	2.07944
roaming	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
rock	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
round	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	4.15888
said	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0.470004
said	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	0.470004
said	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0.470004
said	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0.940007
said	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0.940007
sand	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	2.07944
saw	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.38629
saw	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	1.38629
scores	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
screamed	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
second	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
seer	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
sells	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
sense	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	2.07944
shadow	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	4.15888
should	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
sir	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	1.38629
sir	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.77259
skill	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
sky	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.38629
sky	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	1.38629
smiled	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
smote	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
sneering	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
so	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
space	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
spaces	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
sped	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.77259
sped	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	1.38629
spirit	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
spreads	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
spurred	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
squalor	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
strange	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.77259
strange	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
stupidities	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
suddenly	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
swift	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
sword	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
symbol	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
take	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	2.07944
tale	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
tales	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
that	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	1.38629
that	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	4.15888
the	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0
the	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	0
the	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	0
the	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	0
the	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	0
the	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0
the	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0
the	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0
their	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
then	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	1.38629
then	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
there	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.38629
there	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	1.38629
they	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.07944
think	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	2.07944
this	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.96166
this	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0.980829
this	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	0.980829
through	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	1.38629
through	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.77259
to	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	0.693147
to	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	0.693147
to	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	1.38629
to	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
touched	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	4.15888
tower	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
traveller	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	6.23832
tried	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
truth	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	6.23832
unfairly	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
unhaltered	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
universe	hdfs://0.0.0.0:8020/user/training/crane/a_man_said_to_the_universe.txt	4.15888
vacant	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	2.07944
valleys	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.07944
victory	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
voice	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	4.15888
walked	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	2.07944
was	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	2.77259
was	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	0.693147
was	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	0.693147
was	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	0.693147
well	hdfs://0.0.0.0:8020/user/training/crane/walked_in_a_desert.txt	2.07944
went	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.38629
went	hdfs://0.0.0.0:8020/user/training/crane/a_spirit_sped.txt	2.77259
when	hdfs://0.0.0.0:8020/user/training/crane/a_man_saw_a_ball_of_gold.txt	1.38629
when	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	1.38629
whence	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	2.07944
where	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	6.23832
which	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	1.38629
which	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
while	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	4.15888
wind	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	4.15888
wins	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	2.07944
wisdom	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	1.38629
wisdom	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
world	hdfs://0.0.0.0:8020/user/training/crane/a_newspaper.txt	1.38629
world	hdfs://0.0.0.0:8020/user/training/crane/truth.txt	1.38629
you	hdfs://0.0.0.0:8020/user/training/crane/met_a_seer.txt	1.38629
you	hdfs://0.0.0.0:8020/user/training/crane/pursuing_the_horizon.txt	2.77259

Just as in our test, “a” is not important at all as it appears in 8 documents so it came out with a score of 0. “Where” seems very important for such a common word. But it turns out that is because it shows up 3 times in only 1 file. Remember, TF-IDF is “a numerical statistic which reflects how important a word is to a document in a collection or corpus”. Take a look at words like “you” at the end of the file–it shows up in two different files, but has a different TF-IDF weight for each one. That’s because “you” only appears once in the first file but twice in the second file, making it more important to that document in relation to the whole corpus. You can see this with ‘grep’ commands against the original content.

[training@localhost steve]$ grep -i you crane/met_a_seer.txt 
Of that which you hold. 
[training@localhost steve]$ grep -i you crane/pursuing_the_horizon.txt 
"You can never -- " 
"You lie," he cried,

And with that, we just built an index. If we were to build a search engine against those 8 Stephen Crane poems, then a search for a word would output file ordered by TF-IDF descending. That way the most pertinent (keyword rich) files would come first on the results.

Conclusion

Of course, we could have done this project a lot easier with tools like Lucene and Mahout. They are of course made for this sort of thing, and have a ton of extra features including automatic stoplisting, weight tuning, etc. But it wouldn’t be nearly as fun, right?

This concludes our TF-IDF with Hadoop Streaming in bash exercise. If you have any feedback on better ways to do these tasks (or errata) please let me know in the comments!

Share Button

2 comments

  1. Hi, I don’t know why you can do pipe in hadoop streaming. Whenever I do piping at the reducer, hadoop return fail code with broken pipe.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.