Tag Archives: rhadoop

Installing RHadoop on a CentOS 7 Server

I spent a week struggling with trying to find a way of integrating Hadoop and R. I started with RHide and never could get it to work. Trying to get RHadoop installed and working took a tremendous amount of effort. Since I assume there are others in the same boat, I’d sharing what I found here. I can say for certain that this works on a CentOS 7 Server with a single node Hadoop installation. I did get this installed on a 3-node installation as well. But since I started this process at the same time I worked on upgrading Hadoop, neither of which are completed yet, I make no promises for that scenario.

I will be trying this process on Ubuntu 14 as well and will share my insights in a later post.

RHide is a product provided for free from Revolutionary Analytics. It consists of 5 different applications:

  • ravro
  • plyrmr
  • rmr
  • rhdfs
  • rhbase

This process will show installation of all 3.

Step 1: Install the necessary libraries

Use yum install to add the following:

  • automake
  • libtool
  • flex
  • bison
  • pkgconfig
  • gcc-c++
  • boost-devel
  • libevent-devel
  • zlib-devel
  • python-devel
  • ruby-devel
  • libcurl-devel

Follow the apache Thrift page instructions to install:

  • autoconf
  • automake
  • bison
  • boost

You will need the versions listed in those instructions and yes, install them even if you have them installed via yum. Certain libraries don’t get installed with the yum versions. If you accept the installation defaults, these versions will install under /usr/lib; they won’t interfere with other versions you may have installed via yum.

Once those libraries are installed, you will need to soft link everything in /usr/local/include/thrift to /usr/local/include. You may be better at changing the flags so that the R libraries would see them, but I couldn’t make it work.

You also need to copy /usr/share/aclocal/pkg.m4 to /usr/local/share/aclocal.

Next ,install the R prerequisites. Do this from the command line as root. If you try to do it from the command line as yourself, or try to do it from the command line in RStudio as yourself, you will be the only one that can see them. Since Hadoop MapReduce jobs don’t run as you, this will cause you all sorts of headaches.

Type R from the command line. That gets you into a python-like shell. Use

install.packages(“<package name>”)

for each of the following:

  • reshape2
  • caTools
  • dplyr
  • rmr2
  • R.methodsS3
  • Hmisc
  • lazyeval
  • rjson
  • jJava
  • RJSONIO
  • itertools
  • digest
  • Rcpp
  • httr
  • functional
  • devtools
  • plyr
  • bit64

Those may not be in the correct order for installation. But I believe it is a full list.

Finally, it’s time to install the RHadoop programs. Download them from

https://github.com/RevolutionAnalytics/RHadoop/wiki/Downloads

Change to the root user:

sudo su –

Use the R CMD INSTALL to install each of them. For example:

R CMD INSTALL rhbas_1.2.1.tar.gz

Install rhbase last. I don’t think there was a particular order for the rest of them:

  • ravro
  • plyrmr
  • rmr
  • rhdfs
  • rhbase

Now, set up the following environmental variables. I do so in /etc/environment so that it’s universal. But you can also do so within .bashrc or within R itself.

Outside of R:

which hadoop #let’s you know where the command is

export HADOOP_CMD=/usr/bin/hadoop # this is where it is on my servers

find / -name “hadoop-streaming*jar”

# use the location of the -0.20- library if you’re using hadoop1

export HADOOP_STREAMING=/usr/lib/hadoop-mapreduce/hadoop-streaming.jar

Inside of R (needs to be done every time you start R):

# use the find cmd outside of R to figure out where yours is 

Sys.setenv(HADOOP_CMD=”/usr/bin/hadoop”) 

Sys.setenv(HADOOP_STREAMING=”/usr/lib/hadoop-mapreduce/hadoop-streaming.jar”)

This one can be run as soon as you complete all the above steps.  It comes from Big Data Analytics with R and Hadoop by Vignesh Prajapati

—- begin script —

library(‘rhdfs’)
hdfs.init()

library(‘rmr2’)

small.ints = to.dfs(1:10)

mapreduce(
input = small.ints,
map = function(k,v) {
lapply(seq_along(v), function(r){
x <- runif(v[[r]])
keyval(r,c(max(x), min(x)))
})})

— end script —

After I typed it in, it started running automatically.  Mr. Prajapati suggests that you may have to hit <CTRL>-<ENTER> to get it to start.

At the end of the script run, look for

Output /tmp/<filename>

Use that file to see your output:

> output <- from.dfs(“/tmp/<filename>”)
> table_output <- do.call(‘rbind’,lapply(output$val,”[[“,2))
> table_output

Here’s another example to try. This one uses GDP data that can be found here:

http://data.worldbank.org/data-catalog

The script is based on one found at:

https://bighadoop.wordpress.com

— begin script —-

library(rmr2)
library(rhdfs)

setwd(“/home/amracel/Downloads/”)

gdorig <- read.csv(“GDP.csv”)
gdp &ltp- subset(gdorig,select=c(Abbrev,Rank,Country,GDP))
gdp$GDP <- as.numeric(gsub(“,”,””,gdp$GDP))
#ad(gdp)

hdfs.init()
gdp.values <- to.dfs(gdp)
# AAPL revenue in 2012 in millions USD
aaplRevenue = 156508

gdp.map.fn <- function(k,v) {
key <- ifelse(v[4] < aaplRevenue, “less”, “greater”)
keyval(key, 1)
}

count.reduce.fn <- function(k,v) {
keyval(k, length(v))
}

count <- mapreduce(input=gdp.values,
map = gdp.map.fn,
reduce = count.reduce.fn)

from.dfs(count)

— end script —-

Good luck! Leave comments if you have any issues and I’ll see if I can help!