Setting Up a Hadoop Cluster

This post lists the steps to set up an Hadoop cluster in Ubuntu 11.04. Most codes can be directly copied and pasted.

* Hadoop
** Install Java

#+begin_src shell
sudo apt-get install sun-java6-jdk
sudo update-java-alternatives -s java-6-sun
#+end_src

** Add Hadoop User and Group

#+begin_src shell
sudo addgroup hadoop
sudo adduser --ingroup hadoop hadoop
#+end_src

** Configuring SSH and Password-less Login

#+begin_src sh
  # In the master node
  su hadoop
  ssh-keygen -t rsa -P ""
 
  for node in $(cat /conf/slaves);
  do
      ssh-copy-id -i $HOME/.ssh/id_rsa.pub hadoop@$node;
  done
#+end_src

** Install Hadoop
*** Install

#+begin_src sh
  ## download and install
  cd /home/hadoop/
  tar xzf hadoop-0.21.0.tar.gz
  mv hadoop-0.21.0 hadoop
#+end_src

*** Update .bashrc

#+begin_src sh
  ## update .bashrc
  # Set Hadoop-related environment variables
  export HADOOP_HOME=/home/hadoop/hadoop
  export HADOOP_COMMON_HOME="/home/hadoop/hadoop"
  export PATH=$PATH:$HADOOP_HOME/bin
  export PATH=$PATH:$HADOOP_COMMON_HOME/bin/
#+end_src

*** Update conf/hadoop-env.sh

#+begin_src sh
  export JAVA_HOME=/usr/lib/jvm/java-6-sun
  export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
#+end_src

*** Update conf/core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<!-- In: conf/core-site.xml -->
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://128.125.86.89:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>


</configuration>

*** Update conf/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<!-- In: conf/mapred-site.xml -->
<property>
<name>mapreduce.jobtracker.address</name>
<value>128.125.86.89:54311</value>
</property>

</configuration>

*** Update conf/hdfs-site.xml

#+begin_src html
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<!-- In: conf/hdfs-site.xml -->
<property>
<name>dfs.replication</name>
<value>3</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>

</configuration>
#+end_src

*** Update conf/masters (master node only)

#+begin_src sh
128.125.86.89
#+end_src

*** Update conf/slaves (master node only)

#+begin_src sh
128.125.86.89
slave-ip1
slave-ip2
......
#+end_src

*** Copy hadoop installation and configuration files to slave nodes

#+begin_src sh  
# In the master node  
su hadoop    
for node in $(cat /conf/slaves);  
do
      scp ~/.bashrc hadoop@$node:~;       scp -r ~/hadoop hadoop@#node:~;  
done
#+end_src

** Run Hadoop
*** Format HDFS

#+begin_src sh
hdfs namenode -format
#+end_src

*** Start Hadoop

#+begin_src sh
start-dfs.sh && sleep 300 && start-mapred.sh && echo "GOOD"
#+end_src

*** Run Jobs

#+begin_src sh
hadoop jar hadoop pipes
#+end_src

*** Stop Hadoop

#+begin_src sh
stop-mapred.sh && stop-dfs.sh
#+end_src

** References:
1. http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ 
2. http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
3. http://fclose.com/b/cloud-computing/290/hadoop-tutorial/
4. Fix could only be replicated to 0 nodes instead of 1 error

Install Matlab R2006b

I decide to reinstall MATLAB R2006b mostly because of a new toolbox SymBiology

SimBiology extends MATLAB with tools for modeling, simulating, and analyzing biochemical pathways. You can create your own block diagram model using predefined blocks. You can manually enter in species, parameters, reactions, rules, kinetic laws, and units, or read in Systems Biology Mark-Up Language (SBML) models. SimBiology lets you simulate a model using stochastic or deterministic solvers and analyze your pathway with tools such as parameter estimation and sensitivity analysis.

First get the following MATLAB ISO images at ftp://pxe/software/Matlab2006b (perhaps only available for LAN of USTC)
[Mathworks.Matlab].Mathworks.Matlab.R2006b.UNIX.ISO-TBE-CD1.iso [Mathworks.Matlab].Mathworks.Matlab.R2006b.UNIX.ISO-TBE-CD2.iso [Mathworks.Matlab].Mathworks.Matlab.R2006b.UNIX.ISO-TBE-CD3.iso [Mathworks.Matlab].Mathworks.Matlab.R2006b.UNIX.ISO-TBE.nfo

mount these images and enter the directory where you want to install matlab, create a matlab directory ($MATLAB).

Copy the license file from the first CD. There two license files in CD1/crack license_locked.dat license_server.dat. I copy license_locked.dat to $MATLAB and rename it license.dat. Enter $MATLAB
run CD1/install. The graphic interface is easy to complete.

When I finished the normal install and tried to run matlab. It poped a very lengthy error message java.lang.ExceptionInInitializerError at com.mathworks.mde.filebrowser.FileBrowser.(FileBrowser.java:92) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
and collapsed thereafter. But if I run matlab -nojvm, it worked normally.

Solution: the java compiler that comes together with MATLAB caused the above error. Replace it with my own version of java (jre1.5.0_06)

cd $MATLAB/sys/java
mv java java-backup
ln -s path_of_your_own_java java

And then MATLAB works now. Bingo!

PS: kkk recommended another standalone software, Copasi, to build and simulate biomedical networks. Have a look at it.

COPASI is a software application for simulation and analysis of biochemical networks. COPASI — a COmplex PAthway SImulator. Bioinformatics 22, 3067-74.

Current Features:

• Stochastic and deterministic time course simulation
• Steady state analysis (including stability)
• Metabolic control analysis / sensitivity analysis
• Elementary mode analysis
• Mass conservation analysis
• Calculation of Lyapunov exponents
• Parameter scans
• Optimization of arbitrary objective functions
• Parameter estimation using data from time course and/or steady state experiments
• Sliders for interactive parameter changes
• Global parameter to change multiple kinetic rates at once
• Imports and exports SBML (export only in level 2 version 1, import all levels)
• Loads Gepasi files
• Export in Berkeley Madonna format and C source code of the ODE system generated from the chemical reactions
• Versions for MS Windows, Linux, OS X, and Solaris SPARC
• Command line version for batch processing
• Visit this page often, new releases will contain many more features!