Accessing XCP through libvirt API

Libvirt has enhanced its accesability to XCP or XenServer via its API. Inorder to use this feature you must compile libvirt from source with this feature enables. below are the steps to compile libvirt and its dependencies.

Step 1: There are two modes of accessing XCP via libvirt.
            * From the same machine where XCP is installed
            * On a remote machine

            But i would advice you to install libvirt on the same machine where XCP is installed as some native codes might require direct or local access.

Now,

Log in to the machine where you want to install libvirt.

Step 2: First you need to download and install xenapi package. Download xenapi package (libxenserver) from the following URL

ftp://193.166.3.2/pub/NetBSD/packages/distfiles/libxenserver-5.6.100-1-src.tar.bz2

Step 3: Open a terminal and extract the source

# cd <directory where it is downloaded>
# tar jxvf libxenserver-5.6.100-1-src.tar.bz2
# cd libxenserver

Step 4: Compile libxenserver. To compile you must have installed the following dependencies

* libxml2-dev
* libcurl3-dev
* zlib1g

in a debian/ubuntu based machine you can install like this

# apt-get install libxml2-dev
# apt-get install libcurl3-dev
# apt-get install zlib1g

There is a small change you have to make in the 'Makefile'

# vi Makefile

find the line (line no: 65)

$(TEST_PROGRAMS): test/%: test/%.o libxenserver.so
        $(CC) $(LDFLAGS) -o $@ $< -L . -lxenserver

and replace it as

$(TEST_PROGRAMS): test/%: test/%.o libxenserver.so
        $(CC) -o $@ $< -L . -lxenserver $(LDFLAGS)

Also find (line no: 73)


$(INSTALL_PROG) libxenserver.so.$(MAJOR).$(MINOR) $(DESTDIR)/usr/$(LIBDIR)

replace with

$(INSTALL_DATA) libxenserver.so.$(MAJOR).$(MINOR) $(DESTDIR)/usr/$(LIBDIR)

and then compile

# make LIBDIR=lib/
# ar rcs libxenserver.a
# make install LIBDIR=lib/

Step 5: Download libvirt from

http://libvirt.org/sources/libvirt-0.10.2.tar.gz

Step 6: Configure and compile libvirt

# cd <directory where it is downloaded>
# tar zxvf libvirt-0.10.2.tar.gz
# cd libvirt-0.10.2
# ./configure --with-esx --with-xenapi=<path to libxenserver that you installed earlier>  --prefix=/opt/libvirt

Here --prefix=/opt/libvirt is optional if you dont want it to install in system folders like /usr /bin ,etc

At the end of the above step, we must be able to identify that if libxenserver is included or not, from the output

NOTE:

1. While configuring if you find some errors that some package is missing, then install that package and reconfigure again.

eg:

configure: error: You must install device-mapper-devel/libdevmapper >= 1.0.0 to compile libvirt

Here you must install the package

libdevmapper-dev

Installation always require a development version, even though if it does not point it

2. Even though we have already installed libxenserver, sometimes the compiler cannot find LINK to libxenserver files

you can try the following solutions

# ldconfig

If this does not work, change the following code in "configure" file

# vi configure

Search for the line

        LIBXENSERVER_LIBS="$LIBXENSERVER_LIBS -lxenserver"

Scroll below to find "else" part like this

        if test "$with_xenapi" = "yes"; then
            fail=1
        fi
            with_xenapi=no

Replace it with

        with_xenapi=yes

        LIBXENSERVER_LIBS="$LIBXENSERVER_LIBS -lxenserver"

And then run the above command again.

Next to compile,

# make
# make install

You can also check the api capabilities with respect to xenapi from the following URL.

http://libvirt.org/hvsupport.html

This concludes our step by step installation of libvirt with xenapi support. :)

Hadoop Installation for Beginners

Well folks,
Here i will be giving you step by step procedures to install and configure hadoop (version 1.1.0) on a linux (debian based distro) as a single node cluster. This guide is for beginners and you need to boot into your linux machine as a root user

Step 1: First you need to download hadoop source from the following URL
http://apache.techartifact.com/mirror/hadoop/common/hadoop-1.1.1/hadoop-1.1.1.tar.gz

Open a terminal

# cd <to directory where you downloaded hadoop>
# mv hadoop-1.1.0.tar.gz /usr/local/
# cd /usr/local/
# tar zxvf hadoop-1.1.0.tar.gz

From the above commands, you have actually moved hadoop src to /usr/local and uncompressed that file in /usr/local/

Step 2: Hadoop is a standalone java based application, so it requires java 1.6 as its dependency which is to be installed by your own ( if not already installed).

Step 3: Next you need to add a specific user to associate to hadoop

# adduser hadoop

It prompts you to enter password and few other information

             Adding user `hadoop' ...
             Adding new group `hadoop' (1001) ...
             Adding new user `hadoop' (1001) with group `hadoop' ...
             Creating home directory `/home/hadoop' ...
             Copying files from `/etc/skel' ...
             Enter new UNIX password:
             Retype new UNIX password:
             passwd: password updated successfully
             Changing the user information for hadoop
             Enter the new value, or press ENTER for the default
                   Full Name []:
                   Room Number []:
                   Work Phone []:
                  Home Phone []:
                 Other []:
             Is the information correct? [Y/n] Y

Step 4: Change the configuration files

Befor we configure, type the following to identify java home

# which java

if for example output is

                /usr/bin/java
Then

your JAVA_HOME is /usr

Now,

# cd /usr/local/hadoop-1.1.0/
# cd conf/
# vi hadoop-env.sh

Find the following line

                  # export JAVA_HOME=/usr/lib/j2sdk1.5-sun

and replace it as

                  export JAVA_HOME=/usr/


Next paste the following content into the file core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
   <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:8020</value>
  </property>
</configuration>

Next paste the following content into the file hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>

</configuration>

Next paste the following content into the file mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>localhost:8021</value>
  </property>
</configuration>

Next check the file /etc/hosts if the following content exists as the first line, if not add it

127.0.0.1       localhost <your host name>

Where,
          <your host name> is the hostname of your machine.

you can find the hostname by

# hostname

Step 5: Associate user hadoop to your source folder

# cd /usr/local/

# chown -R hadoop hadoop-1.1.0

Step 6: Format HDFS file system Name node and Data Node

# cd /usr/local/hadoop-1.1.0/bin

# su hadoop

# ./hadoop namenode -format

It provides information like

12/10/19 12:00:20 INFO namenode.NameNode: STARTUP_MSG: 

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = java.net.UnknownHostException: vignesh: vignesh

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 1.1.0

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1394289; compiled by 'hortonfo' on Thu Oct  4 22:06:49 UTC 2012

************************************************************/

12/10/19 12:00:20 INFO util.GSet: VM type       = 64-bit

12/10/19 12:00:20 INFO util.GSet: 2% max memory = 17.77875 MB

12/10/19 12:00:20 INFO util.GSet: capacity      = 2^21 = 2097152 entries

12/10/19 12:00:20 INFO util.GSet: recommended=2097152, actual=2097152

12/10/19 12:00:21 INFO namenode.FSNamesystem: fsOwner=hadoop

12/10/19 12:00:21 INFO namenode.FSNamesystem: supergroup=supergroup

12/10/19 12:00:21 INFO namenode.FSNamesystem: isPermissionEnabled=true

12/10/19 12:00:21 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

12/10/19 12:00:21 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

12/10/19 12:00:21 INFO namenode.NameNode: Caching file names occuring more than 10 times 

12/10/19 12:00:21 INFO common.Storage: Image file of size 112 saved in 0 seconds.

12/10/19 12:00:21 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/tmp/hadoop-hadoop/dfs/name/current/edits

12/10/19 12:00:21 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/tmp/hadoop-hadoop/dfs/name/current/edits

12/10/19 12:00:21 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.

12/10/19 12:00:21 INFO namenode.NameNode: SHUTDOWN_MSG: 

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: vignesh: vignesh

************************************************************/

Similarly format the data node by

# ./hadoop datanode -format

Step 7: Make passwordless ssh for hadoop user

# ssh-keygen -t rsa -P ""

Press enter when it promts

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 

and it generates the key as

Created directory '/home/hadoop/.ssh'.

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

f7:e3:1d:e6:2d:7d:23:2f:64:ea:1c:77:99:26:af:e0 hadoop@vignesh

The key's randomart image is:

+--[ RSA 2048]----+

|                 |

|                 |

|                 |

|                 |

|        S .      |

|         . . o  o|

|           o*oo* |

|          oo+B*+o|

|          .E..B++|

+-----------------+

# cat /home/hadoop/.ssh/id_rsa.pub > /home/hadoop/.ssh/authorized_keys

# ssh hadoop@localhost

type "yes" if it prompts as below

The authenticity of host 'localhost (::1)' can't be established.

RSA key fingerprint is 7e:4a:40:b5:57:06:0d:83:34:58:80:80:c3:e7:18:20.

Are you sure you want to continue connecting (yes/no)? 

After this it logs into hadoop user and you have successfully configured passwordless ssh

Now type

# exit

The above command must be used only once. So you are still as hadoop user

Step 8: Start Hadoop services

# ./start-all.sh

it starts 5 services

 NameNode
 SecondaryNameNode
 DataNode
 JobTracker
 TaskTracker

You can check if the services are running by

# jps

You must see something like this. If not you are facing some errors

26207 TaskTracker
26427 Jps
25847 DataNode
25986 SecondaryNameNode
26089 JobTracker
25738 NameNode

Log into 

       http://localhost:50030

for hadoop map/reduce administration (optional)

Log into 

       http://localhost:50070

for browsing the hdfs file system (optional)

Step 9: Follow these commands

# ./hadoop dfsadmin -report

This command gives you information on your hdfs system

# ./hadoop fs -mkdir test

This command creates a directory "test" in your hdfs file system

# vi test_input

 In the text editor type

 "hi all hello all"

 save and exit the file

# ./hadoop fs -put test_input test/input

This command copyies the file (test_input)  that we just created into hdfs file system (inside test folder)

#./hadoop fs -ls test

This command list all files in folder "test" of  hdfs file system.

#./hadoop jar ../hadoop-examples-1.1.1.jar wordcount test/input test/output

This command runs a mapreduce program (word count) for your input and generates output in "test/output" of hdfs file system.

You can check the output in the following url

http://localhost:50070

Browse the filesystem -> user -> hadoop -> test -> output ->part-r-00000

Step 10: To stop hadoop (optional)

# sh stop-all.sh

Here end our step by step guide to work with hadoop ( for beginners ).