Local JARs for Maven Dependency

1. download jar into /mavenlocal

2. run

mvn install:install-file -DgroupId=classifier4j -DartifactId=Classifier4J -Dversion=0.6 -Dpackaging=jar -DcreateChecksum=true -Dfile=/home/xzhao/mavenlocal/Classifier4J-0.6.jar

3. Add dependency

<dependency>

    <groupId>classifier4j</groupId>

    <artifactId>Classifier4J</artifactId>

    <version>0.6</version>

</dependency>

The following method not working well

1. Download jar file into /lib

2. Use Maven to install to project repo

 mvn install:install-file -DlocalRepositoryPath=repo -DcreateChecksum=true -Dpackaging=jar -Dfile=[your-jar] -DgroupId=[...] -DartifactId=[...] -Dversion=[...]  

maven repository created in /lib/repo

3. Add repository in pom.xml

 <repository>  
   <id>repo</id>  
   <url>file://${project.basedir}/repo</url>  
 </repository>  

4. Add dependencies

sources:

• http://stackoverflow.com/questions/364114/can-i-add-jars-to-maven-2-build-classpath-without-installing-them
• http://blog.dub.podval.org/2010/01/maven-in-project-repository.html

Create Ubuntu Xen image for Eucalyptus

(perform steps 1-6 on node controller; 7- on client machine)

1. create a folder and download ISO file.

sudo mkdir ubuntu-xen-manual
cd ubuntu-xen-manual
wget ***.iso

2. Create a 4GB virtual disk

sudo dd if=/dev/zero of=ubuntu-12.04D.img bs=1M count=4096

3. Create Xen configuration file (xen.cfg) with followings:

 name = "ubuntubox"  
 #make sure kernel is in right place  
 kernel = "/usr/lib/xen-default/boot/hvmloader"  
 memory = 1024  
 builder = "hvm"  
 #make sure device_model is in right place  
 device_model = "/usr/lib/xen-default/bin/qemu-dm"  
 boot = "d"  
 disk = ['file:~/ubuntu-xen-manual/***.iso,hdc:cdrom,r',  
 'file:~/ubuntu-xen-manual/ubuntu-12.04D.img,hda,w']  
 vif = ['']  
 #dhcp="on"  
 vnc = 1  
 vncdisplay = 7  
 pae = 1  

 

4. Start domU

sudo xen create xen.cfg  

5. Connect with a VNC viewer (if desktop version)

sudo apt-get install xvnc4viewer

xvncviewer localhost:7

6. Find out the starting block and the block size of the root file system.

sudo parted ubuntu-12.04D.img

(parted) U
Unit? [compact]? b

(parted) p

You'll see an "unrecognized disk label" message because it is a new drive.

(parted) mklabel msdos

(parted) print free

 Number  Start   End          Size         Type  File system  Flags
        16384B  4294967295B  4294950912B        Free Space

(parted) q

sudo dd if=ubuntu-12.04D.img of=rootfs.img bs=1 skip=16384 count=4294950912

(*the extraction takes quite a while; to test change the image size to smaller 512*)

the root image is created as rootfs.img.

7. Bundle, upload and register root image to Eucalyptus

scp rootfs.img to client machine

open a terminal on client machine:

sudo scp server01@10.1.62.172:~/ubuntu-xen-manual/rootfs.img ~/

source eucarc

cd ~/.euca

source eucarc

euca-bundle-image -i ~/rootfs.img (execute in ~/.euca; otherwise EC2_CERT not found error)

 Checking image  
 Compressing image  
 Encrypting image  
 Splitting image...  
 Part: rootfs.img.part.00  
 Generating manifest /tmp/rootfs.img.manifest.xml  

euca-upload-bundle -b ubuntu -m /tmp/rootfs.img.manifest.xml

 Checking bucket: ubuntu  
 Creating bucket: ubuntu  
 Uploading manifest file  
 Uploading part: rootfs.img.part.00  
 Uploaded image as ubuntu/rootfs.img.manifest.xml  

 euca-register ubuntu/rootfs.img.manifest.xml

 IMAGE     emi-48AE3FD9  

euca-describe-images

 IMAGE     emi-48AE3FD9     ubuntu/rootfs.img.manifest.xml 

292622667431     available public          i386     machine 

eki-855C3923     eri-05BE397E          instance-store  

Configure SIREn, Solr on Jetty

(Default settings: SIREn with Lucene 3.5, Solr 3.6 with default Jetty)

1. create a /lib folder under SOLR_HOME/example/solr, add the following from SIREn targets:

siren-core-0.2.3-SNAPSHOT.jar,
siren-qparser-0.2.3-SNAPSHOT.jar
siren-solr-0.2.3-SNAPSHOT.jar

2. modify solrconfig.xml, add following:

 <!-- Example of Registration of the siren query parser. -->  
  <queryParser name="siren" class="org.sindice.siren.solr.SirenQParserPlugin"/>  
  <requestHandler name="siren" class="solr.StandardRequestHandler">  
   <!-- default values for query parameters -->  
    <lst name="defaults">  
     <str name="defType">siren</str>  
     <str name="echoParams">explicit</str>  
                 <!-- Disable field query in keyword parser -->  
     <str name="disableField">true</str>  
     <str name="qf">  
      ntriple^1.0 url^1.2  
     </str>  
     <str name="nqf">  
      ntriple^1.0  
     </str>  
     <!-- the NTriple query multi-field operator:  
       - disjunction: the query should match in at least one of the fields  
       - scattered: each Ntriple patterns should match in at least on of the fields  
     -->   
     <str name="nqfo">scattered</str>  
     <str name="tqf">  
      tabular^1.0  
     </str>  
     <!-- the Tabular query multi-field operator:  
       - disjunction: the query should match in at least one of the fields  
       - scattered: each tabular patterns should match in at least on of the fields  
     -->  
     <str name="tqfo">scattered</str>  
     <str name="fl">  
      id  
     </str>  
    </lst>  
  </requestHandler>  

3. modify schema.xml, add following and rename fields url and id if they exist in the file already.

 <!-- The ID (URL) of the document   
         Use the 'string' field type (no tokenisation)  
      -->  
        <field name="id" type="string" indexed="true" stored="true" required="false"/>  
       <!-- The URL of the document   
         Use the 'text' field type in order to be tokenised  
      -->  
        <field name="url" type="uri" indexed="true" stored="true" required="true"/>  
 <!-- n-triple indexing scheme -->  
        <field name="ntriple" type="ntriple" indexed="true" stored="true" multiValued="false"/>  
     <!-- tabular indexing scheme -->  
     <field name="tabular" type="tabular" indexed="true" stored="false" multiValued="false"/>  

 <!-- A uri field that uses WhitespaceTokenizer and WordDelimiterFilter to   
      split URIs into multiple compoenents. Stopwords is customized by   
      external files.  
      omitNorms is true since it is a short field, and it does not make   
      really sense on URI.  
      Does not use the ASCIIFoldingExpansionFilter since URIs should not  
      contain accented characters.  
   -->  
   <fieldType name="uri" class="solr.TextField" omitNorms="true" positionIncrementGap="100">  
    <analyzer type="index">  
     <tokenizer class="solr.WhitespaceTokenizerFactory"/>  
     <!-- Splits words into subwords based on delimiters  
        - split subwords based on case change  
        - preserveOriginal="1" in order to preserve the original word.  
        Removed split based on numerics to fix SND-355 and SND-1283   
     -->  
     <filter class="solr.WordDelimiterFilterFactory"   
         generateWordParts="1"   
         generateNumberParts="1"   
         catenateWords="0"   
         catenateNumbers="0"   
         catenateAll="0"   
         splitOnCaseChange="1"  
         splitOnNumerics="0"  
         preserveOriginal="1"/>  
     <!-- Filters out those tokens *not* having length min through max   
        inclusive. -->  
     <filter class="solr.LengthFilterFactory" min="2" max="256"/>  
     <!-- Change to lowercase text -->  
     <filter class="solr.LowerCaseFilterFactory"/>  
     <!-- Case insensitive stop word removal.  
      add enablePositionIncrements=true in both the index and query  
      analyzers to leave a 'gap' for more accurate phrase queries.  
     -->  
     <filter class="solr.StopFilterFactory"  
         ignoreCase="true"  
         words="stopwords.txt"  
         enablePositionIncrements="true"  
         />  
    </analyzer>  
    <analyzer type="query">  
     <!-- whitespace tokenizer to not tokenize URI -->  
     <tokenizer class="solr.WhitespaceTokenizerFactory"/>  
     <!-- Filters out those tokens *not* having length min through max   
        inclusive. -->  
     <filter class="solr.LengthFilterFactory" min="2" max="256"/>  
     <filter class="solr.LowerCaseFilterFactory"/>  
     <filter class="solr.StopFilterFactory"  
         ignoreCase="true"  
         words="stopwords.txt"  
         enablePositionIncrements="true"  
         />  
     <!-- Replace Qnames by their name spaces in URIs. -->  
     <filter class="org.sindice.siren.solr.analysis.QNamesFilterFactory"   
         qnames="qnames.txt"/>  
    </analyzer>  
   </fieldType>  
 <!--  
            The SIREn field type:  
                The top-level analyzers must be defined in the top-level analyzer   
    configuration file (ntriple-analyzers.xml) and the datatype analyzers in   
    the datatype analyzer configuration file (ntriples-datatypes.xml).   
                Field norms are not useful for SIREn fields. Set omitNorms to true reduces  
                memory consumption, and improve ranking.  
    omitTermFreqAndPositions *must* be set to false.  
           -->  
   <fieldType name="ntriple" class="org.sindice.siren.solr.schema.SirenField"  
         omitNorms="true"   
         omitTermFreqAndPositions="false"  
         analyzerConfig="tuple-analyzers.xml"  
         datatypeConfig="tuple-datatypes.xml"/>  
   <fieldType name="tabular" class="org.sindice.siren.solr.schema.SirenField"  
         omitNorms="true"   
         omitTermFreqAndPositions="false"  
         analyzerConfig="tuple-analyzers.xml"  
         datatypeConfig="tuple-datatypes.xml"/>  

 <similarity class="org.sindice.siren.similarity.SirenSimilarity"/>  

4. copy the following files from SIREN_HOME/siren_solr/example/solr/config to SOLR_HOME/example/solr/config

tuple-analyzers.xml
tuple-datatypes.xml
qnames.txt

5. Restart default Jetty in Solr by java -jar start.jar

6. test with sample code in SIREN_HOME/siren_solr/example/

The examples are indexed successfully but the queries return no result.

P.S. SIREn doesn't support SPARQL.

sources:

• https://github.com/rdelbru/SIREn/blob/master/siren-solr/example/INSTALL.txt

Install SIREn on Ubuntu 12.04

1. check jdk and maven installation.

$sudo apt-get install maven (*maven 3 will be installed*)

2. run maven at the SIREn directory

$mvn package

3. check jars 

The jar files are located under /target in each folder.

P.S. If the following error occurs:

[ERROR] Failed to execute goal on project siren-core: Could not resolve dependencies for project org.sindice.siren:siren-core:jar:0.2.3-SNAPSHOT: Failure to find com.google.code.caliper:caliper:jar:1.0-SNAPSHOT in https://oss.sonatype.org/content/groups/public/ was cached in the local repository, resolution will not be reattempted until the update interval of oss-sonatype has elapsed or updates are forced -> [Help 1]

Modify pom.xml under siren-core

Change

 <dependency>  
    <groupId>com.google.code.caliper</groupId>  
    <artifactId>caliper</artifactId>  
    <version>1.0-SNAPSHOT</version>  
    <scope>test</scope>  
   </dependency>  

To

 <dependency>  
 <groupId>com.google.caliper</groupId>  
 <artifactId>caliper</artifactId>  
 <version>0.5-rc1</version>  
 <scope>test</scope>  
 </dependency>  

Sources:

• https://github.com/rdelbru/SIREn/wiki/Getting-Started

Ubuntu Server 12.04 LTS, Eucalyptus and Xen

In this installation, 2 Dell machines (Dual Core, 4GB RAM, 250 GB hard disk) and 1 IBM x3650 machine are used. 

1. Install Eucalyptus cloud and cluster controller (Dell)

1.1. Ubuntu Server 12.04 LTS 

• default installation on openssh
• make network is set correctly

1.2 Install postfix which will be used by Eucalyptus for sending confirmation emails to new registered users.

• sudo apt-get install postfix
• follow this link for configuration and testing: https://help.ubuntu.com/community/PostfixBasicSetupHowto 

1.3 Install Eucalyptus
1.3.1 download and add pub key

•  sudo wget http://www.eucalyptus.com/sites/all/files/c1240596-eucalyptus-release-key.pub

• sudo apt-key add c1240596-eucalyptus-release-key.pub

 
1.3.2 add deb
Create a file in /etc/apt/sources.list.d called eucalyptus.list with the following content:

• deb http://downloads.eucalyptus.com/software/eucalyptus/3.1/ubuntu precise main

On all machines that will run either Eucalyptus or Euca2ools, create a file in/etc/apt/sources.list.d called euca2ools.list with the following content:

• deb http://downloads.eucalyptus.com/software/euca2ools/2.1/ubuntu precise main

1.3.3 install

• sudo apt-get update  
• sudo apt-get install eucalyptus-cloud eucalyptus-cc eucalyptus-walrus

1.3.4 start

• /usr/sbin/euca_conf --initialize (for first time)
  sudo service eucalyptus-cloud start
  sudo service eucalyptus-cc start 
  
  
  1.3.5 verify
  check on Web browser: https://<ip>:8443
  

first time login will allow changing password, set email address admin/admin

2. Install Eucalyptus node controller (IBM)

2.1 Repeat 1.1, 1.3.1, 1.3.2
2.2 install Xen

• sudo apt-get install iproute  iptables module-init-tools python2.5 python2.6
• sudo apt-get  install xen-utils

2.3 Modify GRUB to boot default on Xen

sudo sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Xen 4.1-amd64"/' /etc/default/grub
sudo update-grub

Set the default toolstack to xm (aka xend):

sudo sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xm"/' /etc/default/xen

Now reboot:

sudo reboot

And then verify that the installation has succeeded: 

sudo xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 945 1 r----- 11.3

2.4 Network configuration
check eth0 or eth1 is used. the following assumes eth0 and dhcp are used.

sudo apt-get install bridge-utils

Edit /etc/network/interfaces, and add this:  

auto xenbr0
iface xenbr0 inet dhcp
    bridge_ports eth0

Restart networking to enable xenbr0 bridge: 

sudo service networking restart

2.5 install Eucalyptus node controller

• sudo apt-get install eucalyptus-nc
  sudo service eucalyptus-nc start

3. Register

 Register CC (at CLC machine)

sudo /usr/sbin/euca_conf --register-cluster --partition cluster01  --host  10.1.62.178 --component  cc-10.1.62.178
(this may require root access: using sudo passwd to add a new password for root)

Register NC (at CC machine)

sudo /usr/sbin/euca_conf --register-nodes "<node0_IP_address> ... <nodeN_IP_address>"

Register walrus at CLC machine

sudo /usr/sbin/euca_conf --register-walrus --partition walrus --host <walrus_IP_address> --component <walrus-hostname>

4. Install Eucalyptus client machine (Dell)

4.1 install Ubuntu Desktop 12.04 LTS
4.2 repeat 1.3.1 and 1.3.2
4.3 install euca2ools

• sudo apt-get install euca2ools

apply a new user account from @8443

Get admin credentials

login as eucalyptus/admin @ 8443

download new credentials from admin@eucalyptus

unzip to ~/.euca

mkdir ~/.euca cd ~/.euca unzip <filepath>/euca2-<user>-x509.zip

source eucarc

<*remember to use admin credential to source*>

4.4 Add image from eustore

eustore-describe-images

eustore-install-image -b testbucket -i xxxxxxxx -k xen

<*this will take a while*>

the results look like as follows:

Downloading Image :  CentOS 5 1.3GB root, Hypervisor-Specific Kernels
0-----1-----2-----3-----4-----5-----6-----7-----8-----9-----10
##############################################################

Checking image bundle
Unbundling image
going to look for kernel dir : xen-kernel
Bundling/uploading kernel
Checking image
Compressing image
Encrypting image
Splitting image...
Part: vmlinuz-2.6.27.21-0.1-xen.part.00
Generating manifest /tmp/RXAdDH/vmlinuz-2.6.27.21-0.1-xen.manifest.xml
Checking bucket: testbucket
Creating bucket: testbucket
Uploading manifest file
Uploading part: vmlinuz-2.6.27.21-0.1-xen.part.00
Uploaded image as testbucket/vmlinuz-2.6.27.21-0.1-xen.manifest.xml
testbucket/vmlinuz-2.6.27.21-0.1-xen.manifest.xml
eki-855C3923
Bundling/uploading ramdisk
Checking image
Compressing image
Encrypting image
Splitting image...
Part: initrd-2.6.27.21-0.1-xen.part.00
Generating manifest /tmp/RXAdDH/initrd-2.6.27.21-0.1-xen.manifest.xml
Checking bucket: testbucket
Uploading manifest file
Uploading part: initrd-2.6.27.21-0.1-xen.part.00
Uploaded image as testbucket/initrd-2.6.27.21-0.1-xen.manifest.xml
testbucket/initrd-2.6.27.21-0.1-xen.manifest.xml
eri-05BE397E
Bundling/uploading image
Checking image
Compressing image
Encrypting image
Splitting image...
Part: euca-centos-5.8-2012.07.05-x86_64.part.00
Part: euca-centos-5.8-2012.07.05-x86_64.part.01
Part: euca-centos-5.8-2012.07.05-x86_64.part.02
Part: euca-centos-5.8-2012.07.05-x86_64.part.03
Part: euca-centos-5.8-2012.07.05-x86_64.part.04
Part: euca-centos-5.8-2012.07.05-x86_64.part.05
Part: euca-centos-5.8-2012.07.05-x86_64.part.06
Part: euca-centos-5.8-2012.07.05-x86_64.part.07
Part: euca-centos-5.8-2012.07.05-x86_64.part.08
Part: euca-centos-5.8-2012.07.05-x86_64.part.09
Part: euca-centos-5.8-2012.07.05-x86_64.part.10
Part: euca-centos-5.8-2012.07.05-x86_64.part.11
Part: euca-centos-5.8-2012.07.05-x86_64.part.12
Part: euca-centos-5.8-2012.07.05-x86_64.part.13
Part: euca-centos-5.8-2012.07.05-x86_64.part.14
Part: euca-centos-5.8-2012.07.05-x86_64.part.15
Part: euca-centos-5.8-2012.07.05-x86_64.part.16
Part: euca-centos-5.8-2012.07.05-x86_64.part.17
Part: euca-centos-5.8-2012.07.05-x86_64.part.18
Part: euca-centos-5.8-2012.07.05-x86_64.part.19
Part: euca-centos-5.8-2012.07.05-x86_64.part.20
Part: euca-centos-5.8-2012.07.05-x86_64.part.21
Part: euca-centos-5.8-2012.07.05-x86_64.part.22
Part: euca-centos-5.8-2012.07.05-x86_64.part.23
Part: euca-centos-5.8-2012.07.05-x86_64.part.24
Part: euca-centos-5.8-2012.07.05-x86_64.part.25
Part: euca-centos-5.8-2012.07.05-x86_64.part.26
Part: euca-centos-5.8-2012.07.05-x86_64.part.27
Part: euca-centos-5.8-2012.07.05-x86_64.part.28
Part: euca-centos-5.8-2012.07.05-x86_64.part.29
Part: euca-centos-5.8-2012.07.05-x86_64.part.30
Part: euca-centos-5.8-2012.07.05-x86_64.part.31
Part: euca-centos-5.8-2012.07.05-x86_64.part.32
Part: euca-centos-5.8-2012.07.05-x86_64.part.33
Part: euca-centos-5.8-2012.07.05-x86_64.part.34
Part: euca-centos-5.8-2012.07.05-x86_64.part.35
Generating manifest /tmp/RXAdDH/euca-centos-5.8-2012.07.05-x86_64.manifest.xml
Checking bucket: testbucket
Uploading manifest file
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.00
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.01
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.02
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.03
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.04
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.05
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.06
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.07
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.08
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.09
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.10
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.11
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.12
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.13
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.14
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.15
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.16
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.17
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.18
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.19
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.20
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.21
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.22
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.23
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.24
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.25
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.26
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.27
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.28
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.29
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.30
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.31
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.32
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.33
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.34
Uploading part: euca-centos-5.8-2012.07.05-x86_64.part.35
Uploaded image as testbucket/euca-centos-5.8-2012.07.05-x86_64.manifest.xml
testbucket/euca-centos-5.8-2012.07.05-x86_64.manifest.xml
Installed image: emi-97B83704

run $ euca-describe-images to verify emi-97B83704, results as follows:
IMAGE    eki-855C3923    testbucket/vmlinuz-2.6.27.21-0.1-xen.manifest.xml    292622667431    available    public        x86_64kernel                instance-store
IMAGE    emi-97B83704    testbucket/euca-centos-5.8-2012.07.05-x86_64.manifest.xml    292622667431    available    public        x86_64    machine    eki-855C3923    eri-05BE397E        instance-store
IMAGE    eri-05BE397E    testbucket/initrd-2.6.27.21-0.1-xen.manifest.xml    292622667431    available    public        x86_64ramdisk                instance-store

Images can also be seen @8443, Resource management/Images

5. Generate admin credentials

/usr/sbin/euca_conf --get-credentials admin.zip

unzip admin.zip

sudo -s

source eucarc

sources:
1. Xen setting: https://help.ubuntu.com/community/XenProposed

Working on IBM x3650

Network setting in Ubuntu server 12.10

sudo nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
netmask 255.255.255.0
broadcast 10.1.62.255


auto eth1
iface eth1 inet dhcp
netmask 255.255.255.0
broadcast 10.1.62.255

sudo service networking restart
sudo ifup eth0
sudo ifup eth1

ifconfig -a
will see the results of configuration
eth0 with ip 10.1.62.xxx
eth1 with ip 192.168.2.99

to test ping google.com

Building a Java Application with Nutch 2.0 and Solr 3.6

Apache Nutch 2.0 source @ ~/tmp/apache-solr-3.6.0 SOLR_HOME

Apache Solr 3.6 binary @ ~/tmp/apache-nutch-2.0-src NUTCH_HOME

Apache hbase @ ~/tmp/hbase-0.90.6 HBASE_HOME

1. Preparing libraries in Eclipse

Go to /path/to/solr/dist and open apache-solr-3.6.0.war with your favorite archive manager. Go to /WEB-INF/lib/ and extract everything there to /path/to/solr/dist. This will allow us to include all the libraries we need in our Java application.

2. Creating an Eclipse project

Create a Java project with a class called MyCrawler.

Add a lib folder into the project, then add two sub folders: nutch and solr. Copy everything from NUTCH_HOME/lib to lib/nutch and everything from SOLR_HOME/dist to lib/solr.

Add a plugins folder into the project, add everything from NUTCH_HOME/plugins to it.

Add a urls folder and create a seed.txt file. Put seed urls here.

Create two folders somewhere on the file system: nutchconf and solrconf, and copy all the files from NUTCH_HOME/conf and SOLR_HOME/example/solr/conf to them respectively. 

In Eclipse project properties->Java Build Path, click on Add Class folder  and add nutchconf and solrconf folders in. Go to Order and Export, find the entries for nutchConf and solrConf and move them to the top. 

Change Lucene_core_3.4.0.jar to Lucene_core_3.6.0.jar.

3. Adding code into MyCrawler.

 package websearch.crawler;  
 import java.util.StringTokenizer;  
 import org.apache.hadoop.util.ToolRunner;  
 import org.apache.nutch.indexer.solr.SolrIndexerJob;  
 import org.apache.nutch.crawl.Crawler;  
 import org.apache.nutch.util.NutchConfiguration;  
 import org.apache.solr.client.solrj.SolrQuery;  
 import org.apache.solr.client.solrj.SolrServerException;  
 import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer;  
 import org.apache.solr.client.solrj.response.QueryResponse;  
 import org.apache.solr.common.SolrDocumentList;  
 public class MyCrawler {  
      /**  
       * @param args  
       * @throws Exception   
       */  
      public static void main(String[] args) throws Exception {  
           String crawlArg = "urls -depth 3 -topN 5";  
     // Run Crawl tool  
     try {  
         ToolRunner.run(NutchConfiguration.create(), new Crawler(),  
                 tokenize(crawlArg));  
     } catch (Exception e) {  
         e.printStackTrace();  
         return;  
     }  
     String indexArg = "http://localhost:8983/solr -reindex";  

     //Run Solr index tool
     try {  
       ToolRunner.run(NutchConfiguration.create(), new SolrIndexerJob(),  
               tokenize(indexArg));  
     } catch (Exception e) {  
       e.printStackTrace();  
       return;  
     }  
     // Let's query for something!  
     String url = "http://localhost:8983/solr";  
     CommonsHttpSolrServer server = new CommonsHttpSolrServer( url );  
     SolrQuery query = new SolrQuery();  
     query.setQuery("content:mycontent"); // Searching mycontent in query  
     query.addSortField("content", SolrQuery.ORDER.asc);  
     QueryResponse rsp;  
     try {  
         rsp = server.query(query);  
     } catch (SolrServerException e) {  
         // TODO Auto-generated catch block  
         e.printStackTrace();  
         return;  
     }  
     // Display the results in the console  
     SolrDocumentList docs = rsp.getResults();  
     for (int i = 0; i < docs.size(); i++) {  
         System.out.println(docs.get(i).get("title").toString() + " Link: "  
                 + docs.get(i).get("url").toString());  
     }    
 }  
      /**  
       * Helper function to convert a string into an array of strings by  
       * separating them using whitespace.  
       *   
       * @param str  
       *      string to be tokenized  
       * @return an array of strings that contain a each word each  
       */  
      public static String[] tokenize(String str) {  
           StringTokenizer tok = new StringTokenizer(str);  
           String tokens[] = new String[tok.countTokens()];  
           int i = 0;  
           while (tok.hasMoreTokens()) {  
                tokens[i] = tok.nextToken();  
                i++;  
           }  
           return tokens;  
      }  
 }  

4. Run.
start hbase
start solr
run MyCrawler as Java application


Other resources:

• Building a Java application with Apache Nutch and Solr
• Solr Wiki: http://wiki.apache.org/solr/Solrj
• http://lucidworks.lucidimagination.com/display/solr/Using+SolrJ
• regex checker: http://regexpal.com/