Hom Hadoop
Txheej txheem cej luam: Apache Hadoop yuav muab ntsia tau rau hauv ntau hom as per lub. Cov hom ntau yog configured thaum plahaum. Yog vim, Hadoop yog ntsia tau rau hauv Standalone hom. Yog cov lwm hom Pseudo faib hom thiab faib hom. Lub hom phiaj ntawm no nyeem piav plahaum ntau hom rau hauv ib txoj kev uas yooj yim kom tus nyeem tau ua raws li nws thiab ua lawv cov haujlwm no.
Nyob rau cov tshooj no, Kuv yuav tham txog hom plahaum sib txawv thiab cov lus.
Taw qhia: Peb yeej paub tias Apache Hadoop yog ib tug moj qhib qhov khaum tsocai rau distributed ua rau poob lawm ntau ntawm cov ntaub ntawv txheej nyob txawv nyob siv programming yoojyim. Hadoop muaj peev xwm teev mus txog txhiab tus xyuas hauv computer los ntawm ib tug tib neeg rau zaub mov. Yog li ntawm cov mob no plahaum ntawm Hadoop yuav zoo tshaj plaws. Peb yuav nruab Hadoop nyob rau hauv peb hom sib txawv –
- Hom Standalone – Ntawm ib pawg
- Pseudo distributed hom – Ntawm ib pawg
- Faib hom. – Ntawm ntau pawg
Cov plahaum ntau hom phiaj: Thaum Apache Hadoop siv nyob rau hauv ib cheeb tsam ntau lawm, Cov ntxaij nodes siv rau faib xam. Tab sis, txoj kev nkag siab qhov yooj yim thiab ua si ib ncig nrog Hadoop, tsis muaj leej twg nruab nrab yog txaus. Muaj lwm hom hu ua 'pseudo faib’ hom. Hom ntawv no siv los txhawb cov miv nyuas tej node rau ib zaug xwb.
Nyob rau hauv daim ntawv no peb yuav tham txog seb yuav nruab Hadoop rau Ubunto Linux. Ua nws tej hom, Lub kaw lus yuav tsum muaj java version 1.6.x. installed rau nws.
Standalone hom plahaum: Tam sim no, Qhia rau peb saib tus standalone hom plahaum los tom qab cov kauj ruam hauv qab no.
Nruab Java –
Java (JDK Version 1.6.x) Xws li ntawm lub hnub/Oracle los yog qhib Java yuav tsum tau txhaj.
- Kauj ruam 1 – Yog hais tias koj tsis tau hloov mus rau OpenJDK tsis siv proprietary sun JDK/JRE, nruab hnub-java6 ntawm Canonical tus khub Repository uas siv cov tom qab hais kom ua.
Ceeb toom: Tus Canonical Partner Repository Muaj cov nqi kaw qhov chaw thib peb tog software. Tiam sis tus Canonical tsis tau mus saib tau qhov chaws xwb lawv cia li pob thiab kuaj nws.
Ntxiv tus khub canonical tus khub rau lub apt repositories siv –
[Chaws]
$ Sudo ad-apt-repository “nuj nqis://archive.canonical.com/lucid tus khub”
[/Chaws]
- Kauj ruam 2 – Hloov yus daim ntawv teev npe.
[Chaws]
$ supt-get apt-get
[/Chaws]
- Kauj ruam 3 – Nruab JDK version 1.6.6 ntawm hnub/Oracle.
[Chaws]
$ Sudo apt-get nruab hnub-java6-jdk
[/Chaws]
- Kauj ruam 4 – Thaum JDK plahaum lawm kom paub tseeb tias nws yog tseeb tseeb siv – version 1.6.x ntawm hnub/Oracle.
[Chaws]
user@ubuntu:~# java-version java version “1.6.0_45” Java(QHIA NTXIV TXOG) SE Runtime ib puag ncig (Tsim 1.6.0_45-b02) Java HotSpot(QHIA NTXIV TXOG) Nyob hauv kev pab VM (Tsim 16.4-b01, mixed hom, sib koom)
[/Chaws]
Ntxiv Cov Neeg Siv Khoom Noj
- Kauj ruam 5 – Ntxiv ib nplooj siab Hadoop ntxias neeg siv rau koj thaum uas yog rho tawm no installation ntawm lwm software –
[Chaws]
$ txaus hadoop_admin
[/Chaws]
Download lub Hadoop binary thiab ntsia
- Kauj ruam 6 – Download Apache Hadoop ntawm tus apache web site. Hadoop tuaj hauv daim ntawv ntawm tar-gx daim ntawv. Luam daim qauv no rau hauv lub /usr/zos/installables folder. Lub folder – yuav tsum tau tsim ua ntej hauv /peb/zos ua ntej qhov kauj ruam no. Tam sim no khiav lub commands tom qab commands li sudo
[Chaws]
$ cd/usr/local/installables $ tshawb fawb x xf hapood-0.20.2.gz.gz $ Sudo chown-R hadoop_admin/peb/zos/hadoop-0.20.2
[/Chaws]
Txhais env variable – JAVA_HOME
- Kauj ruam 7 – Qhib Lub Hadoop configuration ntaub ntawv (hadoop-env.sh) qhov chaw nyob – /siv/zos/installables/hadoop-0.20.2/conf/hadoop-env.env thiab txhais tau tias yog tus JAVA_HOME – tsis tau muaj
[Chaws] export JAVA_HOME=/txoj kev/jdk/installed [/Chaws]
(e.g. /rhuav/java)
Plahaum hauv Single mode
- Kauj ruam 8 – Tam sim no mus rau HADOOP_HOME thawj tswj (qhov chaw twg HADOOP yuav muab rho tawm ntawm qhov chaw uas muaj ntawv cog lus hadOOP) thiab khiav qhov hais kom ua raws li hais –
[Chaws]
$ kav hlau
[/Chaws]
Cov tso zis yuav muab tso tawm rau hauv qab no yuav muab tso tawm –
[Chaws] Pab: hadoop [–tshawb] HAIS KOM UA
[/Chaws]
Ib txhia cov COMMAND xaiv tau hais hauv qab no. Muaj lwm txoj kev xaiv thiab yuav tau xyuas seb tus hais kom ua saum toj no.
[Chaws] namenode -daim ntawv ua DFS cov ntaub ntawv DFS secondarynamenode khiav tus DFS theem namenode namenode khiav lub DFS namenode datanode khiav Ib DFS datanode dfsadmin khiav ib DFS admin mradmin khiav ib Map-Reduce admin neeg fsck khiav ib DFS filesystem checking utility
[/Chaws]
Cov zis saum toj no qhia tias Standalone plahaum yog tiav. Tam sim no koj yuav khiav cov qauv piv txwv txog koj xaiv tau –
[Chaws] $ hlwb/hadoop hadoop-*-piv txwv.jar <LUB NPE> <PARAMS>[/Chaws]
Pseudo distributed hom plahaum: Qhov no yog ib tug simulated tej node ib puag ncig raws li ib zaug xwb tsis muaj nyiaj.
Ntawm no tus thawj kauj ruam yuav tsum tau configure tus SSH thiaj li yuav siv tau thiab tswj lub nodes sib txawv. Li no mas nws yog tus cob moos yuav tsum tau cov SSH saib tau rau cov ntxaij nodes. Thaum lub SSH configured, enabled thiab yog peb yuav tsum tau pib configuring lub Hadoop. Cov ntaub ntawv nram qab no yuav tsum tau hloov –
- conf/tub ntxhais-site.xml
- conf/hdfs-site.xml
- conf/mapred.xml
Qhib tag nrho cov ntaub ntawv configuration vijtsam editor thiab hloov qhov configuration.
Configure core-site.xml ntawv:
[Chaws]$ vif/core-site.xml[/Chaws] [Chaws]<configuration><khoom><lub npe>FS.default.name</lub npe><tus nqi>hdfs://localhost:9000</tus nqi></khoom><khoom><lub npe>taub hau.tmp.dir.</lub npe><tus nqi>/tmp/hadoop-${user.name}</tus nqi></khoom></configuration>[/Chaws]
Configure hdfs-site.xml file:
[Chaws]$ vif/hdfs-site.xml[/Chaws] [Chaws]<configuration><khoom><lub npe>dfs.replication</lub npe><tus nqi>1</tus nqi></khoom></configuration>[/Chaws]
Configure mapred.xml file:
[Chaws]$ vif/mapred.xml[/Chaws] [Chaws]<configuration><khoom><lub npe>mapred.Job.tracker</lub npe> <tus nqi>localhost:9001</tus nqi></khoom></configuration>[/Chaws] Thaum cov kev hloov no yog ua li cas, Peb yuav tsum tsim lub npe node los ntawm kev siv lub npe tom qab hais kom ua. Qhov hais kom ua raws li yuav qhia tag nrho cov lus ib tom qab thiab thaum kawg successages. [Chaws]$ koob/hadpood namenode – hom ntawv[/Chaws] Tam sim no peb setup yog ua li cas rau pseudo faib node. Wb tam sim no pib lub node cluster nram qab no hais kom ua. Nws yuav rov qhia tej txheej lus rau qhov hais kom ua kom ua thiab pib cov txheej txheem. [Chaws]$ /rau hauv/pib-all.sh[Chaws] Tam sim no peb yuav tsum xyuas cov txheej txheem Hadoop los executing cov jps command li qhia hauv qab no. Nws yuav qhia tag nrho cov txheej txheem khiav. [Chaws]$ jps 14799 NameNode1497 SecondaryNameNode 15183 DataNode1596 JobTracker1597 TaskTracker[/Chaws]
Tsis siv lub Single node Cluster: Peb yuav tiv thaiv tsis tau lub node cluster uas siv cov tom qab hais kom ua. Tus hais kom ua raws yuav zaub tag nrho cov txheej txheem siv ceev xwmphem.
[Chaws]$ bin/stop-all.shopping stopping jobtrackercalhost: siv ceev ceev tasktrackerstopping namenodelocal: siv ceevxwm datanodelocal: nres secondarynamenode[/Chaws]
Distributed hom plahaum:
Ua ntej peb pib distributed hom plahaum, Peb yuav tsum xyuas kom meej tias peb muaj cov pseudo faib setup ua thiab peb muaj tsawg kawg yog ob cav tov, ib kuj ua tswv thiab lwm yam kuj ua ib qhev. Tam sim no peb khiav cov commands hauv sequence.
· $ hlwb/stop-all.sh – Xyuas kom tsis muaj cov nodes yog khiav
- Qhib lub /etc/hosts ntaub ntawv thiab ntxiv lub nram qab no rau cov teeb meem thiab qhev –
<IP CHAW NYOB> tswv
<IP CHAW NYOB> qhev
- $ ssh-copy-id-i $HOME/ssh/id_rsa pub qhev – Qhov no hais kom ua raws li no yuav tsum tau tseg rau master kom muaj cov passwordless tshwm sim. Peb yuav tsum tau login siv tib tus neeg siv rau tag nrho cov cav tov. Yog peb xav tau ib lo lus zais, Peb yuav teev nws manually.
- Tam sim no peb qhib cov ntaub ntawv ob – conf/master thiab conf/slaves. Lub conf/master txhais lub npe nodes ntawm peb multi node cluster. Lub conf/slaves ntaub ntawv teev cov hom tsiaj uas cov Hadoop Slave yuav tau khiav.
- Edit lub conf/core-site.xml ua tau raws li cov nram qab no –
<khoom>
<lub npe>FS.default.name</lub npe>
<tus nqi>hdfs://tswv:54310</tus nqi>
</khoom>
- Edit lub conf/mapred-site.xml xml kom muaj cov nram qab no –
<khoom>
<lub npe>mapred.Job.tracker</lub npe>
<tus nqi>hdfs://tswv:54311</tus nqi>
</khoom>
- Edit lub conf/hdfs-site.xml ua tau cov nram no –
<khoom>
<lub npe>dfs.replication</lub npe>
<tus nqi>2</tus nqi>
</khoom>
- Edit lub conf/mapred-site.xml xml kom muaj cov nram qab no –
<khoom>
<lub npe>zajlus.koog tsev kawm ntawv..</lub npe>
<tus nqi>${ntoo tshauv}/tswj/zej zog</tus nqi>
</khoom>
<khoom>
<lub npe>map.map.tasks</lub npe>
<tus nqi>50</tus nqi>
</khoom>
<khoom>
<lub npe>txo... txo cov kev ua hauj lwm</lub npe>
<tus nqi>5</tus nqi>
</khoom>
Tam sim no pib lub tswv yim los ntawm kev siv tus hais kom ua raws li hais.
[Chaws] hlwb/start-dfs.sh [/Chaws]
Ib zaug pib, Xyuas tus pej xeem me nyuam ntawm tus tswv uas siv jps hais kom ua. Koj yuav tsum tau ua raws li cov lus qhia nram no –
[Chaws]
14799 Npe Menyuam Yaus
15314 Jps
16977 secondaryNeNode
[/Chaws]
Rau cov qhev tso zis yuav tsum muaj raws li hauv qab no.
[Chaws]
15183 DataNode
15616 Jps
[/Chaws]
Tam sim no pib lub MapReduce daemons uas siv cov hais raws li hais.
[Chaws]
$ bin/start-mapred.sh
[/Chaws]
Thaum pib xyuas cov pej xeem Mes Kas uas siv jps command. Koj yuav tsum tau ua raws li cov lus qhia nram no –
[Chaws]
16017 Jps
14799 Npe Menyuam Yaus
15596 JobTracker
14977 SecondaryNameNode
[/Chaws]
Thiab rau cov qhev cov tso zis yuav tsum muaj raws li nram qab no.
[Chaws]
15183 DataNode
15897 TaskTracker
16284 Jps
[/Chaws]
Txoj kev: Nyob rau saum toj no peb tau them ntau hom hadoop hadoop txuj thiab lawv tej lus. Tab sis peb yuav tsum ceev faj thaum xaiv cov plahaum plahaum. Hom muaj lawv lub hom phiaj. Ces tus beginners yuav tsum pib nrog tib hom plahaum thiab mam li npaj nrog lwm txoj kev xaiv.
Peb summarize peb sib tham nrog lub mos txwv nram qab no
- Apache Hadoop yuav muab ntsia tau rau hauv peb hom kev sib txawv –
- Ncaws pob
- Pseudo faib node
- Faib ntab
- Ib zaug xwb hom yog qhov yooj yim tshaj plaws rau nruab nrab thiab tau pib.
- Yog peb xav tau clusters tab sis muaj ib tug tsis muaj leej twg muaj, Ces peb yuav tsum mus rau Pseudo faib tawm hom
- Rau nruab nrab lub hom phiaj peb yuav tsum muaj cov pseudo faib tawm mode ntsia ua ntej.
