Cov qauv tsim MapReduce
Txheej txheem cej luam: Tsim txawv yog ib lub sij hawm uas tam sim no siv tau yuav luag txhua teb ntawm txoj kev loj hlob software. Ntxwv no tsis muaj dab tsi tab sis tej pov thawj thiab qhabnias tsim caag siv yuav kom daws tau cov teeb meem ua lag ua luam. MapReduce yog ib tug tshuab sau tshiab thiab nws kuj muaj nws cov qauv tsim los daws cov teeb meem le caag.
Nyob rau cov tshooj no, Kuv yuav piav txog ntau tsim qauv siv nyob rau hauv lub moj khaum MapReduce.
Taw qhia: Txo tej daim ntawv qhia yuav siv cov ntaub ntawv cov ntaub ntawv uas nyob lub npe ntawm tus nab npawb ntawm cov computers. Nws muab Area technical college, thiab yus ntawm peb yuav zoo li cas thiab peb tsis tau dab tsi. Qhov no ua rau cov kev peb ua tau, lesser. Tib lub sijhawm, peb yuav xyuas tau seb yuav ua li cas los daws kom tau ib qho teeb meem nrog xyuas. Qauv tsim muaj tau soj ntsuam kuaj txog teeb meem ntau txij ua ke mus ua ke thiab pom zoo txog qib uas abstraction.
MapReduce tsim cov neeg nyob lub luag hauj lwm tib yam rau ib qhov chaw me me thiab cov teeb meem thiab cov kev daws. Lawv muab ib tug hais moj khaum los daws kom tau peb le caag tej ntaub ntawv hais txog tej teeb meem, tsis ywm rau cov teeb meem sau. MapReduce tsim qauv qhia ib qho lus nrog ua num ua ke nyob rau kev siv MapReduce.
MapReduce tsim txawv: Tam sim no peb tham txog qauv siv ntau le caag ib yog ib tug nyob rau sab lus nram qab no.
Xam xyuas:
Teeb meem Statement – Peb xav tias peb muaj ib pawg ntawm cov ntaub ntawv uas txhua daim ntawv muaj cov txheej lus. Tam sim no peb muaj ib tug uas hais kom xam cov lus ntawm txhua lub sij hawm nyob rau daim ntawv.
Kua- Nyob rau hauv qhov chaws snippet hauv qab no peb muaj ib Mapper uas tam sim no ib tsab rau '1' rau txhua lub sij hawm nws encounters. Lub reducer no traverses txhua daim ntawv ntawm no sawv daws yuav, sums thiab ua rau cov.
Qhia 1: Xam Code – 1
[Chaws]
hoob kawm Mapper
daim ntawv qhia txoj kev(daim docid, doc d)
rau txhua lub sij hawm t hauv doc d ua
Emit(lub sij hawm t, suav 1)
hoob kawm Reducer
txoj kev txo tej(lub sij hawm t, suav [muaj c1, C2,…])
cnt = 0
rau cov suav c hauv [muaj c1, C2,…] ua
cnt = cnt + c
Emit(lub sij hawm t, suav cnt)
[/Chaws]
Ho yooj yim, tab sis cov cai saum toj no muaj ib tug disadvantage cuab kev uas lub siab npaum li cas dummy kom yog tawm txim liab yog lub Mapper. Tam sim no peb yuav txwv tau kom qhov no los ntawm summing rau tej txee rau ib daim ntawv.
Qhia 2: Xam Code – 2
[Chaws]
hoob kawm Mapper
daim ntawv qhia txoj kev(daim docid, doc d)
HArray = AssociativeArray tshiab
rau txhua lub sij hawm t hauv doc d ua
HArray{t} = HArray{t} + 1
rau txhua lub sij hawm t hauv HArray ua
Emit(lub sij hawm t, suav HArray{t})
[/Chaws]
Tam sim no, thiaj li yuav noog ntawm tej txee rau tag nrho cov ntaub ntawv cia tau combiners –
Qhia 3: Xam Code – 3
[Chaws]
hoob kawm Mapper
daim ntawv qhia txoj kev(daim docid, doc d)
rau txhua lub sij hawm ter hauv doc d ua
Emit(lub sij hawm ter, suav 1)
hoob kawm Combiner
txoj kev uas muab(lub sij hawm ter, [muaj c1, C2,…])
zaum = 0
rau cov suav cnt hauv [muaj c1, C2,…] ua
zaum = zaum + cnt
Emit(lub sij hawm t, suav zaum)
hoob kawm Reducer
txoj kev txo tej(lub sij hawm t, suav [muaj c1, C2,…])
totalCnt = 0
rau cov suav cnt hauv [muaj c1, C2,…] ua
totalCnt = totalCnt + cnt
Emit(lub sij hawm t, suav totalCnt)
[/Chaws]
Collating:
Teeb meem Statement –
Peb muaj ib pawg khoom thiab tej nuj nqi ntawm ib yam khoom. Tam sim no nws yuav tsum tseg tag nrho cov khoom uas muaj nqi ntawm cov kev ua los yog ua tej lwm le caag uas yuav tsum tau tag nrho tej khoom uas yuav tso npe rau hauv ib pawg.
Kua-
Peb muaj cov tshuaj uas tus mapper computes tus zauv muaj nuj nqi rau ib yam khoom thiab rov leg ib qhov tseem ceeb thiab qhov khoom raws li nws tus nqi. Lub luag hauj lwm ntawm tus reducer yog nqa tag nrho cov grouped khoom thiab txheej txheem rau lawv los yog txuag lawv.
Filtering (Grep), Parsing thiab Validation:
Teeb meem Statement –
Peb muab peb txheej cov ntaub ntawv muaj qhov yuav tsum tau mus sau tag nrho cov ntaub ntawv uas muaj tej yam kev mob los txia tau cov ntaub ntawv no mus rau ib lwm sawv cev tawm tswv yim no. Thib ib sab ntawm qhov teeb meem nrog rau cov paub tab li – ntawv parsing thiab extraction ntawm lub qhov tseem ceeb.
Kua-
Tshuaj rau qhov teeb meem no yog heev ncaj nraim rau pem hauv ntej – peb muaj ib tus mapper uas yuav siv ib daim zuj zus thiab rov cov khoom uas ua tau qhov kev.
Faib hauj lwm tiav:
Teeb meem Statement –
Cia peb muab peb muaj coob computational qhov teeb meem uas tau muab faib ua ntau yam ntawm tag nrho cov qhov xav tau chav tuav qhov tshwm sim zaum kawg.
Kua-
Tshuaj rau qhov teeb meem no yog phua lub specifications rau txheej specification uas yog muab raws li tej ntaub ntawv input rau lub mappers. Hnub no mappers yuav siv sij hawm ib specification ib lub sij hawm ua ntaub ntawv input thiab processes lawv thiab ua rau cov ntsiab. Cov hauj lwm ntawm tus reducer yuav muab tag nrho cov qhab thiab ua tus kawg ntog.
Lem iterative lus:
Teeb meem Statement –
Cia peb muab peb lub network ntawm cov chaw muaj muaj tshwm sim tej kev txawj lawv. Peb yuav tsum los laij lub xeev ntawm ib qhov chaw nyob ntawm cov khoom teejtug uas yog lwm qhov chaw hauv lub tsev. Lub xeev no yuav siv tau los sawv cev rau tus nyob deb li cas rau lwm cov ntshav uas yog tias muaj ib tug neeg nyob ze nrog tej yam khoom thiab cov yam ntxwv and.
Kua-
Peb muaj ib tug network uas stores txheej o thiab txhua ntawm muaj cov ntaub ntawv ntawm ib daim ntawv teev uas nyob ib sab ntawm tawm lawv tus ID. Conceptually, cov hauj lwm uas MapReduce muaj tau txoj kev iterative thiab rau txhua tus iteration tus ntawm xa lus rau nws tej zej zog. Thiab ces tus neeg nyob ze ib siab nws xeev los hais uas tau txais. Cov iterations yog haujlwm los ntawm tej yam mob e.g. tsau ntau iterations maximal. Los ntawm cov kev xav thiab pom, lub Mapper rov lus rau txhua ntawm muab daim ID rau ntawm qhov uas nyob ib sab ntawm los ua ib qhov tseem ceeb. Yog li ntawd, tag nrho cov lus grouped yog los ntawm cov khoom ntawm thiab li tus reducer yuav los txheeb peev hauv lub xeev thiab rewrites tus ntawm uas lub xeev tshiab.
Qhia 4: Iterative tam huas
[Chaws]
hoob kawm Mapper
daim ntawv qhia txoj kev(daim ID nId, kwv NObj)
Emit(daim ID nId, kwv NObj)
rau txhua daim mus rau NObj.OutgoingRelations ua
Emit(npav mus, lus sam getMessage(NObj))
hoob kawm Reducer
txoj kev txo tej(daim ID m, [s1, s2,…])
M = thov
lus = []
rau tag nrho s hauv [s1, s2,…] ua
yog IsObject(s) ces
M = s
ntxiv // s yog lus
Messages.ADD(s)
M.State = calculateState(lus)
Emit(daim ID m, yam khoom M)
[/Chaws]
Qhov dav thawj nrhiav (Qhov no yog ib cov ntaub ntawv kawm):
Teeb meem Statement –
Cia peb muab peb muaj ib lub teeb thiab nws yuav tsum los xam kom cov kev coj ntawm ib qhov ntawm rau tag nrho lwm yam o nyob rau hauv lub teeb. Qhov no yog hu ua cov hops.
Kua-
Cov tshuaj yuav tau, thawj lub plua ntawm emits 0 rau tag nrho nws tej zej zog. Ces tus tej zej zog propagate cov txee no tom qab nws yog incrementing 1 rau ib plhaw.
Yus qhov tseem ceeb:
Teeb meem Statement –
Peb xav tau cov ntaub ntawv uas muaj liaj teb M thiab N txheej peb. Lub no yog xam tag nrho nws qhov tseem ceeb ntawm daim teb M rau txhua tus subset ntawm tus tib neeg N.
Kua-
Cov tshuaj rau qhov teeb meem no tau muab teev tseg rau hauv theem ob. Nyob hauv cov thawj theem, cov mapper ua dummy tej txee rau ib khub M thiab N. Ces tus reducer thiaj suav tau ceeb tag nrho cov lus rau ib khub. Lub hom phiaj rau theem no kom muaj uniqueness ntawm M qhov tseem ceeb yog. Nyob rau theem ob, officers yog grouped los ntawm N thiab cov zauv ntawm tag nrho cov payment rau ib pawg twg.
Txoj kev: Nyob rau cov tshooj no peb muaj sab laj tsim txawv le caag uas feem ntau siv los daws kom tau cov ntaub ntawv hais txog le caag. MapReduce tsim qauv yog tujtaws evolving, ces peb yuav pom ntau ncaav tsim hauv near lawm yav tom ntej. Peb cia saib lub ntsab lug peb sib tham nyob rau nram qab no txhob –
- Daim ntawv qhia kom tsis txhob yuav siv cov ntaub ntawv cov ntaub ntawv uas nyob lub npe ntawm tus tswv tsev uas xyuas hauv computer.
- Qauv tsim yog siv yuav kom daws tau cov teeb meem.
- Ib txhia yeej siv tsim qauv no muaj nyob rau hauv – MapReduce
- Xam xyuas – suav thiab Summing
- Collating
- Filtering Parsing thiab Validation
- Faib hauj lwm tiav
- Lus sam iterative dua
- Qhov dav thawj nrhiav
- Tsis mus nrhiav yus
