ApacheTez:NextGenerationExecutionEngineuponHadoopJeffZhangPag
eOutlineTezIntroductionTezAPITezInternalTezProjectStatusQ&
ATez–IntroductionDistributedexecutionframeworktargetedtowa
rdsdata-processingapplications.Basedonexpressingacomputatio
nasadataflowgraph.Highlycustomizabletomeetabroadspectru
mofusecases.BuiltontopofYARN–theresourcemanagementfra
meworkforHadoop.OpensourceApacheprojectandApachelicensed.
PageHadoop1->Hadoop2LayeredResourceManagement–YARNExecuti
onEngine–TezUserAPI–Hive,Pig,Cascading,YourApp!Monolit
hicResourceManagementExecutionEngineUserAPIHADOOP2.0HADOOP1.
0DataFlowPigSQLHiveOthers(Cascading)BatchMapReduceRealTimeStre
amProcessingStormOnlineDataProcessingHBase,AccumuloTez(executi
onengine)Pig(dataflow)Hive(sql)Others(cascading)YARN(clusterre
sourcemanagement)MapReduce(clusterresourcemanagement&datapr
ocessing)HDFS2(redundant,reliablestorage)HDFS(redundant,reliab
lestorage)Pig/Hive-MRversusPig/Hive-TezJob1Job2I/OSynchroni
zationBarrierI/OSynchronizationBarrierSingleJobJob3Pig/Hive
-TezPig/Hive-MRPageOutlineTezIntroductionTezAPITezInternal
TezProjectInfoQ&ATez–ExpressingthecomputationTezprovides
thefollowingAPIstodefinetheprocessingDAGAPI(Vertex,Edg
e)Definesthestructureofthedataprocessingandtherelations
hipbetweenproducersandconsumersEnabledefinitionofcomplexd
ataflowpipelinesusingsimplegraphconnectionAPI’s.Tezexpan
dsthelogicalDAGatruntimeThisishowallthetasksinthejob
getspecifiedRuntimeAPI(Task)Definestheinterfacesusingwh
ichtheframeworkandappcodeinteractwitheachotherAppcodet
ransformsdataandmovesitbetweentasksThisishowwespecifyw
hatactuallyexecutesineachtaskontheclusternodesPageTez–
DAGAPISimpleDAGdefinitionAPI//DefineDAGDAGdag=newDA
G();//DefineVertexVertexmap1=newVertex(MapProcessor.clas
s);//DefineEdgeEdgeedge1=Edge(map1,reduce1,SCATTER_GA
THER,PERSISTED,SEQUENTIAL,Output.class,Input.class);//Conn
ectthemdag.addVertex(map1).addEdge(edge)…PageTez–DAGA
PIEdgepropertiesdefinetheconnectionbetweenproducerandcons
umertasksintheDAGDatamovement–Definesroutingofdatabetw
eentasksOne-To-One:Datafromtheithproducertaskroutestot
heithconsumertask.Broadcast:Datafromaproducertaskroutes
toallconsumertasks.Scatter-Gather:Producertasksscatterda
taintoshardsandconsumertasksgatherthedata.Theithshard
fromallproducertasksroutestotheithconsumertask.Schedulin
g–DefineswhenaconsumertaskisscheduledSequential:Consume
rtaskmaybescheduledafteraproducertaskcompletes.Concurren
t:Consumertaskmustbeco-scheduledwithaproducertask.Data
source–Definesthelifetime/reliabilityofataskoutputPersist
ed:Outputwillbeavailableafterthetaskexits.Outputmaybe
lostlateron.Persisted-Reliable:Outputisreliablystoredand
willalwaysbeavailableEphemeral:Outputisavailableonlywhi
letheproducertaskisrunningPageTez–RuntimeAPI(IPO)Flexib
leInputs-Processor-OutputsModelThinAPIlayertowraparoundar
bitraryapplicationcodeComposeinputs,processorandoutputsto
executearbitraryprocessingEventroutingbasedcontrolplanearc
hitectureApplicationsdecidelogicaldataformatanddatatransfe
rtechnologyCustomizeforperformanceBuilt-inimplementationsfor
Hadoop2.0dataservices–HDFSandYARNShuffleService.Builto
nthesameAPI.Yourimplsareasfirstclassasours!InputProces
sorOutputPageTez–RuntimeAPI(VertexManager)VertexManagerCont
rolontheflowexecutionengineinvertexlevelTez–LogicalDAG
expansionatRuntimeMap1Map2Reduce1Reduce2JoinPageTez–Perform
anceBenefitsofexpressingthedataprocessingasaDAGReducingo
verheadsandqueuingeffectsGivessystemtheglobalpictureforb
etterplanningEfficientuseofresourcesRe-useresourcestomaxim
izeutilizationPre-launch,pre-warmandcacheLocality&resource
awareschedulingSupportforapplicationdefinedDAGmodifications
atruntimeforoptimizedexecutionChangetaskconcurrencyChange
taskschedulingChangeDAGedgesChangeDAGverticesPageTez–Ben
efitsofDAGexecutionFasterExecutionandHigherPredictabilityE
liminatereplicatedwritebarrierbetweensuccessivecomputations
.Eliminatejoblaunchoverheadofworkflowjobs.Eliminateextras
tageofmapreadsineveryworkflowjob.Eliminatequeueandresou
rcecontentionsufferedbyworkflowjobsthatarestartedaftera
predecessorjobcompletes.Betterlocalitybecausetheenginehas
theglobalpicturePig/Hive-TezPig/Hive-MRPageOutlineTezInt
roductionTezAPITezInternalTezProjectInfoQ&ATezSystemDiagr
amTez–ContainerRe-UseReuseYARNcontainers/JVMstolaunchnew
tasksReduceschedulingandlaunchingdelaysSharedin-memorydata
acrosstasksJVMJITfriendlyexecutionYARNContainer/JVMYARNCo
ntainerTezApplicationMasterTezChildStartTaskTezTask1TaskDoneS
haredObjectsTezTask2StartTaskPageTez–CustomizableCoreEngin
eStartvertexVertexManagerDeterminestaskparallelismDeterminesw
hentasksinavertexcanstart.DAGSchedulerDeterminespriority
oftaskTaskSchedulerAllocatescontainersfromYARNandassignst
hemtotasksGetcontainerVertex-1GetPriorityDAGSchedulerTaskSche
dulerStartvertexVertexManagerStarttasksVertex-2GetPriorityGetc
ontainerPageTez–SessionsSessionsStandardconceptsofpre-launc
handpre-warmappliedKeyforinteractivequeriesRepresentsacon
nectionbetweentheuserandtheclusterMultipleDAGsexecutedin
thesamesessionContainersre-usedacrossqueriesTakescareofd
atalocalityandreleasingresourceswhenidleStartSessionSubmit
DAGClientApplicationMasterTaskSchedulerPreWarmedJVMSharedObj
ectRegistryContainerPoolPageUsecaseofTezSplitGroupby+J
oinOrderbyAutomaticReduceParallelismReduceSlowStart/Pre-launc
hSplitGroupby+Joinf=load‘foo’as(x,y,z)f1=GroupfByx
;f2=GroupfByy;j=Joing1bygroup,g2bygroupOrderbyf=Lo
ad‘foo’as(x,y);o=Orderfbyx;Tez–AutomaticReduceParall
elismEventModelMaptaskssenddatastatisticseventstotheRedu
ceVertexManager.VertexManagerPluggableapplicationlogicthat
understandsthedatastatisticsandcanformulatethecorrectpar
allelism.AdvisesvertexcontrolleronparallelismAppMasterData
SizeStatisticsVertexManagerMapVertexSetParallelismVertexSta
teMachineRe-RouteReduceVertexCancelTaskPageTez–ReduceSlowS
tart/Pre-launchEventModelMapcompletioneventssenttotheReduc
eVertexManager.VertexManagerPluggableapplicationlogicthatu
nderstandsthedatasize.Advisesthevertexcontrollertolaunch
thereducersbeforeallmapshavecompletedsothatshufflecan
start.AppMasterTaskCompletedVertexManagerMapVertexStartTask
sVertexStateMachineReduceVertexStartPageOutlineTezIntroductio
nTezAPITezInternalTezProjectStatusQ&ATez–CurrentstatusAp
acheTopLevelProjectRapiddevelopment.Over1500jirasopened.
Over1100resolvedGrowingcommunityofcontributorsandusersLate
streleaseis0.5FocusonstabilityTestingandqualityarehighes
tpriorityCodereadyanddeployedonmulti-nodeenvironmentsats
caleSupportforavasttopologyofDAGsAlreadyfunctionallyequi
valenttoMapReduce.ExistingMapReducejobscanbeexecutedon
TezwithfewornochangesApacheHive0.13releasesupportsTez
asanexecutionengine(HIVE-4660)ApachePigporttoTezisalso
done(PIG-3446)Cascading3.0supportTezPageTez–AdoptionApache
HiveHadoopstandardfordeclarativeaccessviaSQL-likeinterfac
eApachePigHadoopstandardforproceduralscriptingandpipeline
processingCascadingDeveloperfriendlyJavaAPIandSDKScalding(S
calaAPIonCascading)CommercialVendorsETL:UseTezinsteadof
MRorcustompipelinesAnalyticsVendors:UseTezasatargetpla
tformforscalingparallelanalyticaltoolstolargedata-setsPag
eTez–CommunityEarlyadoptersandcodecontributorswelcomeAdop
terstodrivemorescenarios.Contributorstomakethemhappen.Te
zmeetupfordevelopersandusershttp://www.meetup.com/Apache-Tez
-User-Grouphttp://http://www.meetup.com/Apache-Tez-User-Groupwww.
meetup.com/Apache-Tez-User-GroupTechnicalblogserieshttp://hort
onworks.com/blog/apache-tez-a-new-chapter-in-hadoop-data-processi
nghttp://http://hortonworks.com/blog/apache-tez-a-new-chapter-in-
hadoop-data-processinghortonworks.com/blog/apache-tez-a-new-chapt
er-in-hadoop-data-processingUsefullinksWorktracking:https://i
ssues.apache.org/jira/browse/TEZhttps://issues.apache.org/jira/br
owse/TEZCode:https://github.com/apache/incubator-tezhttps://http
s://github.com/apache/incubator-tezgithub.com/apachehttps://githu
b.com/apache/incubator-tez/tez?Developerlist:mailto:dev@tez.inc
ubator.apache.orgdev@mailto:dev@tez.incubator.apache.orgtez.apach
e.org?Userlist:mailto:user@tez.incubator.apache.orguser@mailto:
user@tez.incubator.apache.orgtez.apache.org?Issueslist:mailto:i
ssues@tez.incubator.apache.orgissues@mailto:issues@tez.incubator.
apache.orgtez.apache.orgPageTezVSSparkTezSparkSolveDAGComput
ationSolveDAGComputationIntegrateYarnfromitsbeginningBorrow
lotsofworkfromMapReduce(e.g.Shuffle)StartfromscratchDesi
gnforcomputingengineForgeneralapplicationdeveloperMoreAPI
ontheexecutionengineMorefriendlyonAPIGoodperformance(hav
epotentialtoimproveoncethestreamingshuffleisimplemented
orthein-memoryHDFSisintegrated)BetterperformanceBetterscal
abilityandstableThankYou!Questions&AnswersPageForanyonewh
ohasbeenworkingonMapReduce,thereisthisage-oldproblemar
ound“howdoIfigureoutthecorrectnumberofreducers?”.Wegu
esssomenumberatcompile-timeandusuallythatturnsouttobe
incorrectatrun-time.Let’sseehowwecanusetheTezmodelto
fixthat.SohereisthisMapVertexandthisReduceVertex,whic
hhavethesetasksrunningandyouhavetheVertexManagerrunnin
ginsidetheframework…[CLICK]TheMapTaskscansendDataSize
StatisticstotheVertexManager,whichcanthenextrapolatethos
estatisticstofigureout“whatwouldbethefinalsizeofthed
atawhenalloftheseMapsfinish?”.Basedonthat,itcanrealiz
ethatthedatasizeisactuallysmallerthanexpected,andIcan
actuallyruntworeducetasksinsteadofthree.[CLICK]TheVerte
xManagersendsaSetParalellismcommandtotheframeworkwhich
changestheroutinginformationin-betweenthesetwotasksandal
socancelsthelasttask.Sothenewplanlookslikethis…which
ishowTezhandlestheconcurrencyproblemofplanre-configurati
on.Whatabout“whendoyouactuallywanttolaunchthosetasks?”
?InMapReduce,thereisaninterestingconceptofpre-launchingt
heReducers,inwhicheventhoughthereducersaregoingtoreadtheoutputfrom100mappers,wemaywanttolaunchtheReducerswhenonly50mappershavefinished–becausethentheycanusethecapacityintheclustertostartpre-fetchingthedatafromthecompletedmapperswhiletherestoftheirinputsstillcomplete.HowcanwedothisinTez?Again,thesameVertexManagercomesintothepicture.[CLICK]AstheMapTaskscompletetheysendTaskCompletednotificationstotheVertexManager.ThereitcandecidethatenoughMaptaskshavefinishedthatIneedtostartoneofmyreducerssothatitcanstartpre-fetchingthedata.[CLICK]ItsendsaStartTasknotificationtotheframework,whichresultsinoneoftheReduceTasksstarting.Hivehaswrittenit’sownprocessor |
|
