>>>>> Databricks Cluster UI (~ Ref Youtube Video : ~Mr.Ktalks Tech ) # Cluster Diagram Explanation: Cluster : Nodes and Executors Set of VM 1 Node = 1 Executor Each Executor : has 1/more Cores 1 Core = 1 Partition Each Core : takes 1 task and used for parallism Driver : Step 1: Write Code in Driver Step 2: Everything get's Divides into STAGES and TASKS This is Done with help of DAG DAG: divides all jobs in form of stages and task each TASK: executes on executor each EXECUTOR: it made of essential cores each CORE: define degree of parallelism that will happen when JOB RUNS.... # Cluster: Set of Virtual Machines to do work Creating Computer Resources for processing BigData Types: 1] All Purpose Compute 2] Job Compute 1] All-Purpose Compute [Everything] Analyze data in NB Create, Terminate and Restart Cost: Expensive 2] JOB Compute [Just to run NB as Job with ADF pipeline and Databricks] Just Support Running a NB as Job No restart Cost : L...
Comments
Post a Comment