spark-ExternalClusterManager.adoc

ExternalClusterManager

ExternalClusterManager is a contract for pluggable cluster managers.

Note	It was introduced in SPARK-13904 Add support for pluggable cluster manager.

It is assumed that ExternalClusterManager implementations are available as Java services (with service markers under META-INF/services directory).

Note	ExternalClusterManager is a private[spark] trait in org.apache.spark.scheduler package.

Note	The only known implementation of the ExternalClusterManager contract in Spark is YarnClusterManager.

Note	SparkContext finds a ExternalClusterManager for a master URL using an internal getClusterManager.

ExternalClusterManager Contract

initialize

initialize(scheduler: TaskScheduler, backend: SchedulerBackend): Unit

canCreate

canCreate(masterURL: String): Boolean

Note	It is used when finding the external cluster manager for a master URL (in SparkContext).

createTaskScheduler

createTaskScheduler(sc: SparkContext, masterURL: String): TaskScheduler

createSchedulerBackend

createSchedulerBackend(sc: SparkContext,
  masterURL: String,
  scheduler: TaskScheduler): SchedulerBackend