spark-blocktransferservice.adoc

BlockTransferService

BlockTransferService is a contract for specialized ShuffleClient objects that can fetch and upload blocks in synchronously and asynchronously.

Note	BlockTransferService is a private[spark] abstract class.

Note	NettyBlockTransferService is the only available implementation of BlockTransferService Contract.

BlockTransferService Contract

Every BlockTransferService offers the following:

• init that accepts BlockDataManager for storing or fetching blocks. It is assumed that the method is called before the service is considered fully operational.

  init(blockDataManager: BlockDataManager): Unit

• port the service listens to.

  port: Int

• hostName the service listens to.

  hostName: String

• uploadBlock to upload a block (of ManagedBuffer identified by BlockId) to a remote hostname and port.

  uploadBlock(
    hostname: String,
    port: Int,
    execId: String,
    blockId: BlockId,
    blockData: ManagedBuffer,
    level: StorageLevel,
    classTag: ClassTag[_]): Future[Unit]

• Synchronous (and hence blocking) fetchBlockSync to fetch one block blockId (that corresponds to the ShuffleClient parent’s asynchronous fetchBlocks).

  fetchBlockSync(
    host: String,
    port: Int,
    execId: String,
    blockId: String): ManagedBuffer

  fetchBlockSync is a mere wrapper around fetchBlocks to fetch one blockId block that waits until the fetch finishes.

• Synchronous (and hence blocking) uploadBlockSync to upload a block (of ManagedBuffer identified by BlockId) to a remote hostname and port.

  uploadBlockSync(
    hostname: String,
    port: Int,
    execId: String,
    blockId: BlockId,
    blockData: ManagedBuffer,
    level: StorageLevel,
    classTag: ClassTag[_]): Unit

  uploadBlockSync is a mere wrapper around uploadBlock that waits until the upload finishes.

NettyBlockTransferService - Netty-Based BlockTransferService

Caution

FIXME