Skip to content

spark-submit 1.3.0

Compare
Choose a tag to compare
@PApostol PApostol released this 30 Jul 13:39
· 18 commits to master since this release

Features:

  • Addition of spark_submit.SparkJob.get_id() method to obtain the Spark submission ID
  • Use all environment variables from os.environ if use_env_vars=True in spark_submit.SparkJob.submit()
  • Additional environment variables can be used in spark_submit.SparkJob.submit() with extra_env_vars argument

Misc/Internal

  • spark_submit.SparkJob.submit() now uses keyword arguments only
  • Argument await_result is changed to poll_time in spark_submit.SparkJob.submit()
  • No default environment variables to be used with spark_submit.SparkJob.submit()
  • Code standards improvements and general refactoring