spark-submit 1.3.0
Features:
- Addition of
spark_submit.SparkJob.get_id()
method to obtain the Spark submission ID - Use all environment variables from
os.environ
ifuse_env_vars=True
inspark_submit.SparkJob.submit()
- Additional environment variables can be used in
spark_submit.SparkJob.submit()
withextra_env_vars
argument
Misc/Internal
spark_submit.SparkJob.submit()
now uses keyword arguments only- Argument
await_result
is changed topoll_time
inspark_submit.SparkJob.submit()
- No default environment variables to be used with
spark_submit.SparkJob.submit()
- Code standards improvements and general refactoring