Still maintained?

[elgalu]

Is this going to be still maintained?

I see duplicated effort on Spark 2.3 and also on https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

[liyinan926]

We stopped development in this fork, and have been focusing on upstreaming since late last year. We generally encourage people to use the upstream 2.3 release instead of this fork. https://github.com/GoogleCloudPlatform/spark-on-k8s-operator is a separate project for making running Spark on Kubernetes idiomatically.

[witten]

According to these docs, upstream Spark on Kubernetes does not include support for things like PySpark, and those docs also indicate that new Spark on Kubernetes features will be "incubated" in apache-spark-on-k8s. So that leaves this project as the only option for use cases not yet supported by upstream. Do you recommend that anyone still using your fork switch to consuming unreleased changesets in branch-2.2-kubernetes? Or do you really plan to halt all development?

In any case, I'd recommend updating the README to reflect the current state of the project, so it's clear that new or existing users should steer clear and use upstream instead if that's your intent.

[harbesc]

Any update? We are looking forward to using secure HDFS with kerberos and other improvements not in the latest release!

[ifilonenko]

I am working on secure HDFS support and SparkR. I just put up a PR for PySpark :) in PR 21092.