Upgrading Kedro is hard

[astrojuanlu]

Description

Our users find it difficult to upgrade Kedro in their projects. Just from a couple of recent user interviews:

User 1:

• "We sometimes try to upgrade the version [paraphrasing] and upgrading versions impacts other packages that we use"
• "It creates trouble for us when we try to redeploy something that we deployed one year ago"

User 2:

• "I begin a project, and after 3 months I need to upgrade the project with a new Kedro"
• "It’s very laborious for us to upgrade"
• They end up with tens of projects, each of them with slightly different versions of Kedro, which is difficult for the team to maintain

There are also very clear signs that old Kedro versions tend to live on for a very long time:

People like @inigohidalgo have been reporting their long journey to upgrade from old Kedro versions in our public Slack.

And finally, we also have lots of internal evidence as well that big projects get stuck on old Kedro versions.

Is this a problem?

One could argue: "it it works, don't touch it". So the fact that Kedro is pinned to the latest version is not necessarily a bad thing.

However, with resource constrained teams maintaining many projects, each of them with slightly different versions of Kedro, this can become a mess to maintain.

For those teams who would wish to have a uniform Kedro versioning, we should provide a more clean upgrade path.

What has been done

For 0.19 we went ahead and added a detailed migration guide https://docs.kedro.org/en/latest/resources/migration.html#migrate-an-existing-project-that-uses-kedro-0-18-to-use-0-19

Where to go from here

However, it seems to not yet be enough.

What else can we do to make these migrations easier?

[datajoely]

Anecdotally our most sophisticated users are the ones that get stuck the most since they typically verge into complex hooks, dynamism and coupling to some of the internals. With 1.0.0 on the horizon hopefully these internals have stabilised and won't be as incompatible/painful going forward. This is my main hypothesis why I think some sort of automated tooling may not move the needle.

I'm still very much of the opinion that we need to build user-facing superpowers that make the effort to upgrade worth it. Introducing the settings.py in 0.18.x was a breaking change very much more important for the Kedro developers rather than Kedro users. Cynically one could argue we could do a better job making something like OmergaConf a 0.19.x, even if it technically could be made to work in a non-breaking way.

[astrojuanlu]

Oops closing this in favour of #3960 - will copy your comment over @datajoely 🙏🏼