Rewrite _describe_git in KedroSession to use python based implementation of the git commands

[noklam]

https://github.com/FriendCode/gittle seems to be the easiest option, although it hasn't been maintained for years and it's probably not a good idea to rely on a seemingly abandoned project. Another option is to implement the two functions we need ourselves as per this example here: https://wyag.thb.lt/#orgd8ab0ff (which is more fun, but probably harder, especially for the git status check).

Background

Traditionally we have been using git within kedro run to track metadata about the working directory. The current way we do is using subprocess which calls the git executable directly. However, it causes some issues when we are in some read-only / Databricks's dbfs / spark-submit job.

The idea is to use native python calls to eliminate the need for an external process.

Todo:

• Find alternatives (open-source library or re-implement the 2 functions we need)
  Possible solutions

• https://github.com/jelmer/dulwich (A low-level library that is still actively maintained)
• https://github.com/FriendCode/gittle (A high-level library, but is no longer maintained)
• Implement ourselves (reference: https://wyag.thb.lt/#orgd8ab0ff)

Use Case

• _describe_git

  kedro/kedro/framework/session/session.py

  Lines 50 to 71 in b3c78e5

  	def _describe_git(project_path: Path) -> Dict[str, Dict[str, Any]]:
  	project_path = str(project_path)
  	try:
  	res = subprocess.check_output(
  	["git", "rev-parse", "--short", "HEAD"],
  	cwd=project_path,
  	stderr=subprocess.STDOUT,
  	)
  	git_data = {"commit_sha": res.decode().strip()} # type: Dict[str, Any]
  	git_status_res = subprocess.check_output(
  	["git", "status", "--short"],
  	cwd=project_path,
  	stderr=subprocess.STDOUT,
  	)
  	git_data["dirty"] = bool(git_status_res.decode().strip())
  	# `subprocess.check_output()` raises `NotADirectoryError` on Windows
  	except (subprocess.CalledProcessError, FileNotFoundError, NotADirectoryError):
  	logging.getLogger(__name__).warning("Unable to git describe %s", project_path)
  	return {}
  	return {"git": git_data}

  • ["git", "rev-parse", "--short", "HEAD"]
  • ["git", "status", "--short"],

[idanov]

Unfortunately gitpython will not fit the bill as it needs the git executable, which is exactly what we want to avoid depending on.

[avan-sh]

I'm not sure if this error would still pop up after this change to the function. The fatal error was also in directories without .git as fixed by that change.

If I remember right, the error on read-only projects was also related to not being being able to access info.log.

[antonymilne]

As per @avan-sh, now we catch this error I think changing this is not as high priority as it was before. I guess we still want to do it, just it's not essential to get things working on databricks any more.

FYI @avan-sh the thing about not being able to write to log files on a real-only file system is also fixed now and will be release in 0.18.2.

[avan-sh]

@AntonyMilneQB while I understand this is not a priority anymore, I don't get why this is still needed. It might be that using subprocesses is not be ideal.

[merelcht]

Closing in favour of #2051