Flatpak version of EDMC

[nolt]

Flatpak version for Steam Deck

It's nice to see flatpak version of EDMC.

Expected behavior

As you may know or not in default on SteamOS Steam Deck additional installation of the packages with pacman is disabled and not recommended (after every update system partition is formatted thus non default packages will be reverted but not flatpak apps). So the only way to have working EDMC is all the time reinstalling it or installing flatpak version of EDMC which is missing.

[Athanasius]

I'm not sure any of the development team has any experience with producing Flatpaks. Certainly I don't know that any of us have a Steam Deck in order to test if any Flatpak we produce definitely works on one.

I'm not saying no, but this will be contingent on either @A-UNDERSCORE-D thinking it's an interesting thing or someone else stepping up with a PR to facilitate building a Flatpak via GitHub actions.

[A-UNDERSCORE-D]

Without a bunch of work we'd likely need to be careful about permissions and such. Building it is done their side iirc but yeah it's not something I really want to do. I don't have direct access to a deck to verify behaviour either

[nolt]

Well if you worry about testing this is not a big deal. I think many people with deck could help including me. Besides you could build test environment on quemu using i.e. https://blogs.igalia.com/berto/2022/07/05/running-the-steam-decks-os-in-a-virtual-machine-using-qemu/ or any other virtualization options. In same way I successfuly build and run EDMC without messing around on the device itself.

As @Athanasius mentioned and what is the biggest problem is lack of Flatpak experience in development team, maybe someone from the EDMC community could help in future or so.

Thanks anyway for your reply 👍

[alterNERDtive]

So the only way to have working EDMC is all the time reinstalling it or installing flatpak version of EDMC which is missing.

Uh … just clone and run it?

[nolt]

Not really, you have to enable pacman (turn system as not read only). Install necessary packages (pip,tk) +requirements and run it. But as I said every SteamOS update I think packages will be removed.

[der-eismann]

While it really does make sense for the deck, I think most Linux users would also profit from a Flatpak. From my perspective I always struggle to find information about where to clone it locally, how the *.desktop file should look like etc. - having a Flatpak to install would increase comfort for most users. Also you don't have to worry about having the correct Python version, pip and all this stuff.

I also don't have much experience with building them, but I invested some time and tinkered something here. To build it yourself, clone the repo and exec

$ flatpak install flathub org.freedesktop.Platform//22.08 org.freedesktop.Sdk//22.08
$ flatpak-builder --user --install build-dir io.edcd.EDMarketConnector.yaml --force-clean
$ flatpak run io.edcd.EDMarketConnector

It runs fine and also integrates well with Gnome Software

One thing I'm unsure about is the permissions - user home or ~/.steam should be enough for most, but if you put it on another drive when on desktop it would not be working there. What do you think?

[A-UNDERSCORE-D]

if you want to do it properly within flatpak, steamapps, compatdata, and so on down to the c disk for the game. Any other users can use flatseal to move around perms

[nolt]

Well why we should do it in compatdata (emulated c drive)? I think EDMC should be located at /home/$USER/.local/share/edmc. This would be the best, because who knows where we have SteamLibrary (SD-card, other not default location). It would be faster to point out where are journals and other stuff.

[A-UNDERSCORE-D]

compatdata/$ElitesSteamID/path/to/journals

[A-UNDERSCORE-D]

as EDMC needs access to the game logs to have any use

[A-UNDERSCORE-D]

EDMC doesnt have to be rooted there, but it needs access to those directories, but not much more. Limited access is the philosophy to follow if you're gonna limit it

[der-eismann]

You're right, most people probably have it in their default path, the rest should be able to read the readme where we could add a hint regarding Flatseal. I just added https://github.com/der-eismann/edmc-flatpak/commit/74317b1629ebf63207e4739deba98bc6153c0f30 which also makes it easier to use because the steam folders are the only ones you see in the file picker for the journal.

Regarding config and plugins: these are located at ~/.var/app/io.edcd.EDMarketConnector/config/EDMarketConnector & ~/.var/app/io.edcd.EDMarketConnector/data/EDMarketConnector, which are default for Flatpak apps.

[der-eismann]

Any updates on this? Were you able to test it as I described? Should I request a repository for flathub?

[nolt]

I'll try to test it this weekend 👍

[nolt]

@der-eismann
I just tested application on SteamDeckOS VM. App builds and works, mine logs was found but at first time I had to point it out anyways (don't know if this is VM isssue or that I pushed logs after 1st run EDMC flatpak app).

After start there is a Warrning: about edmc-journal-lock.txt file (i've attached output from terminal)
but file has been created

(deck@steamdeck Elite Dangerous)$ ls -ltr
total 64
-rw-rw-r-- 1 deck deck   164 Nov  9 08:13 Journal.2022-11-09T171253.01.log
-rw-rw-r-- 1 deck deck   106 Nov  9 08:27 Cargo.json
-rw-rw-r-- 1 deck deck  1876 Nov  9 08:42 ModulesInfo.json
-rw-rw-r-- 1 deck deck   144 Nov  9 08:46 ShipLocker.json
-rw-rw-r-- 1 deck deck   323 Nov  9 08:48 NavRoute.json
-rw-rw-r-- 1 deck deck 32994 Nov  9 08:48 Journal.2022-11-09T172541.01.log
-rw-rw-r-- 1 deck deck    69 Nov  9 08:48 Status.json
-rw-r--r-- 1 deck deck   143 Dec  3 02:07 edmc-journal-lock.txt

To perform another test we need flathub repo to avoid building process.

terminal output: output.txt

[der-eismann]

I had to manually choose the log path as well, not sure if this is Linux-related in general. Regarding the lock file - in the output you posted there is neither lock nor warn when I search for it. Wrong logs maybe?

[nolt]

Hmm strange, I double checked again and no warrning. But at first run, I had to paste logs from another run of the application. NVM then, it seems to work.

About logs I installed flatseal to check maybe something is wrong with permissions but no, application had access. Well for me and I think other this is a minor thing and for now we shouldn't worry about it.

Can you take care repo on flathub to avoid build process?

[der-eismann]

I want to wait @A-UNDERSCORE-D's input first, a) because I'm not sure about the application ID and b) as they state it in their wiki:

Flathub is primarily intended as a service that is used by app developers to distribute their apps.

I'm happy to help wherever I can, but I don't want to be hasty without asking for permission first 🙂

[nolt]

Fully understand and agree with you. 👍

[A-UNDERSCORE-D]

Not currently in the country to really comment. Either way I'd want @Athanasius to weigh in as well

[Athanasius]

Well, as I got pinged, some comments:

1. For the Application ID, so long as flathub is OK with it not actually being a valid URL, I think io.edcd.EDMarketConnector is fine so long as we end up 'running'/owning this. Otherwise it should be something based on who is owning this flatpak, even if that differs from "application author".
2. Please don't use only 'edmc' and 'edmc.sh' for the main GUI application. That neglects the CLI program EDMC.py, and has the potential to confuse users. Call the GUI application edmarketconnector throughout. Yes, I know I'm about to use 'EDMC' to refer to the whole application below.
3. There might be some EDMC plugins that want to look at/fiddle with game binds/options files, so you'll need to provide access to that location as well, not just the Journals location.
4. Unless this is a "chicken and the egg" situation where you need to have done this before having the EDMC archive available ... Automating generating python3-requirements.yaml, or just using our requirements.txt instead of apparently hard-coding modules and their versions in python3-requirements.yaml will save you some work in the long run.
5. How much general github functionality is available? I'm thinking that, in the long run, this would work better if it just goes via the https://github.com/EDCD/EDMarketConnector/releases/latest URL and then picks out the necessary archive file, but in a github API manner. There is e.g. https://github.com/EDCD/EDMarketConnector/releases/latest/download/hashes.sum - this URL form only works for manually uploaded assets though. We can tweak the build workflow to also have a version.txt be there to make it very simply to find the latest.

[der-eismann]

Thanks for your valuable input!

For the Application ID, so long as flathub is OK with it not actually being a valid URL, I think io.edcd.EDMarketConnector is fine so long as we end up 'running'/owning this.

As they say in the wiki:

The Application ID should be a real URL of a domain that the app author has control over

So it's not an absolute must-have. For the tick detector project I saw that they use the URL https://tick.edcd.io, so at least edcd.io is registered and owned by someone affiliated with the orga? Maybe they could create a redirect for the EDMC project. But I think this should be fine.

Call the GUI application edmarketconnector throughout

Fine for me 🙂 However I'm unsure if it makes sense/is possible to make the CLI program available via flatpak as well 🤔

There might be some EDMC plugins that want to look at/fiddle with game binds/options files, so you'll need to provide access to that location as well, not just the Journals location.

I'm not an EDMC power user - where are these located? Also in the C:\Users stuff? Or do you mean the game files themselves?

Automating generating python3-requirements.yaml, or just using our requirements.txt instead of apparently hard-coding modules and their versions in python3-requirements.yaml will save you some work in the long run.

Actually this is kinda automated. As you can see at the top of the file it was generated with the flatpak-pip-generator. It is meant to be used with flatpak this way, so I'd keep it. Also regarding 5. I don't think it's possible to point to latest, as every commit in the flathub repo should point to a single, clearly defined release.

My ideal workflow would look like this:

• Be notified by GitPunch about a new release
• Run a script that fetches the archive for the release, calculates the checksum, updates python3-requirements.yaml and opens a PR in the flathub repo

Alternatively the last step could also be a post-release GitHub action that is run on this repo.

[Athanasius]

1. Yes, edcd.io is what EDMarketConnector is affiliated with.
2. Yes, please do support EDMC.py as well. Although I don't know if there's any common use of it besides with Trade Computer Extensions Mk.II, and that might be Windows-only.
3. In Windows terms they're in %LOCALAPPDATA%\Frontier Developments\Elite Dangerous\Options\Bindings. So they'd be in the Steamdeck equivalent of that.
4. OK, per-release is fine especially with things like python dependencies already being automatically generated.
5. If 'io.edcd.edmarketconnector' is to be used then I feel it would need to be a part of this repo. Given this is powered by PRs against the flathub repo ... does any part of that require the flatpak be in its own repo, or can it live as a sub-directory of this one ?

My aim here is to have this be as official as it can be (unless that application ID is changed), but it would then be on 'best efforts' basis, with no-one here actually testing if new releases still work.

[A-UNDERSCORE-D]

For 4, test on an actual Installation, proton puts localappdata in goofy places sometimes

[Athanasius]

Ah, the fun of us building something for Linux, but the use case being against steam-mediated 'Windows' paths. i.e. no way to tell flatpak "Just use %USERPROFILE% for this bit".

[der-eismann]

I don't think it's that goofy - in my case it is in ~/.steam/steam/steamapps/compatdata/359320/pfx/drive_c/users/steamuser/AppData/Local/Frontier Developments/Elite Dangerous/Options/Bindings. I mean it's long, but not unpredictable 😄

[der-eismann]

5. If 'io.edcd.edmarketconnector' is to be used then I feel it would need to be a part of this repo. Given this is powered by PRs against the flathub repo ... does any part of that require the flatpak be in its own repo, or can it live as a sub-directory of this one ?

Unfortunately it needs a separate repo in the flathub organization, since they manage the whole build & deployment infrastructure there. We could move some parts here, e.g. the *.desktop file and the appdata.xml, but the main definition would still need to remain in the flathub repo.

[Athanasius]

5. If 'io.edcd.edmarketconnector' is to be used then I feel it would need to be a part of this repo. Given this is powered by PRs against the flathub repo ... does any part of that require the flatpak be in its own repo, or can it live as a sub-directory of this one ?

Unfortunately it needs a separate repo in the flathub organization, since they manage the whole build & deployment infrastructure there. We could move some parts here, e.g. the *.desktop file and the appdata.xml, but the main definition would still need to remain in the flathub repo.

That's not what I meant.

I assume that the in-flathub PR is something to point at where the files currently hosted in https://github.com/der-eismann/edmc-flatpak/ are. I was asking if the latter has to be a self-contained repo with nothing else in it, or if it can just be a sub-directory within a repo used for other things.

[der-eismann]

But that's what I meant - each app that should be available on flathub gets its own repo in the GitHub.com/flathub orga. And that's the place where all the files that I currently have in my repo are hosted. And it needs to be there, self-contained. See this for example:
https://github.com/flathub/org.freefilesync.FreeFileSync

[Athanasius]

Yeah, I was a bit afraid of breaking changes and didn't have time/forgot to test it. Did you tag the new maintainers of this project?

Yup, that's who I tagged in.

[dvdmuckle]

Not to potentially add to the scope of this issue, but could we get a mention of the Flatpak in the wiki? It works and is good to go aside from an issue on the Flatpak side and an issue on this side (that doesn't affect performance).

[Rixxan]

OK, Looking at this, I do not appear to have permissions on the flatpak repository. I think it looks like @der-eismann is the main contact for that repo?

Looks like flathub/io.edcd.EDMarketConnector#32 was automatically opened to update to 5.10. I'm not sure what additional would be needed from myself or @C1701D in order to make this happen.

I'm not very familiar with the Flatpak build system. If there is something that needs done, please let me know and I'll do it today or tomorrow.

To answer @dvdmuckle I will make sure when we're doing the updates (see: #2115) we will include references to the FlatPak repo.

[dvdmuckle]

The Flatpak has been set up to be mostly automatic. There's some config that will update any dependencies, which includes any new releases of EDMC. It will then open up a PR like you linked. Said PR just needs to be approved and maaaaybe one other version number somewhere needs to be bumped, but besides that, the build process is all automatic. Happy to help with this portion of EDMC as well.

[Rixxan]

That's awesome that you guys have that! We'll be sure to shout it out in the "run on Linux" portion of the docs and release notes. I think that classifies the Flatpak distro as "unofficial but supported" and we'll defer to y'all on keeping that section going. Let us know if there's any support the core team here can provide.

I think that then closes out this issue? Unless there's anything unresolved we haven't sorted.

[Rixxan]

Hey @der-eismann / @dvdmuckle just wanted to loop back to this one. Anything the core team can do to allow 5.10 to be pushed to the flatpack users?

[dvdmuckle]

I unfortunately do not have write access to the Flatpak repo, I think that'll come down to @der-eismann.

[der-eismann]

Hey everyone, sorry for the late reply. I recently became a dad, so my E:D play time went down rapidly, at the same time I didn't want to merge an untested update, especially after the latest bigger changes. I'll give it a go tomorrow and merge if everything's fine.

If somebody volunteers we can request additional maintainership on Flathub, I just wanted to get the project going. I also saw on other projects that it is possible to host the Flatpak manifest inside the actual project repository, that would allow for a more streamlined update process - a flathub PR is opened automatically for a new release, however the python dependencies include some manual work atm.

[dvdmuckle]

@der-eismann first off, congrats! Second, I'd be happy to be a maintainer on the Flatpak. I just got a Steam Deck so I'll also be able to do additional testing on there.

[der-eismann]

I created this issue (flathub/flathub#4823), let's see how long it takes.
Regarding the update I opened #2125, if it's too hard to fix I'll see how hard it is to add Git to the Flatpak.

[dvdmuckle]

Thanks! Invite accepted and I have write access to the repo! @Rixxan want me to bump the version to 5.10?

[Rixxan]

Thanks! Invite accepted and I have write access to the repo! @Rixxan want me to bump the version to 5.10?

Hold to 5.10.1 please. Bug exists that would break on installs from source without Git installed. @dvdmuckle

[Rixxan]

(breaking issue: #2125)

[Rixxan]

Breaking issue now cleared. Looks like flathub/io.edcd.EDMarketConnector#34 is ready to go!

[dvdmuckle]

@der-eismann I believe we first need to revert the git patch before merging in the 5.10.1 release for the Flatpak, right?

[dvdmuckle]

Just a heads up, the Flatpak has been updated to the latest version of EDMC.

[Rixxan]

Closing this as it seems to be sorted pretty well :)

If there's any assistance y'all need, always feel free to open an issue or get in touch.