When Mopidy itself is installed via apt as system Python package, it requires any extensions to be installed on system level as well. Recent pip however requires the --break-system-packages flag for this to work. Moreover, most recent pip tries (and fails) to uninstall any module installed via apt to /usr/lib/python3, before installing same module to /usr/local/lib/python3.*. This can even cause failures with Python dependencies pulled by the mopidy package via apt, and such pulled by pip from PyPI when installing extension.
pip outside of venv/virtualenv/… is basically dead on any recent distro which offers Python packages with its own package manager.
Hence I do wonder whether there is any plan or idea to allow installing extension in a different way. Mopidy, as system package, does not see Python modules in any random venv. And calling Mopidy with the venv Python instance will miss its APT dependencies. Unless … with --system-site-packages it would have access to them. So maybe this would work:
But this does not look like something to cleanly implement into distro package.
Other projects like OctoPrint have internal plugin management, using an appropriate site to install plugins via pip, based on the Python environment it was called with. But that is very complex, and there were quite some issues with the site detection logic and with recent pip changes etc. Hence not something I would exactly suggest for Mopidy .
Easiest would be if there was somehow am optional way to tell Mopidy where to (additionally) look for extensions. Does Python have a native way to add additional import paths?
We’re aware of the issue with --break-system-packages, but we have sadly never decided on a new recommended way for how to work with a mix of extensions installed from distro and PyPI packages. It could be useful to start enumerating options, as I don’t know of any existing golden paths here.
That said, after maintaining Mopidy for 15 years, we have a lot less time on our hands to dedicate to the project. I prefer to focus my efforts on the Python codebase and leave packaging to others. Thus, I’d clearly prefer solutions where the Mopidy project just follows established conventions that makes life easier for distro packagers. Diving deep into building our own extension management systems to make mixing distro- and PyPI-installed extensions is far outside the scope we’re willing to take on.
It is possible to control where Mopidy looks for extensions by setting the PYTHONPATH environment variable (or site.addsitedir()). By installing extensions with pip into some venv and then use PYTHONPATH to make system-wide Mopidy find those, you’d be able to install PyPI packages without affecting any other system-wide Python programs, which is an improvement over --break-system-packages. (Note: This could still break the Mopidy installation, as PyPI-installed extensions could depend on packages that are incompatible with what Mopidy or distro-installed extensions depends on. To solve this fully, you’d need dependency resolving across distro- and PyPI-installed extensions, which is a bit much to wish for.)
Right, I hope this topic can serve that purpose. If anyone knows a similarly structured Python project with extensions, it would be surely interesting to see how/whether this apt + system-level pip mix is solved there. But I have the impression that pip broke this mix systematically and intentionally during the last 2-3 years, not intending a clean solution, whether for good (forcefully avoiding possible environment conflicts) or bad.
Absolutely fair. And this really is an issue with the APT/distro packages only, as pip install mopidy mopidy Mopidy-Iris inside any type of virtual environment works just perfectly fine. Strictly seen a task to solve for distro maintainers and/or apt.mopidy.com maintainers. But maybe a small change in Mopidy core can make it easier to implement a cleaner solution for packagers, keeping it simple for end users without the feeling to act increasingly against Python/pip development directions, possibly facing implicit issues.
Agreed, there must be a native Python way for this.
That seems pretty straight forward. Just testing on Debian Trixie:
I used --system-site-packages so that Python modules already pulled in as dependencies via APT are not again installed into the venv. Since Mopidy loads extensions just as Python modules like its core dependencies, extensions must be compatible with Mopidy dependencies vice versa. This is and was the same before. I don’t think there is a way to let Mopidy imports dependency modules from system site A only, but extensions from the venv as well, while the extensions again import their dependencies from the venv. Within the same interpreter/process, there is just one global instance of module possible. Hence there is no point to isolate the extensions venv. If for some reason a particular module is wanted in a different version, it can always be manually installed into the venv.
Passing PYTHONPATH via systemd unit was just the quick way. Packagers will probably want to put this into some /etc/default/mopidy environment file and load that from the systemd unit. This makes it easier for admins to add/change the path there.
But since this applies to the systemd unit only, mopidyctl will not see the extensions. E.g. I installed latest mopidy-local into the venv, since Debian Trixie’s version is incompatible with Python 3.13: https://bugs.debian.org/1084644Not an issue with apt.mopidy.com anymore.
And mopidyctl local scan does not work without passing PYTHONPATH. In case of the Iris web interface (which I was testing in turn), it calls sudo mopidyctl local scan, and sudo forcefully clears the environment. It also does sudo python3 -m pip install --upgrade mopidy-iris to upgrade itself, which would install a duplicate at the system site, and fail unless break-system-packages=true is set in whichever effective pip.conf.
I think the following could be enhanced to make it easier and address the above limitations, not saying anything is needed, just things that come to my mind testing the Iris web UI:
Add a new mopidy.conf setting extension_site or something like that, which Mopidy internally applies via site.addsitedir(). That way mopidyctl and mopidy (with matching --config) calls would see extensions without the need to pass PYTHONPATH.
a) Allow mopidyctl calls as mopidy user, resp user which runs mopidy.service. Not sure how mopidyctl derives the config path(s) and executing user correctly, and I hope doing this does not require root permissions for another reason. But in the end it is just a wrapper for mopidy, calling it with correct arguments as correct user. Omitting the user switch, if executed by the same, should be possible. That makes it easier for extensions to call mopidyctl without needing sudo (like Iris to trigger local library scans).
b) Alternatively they could call mopidy directly, but then they would need to know/derive the config path(s) somehow. Not sure whether there is some internal API they can use already? Ah, they can just check sys.argv, isn’t it? Maybe something to enhance at Iris, making the sudo mopidyctl and $IRIS_CONFIG_LOCATION (for containers) obsolete.
c) This actually seems to be needed if extensions use features of other extensions iris -> local, otherwise they can use the API, of course. Isn’t mopidy-local exposing an API method to trigger a local library scan? I see there is an optional refresh method that library providers can provide: mopidy.backend — Backend API — Mopidy 3.4.2 documentation
That would be a lot cleaner, of course. Triggering a library rescan from frontends/web clients is a pretty expected feature.
Letting extensions update themselves or install new extensions is probably not such an intended feature, but can be convenient of course. I wouldn’t have thought about it at all if I hadn’t seen their setup instructions and the related script: Iris/mopidy_iris/system.sh at master · jaedb/Iris · GitHub
This is currently broken in any case without further configuring pip on the host. If it shall work using a venv, extensions would need to know the path to the pip instance which installs into that venv (or other environment), like /opt/mopidy_venv/bin/pip in my case. This is probably more something to do in Iris, like an extension setting. I don’t know another extension which tries this.
Maybe similarly not something common across extensions is to restart Mopidy. It feels entirely wrong that Iris makes use of sudo, while Mopidy runs as unprivileged user, even that it needs it for its script only. Maybe there is already an API method to trigger a reload/restart of the Mopidy process internally? Or a SIGHUP handler, which extensions can make use of?
.