ANNOUNCEMENT: New Extension Mopidy-BeetsLocal

Show and tell
1

I would like to announce the release of a new extension.
The existing mopidy-beets extension presented some issues to me and so i decided to try building my first mopidy extension.
In the hope it might be useful to others i released it here:

Elevator Pitch:
Mopidy-BeetsLocal uses beets internal core api to present a local music collection to mopidy clients.
It is different from mopidy-beets as it does not require a beets web process running in the background. This greatly reduces error-prone complexity. Local tracks are not streamed but accessed as local files. Distributed setups are still possible via network file shares.

Using beets internal core api allows for more advanced queries. For example it is now possible to query beets for genre etc.
In theory all of beets fields can be queried, although when querying together with other backends queries need to be limited to the fields mopidy knows about.

It also returns a last_modified track parameter. This allows clients that maintain their own collection repository like rompr to update the collection in a more effective way.

Thanks to fatg3rman there is an adapted version of rompr in trunk now (v463) , that takes advantage of the features described.
Please donā€™t expect stability yet.
I hope i jumped through all the hoops required for a release correctly.
The cookiecutter template and the detailed documentation of mopidy sure were a great help.
Feedback is appreciated.

2

Hey there @rawdlite - Iā€™m trying to get your extension going but seems to have a configuration error issue:

2015-01-03 20:04:33,448 - WARNING  Found beetslocal configuration errors, the extension has been automatically disabled:
2015-01-03 20:04:33,450 - WARNING    beetslocal/static_dir unknown config key.
2015-01-03 20:04:33,453 - WARNING  Please fix the extension configuration errors or disable the extensions to silence these messages.

I am running the latest pimusicbox distribution and alongside rompr - but so far I cannot connect my beets library.

Any ideas?
Thanks!

Here is my config:

[beetslocal]
enabled = true
beetslibrary = /home/pi/data/musiclibrary.blb
use_original_release_date = false

PS - Once this is all working - I whould be able to update my library in mopidy through a refresh in rompr as long as the import is done in beets right?

3

Are you sure you donā€™t have a line starting with static_dir somewhere in your config file after [beetslocal]?

If thereā€™s no other config section header (e.g. [http] without a # comment marker in front) inbetween, then your static_dir config will be considered part of the [beetslocal] config section.

1 Like
4

Yup! That was exactly what was happening. Learning my way around now, thanks for that quick one.

Im still yet to see how the Mopidy-BeetsLocal will help make refreshing Mopidy playlists possible without a total restart and local scan. I was sort of hoping that once Mopidy was using the beets library, that any updates to beets database would be reflected inside Mopidy - but I think thats not possible at the moment?

5

Glad your issue got fixed so quickly and beetslocal is working for you.
Have you disabled [local] and [beets] in your config?

I am not quite sure what you mean by refreshing Mopidy playlists.
All queries to beetslocal go directly to the beets BE, there is no caching involved.
So any change to you beets repository is reflected instantly.

OTOH the Library browsing in rompr is done through its own repository.
When you feel up to it you can try the latest rompr in subversion trunk. It supports updating the rompr library via a http request.

curl http://ip.address.of.webserver/rompr/albums.php?rebuild=yes > /dev/null

Beets supports customizable hooks to run after every update.

HTH

6

Hey there - ok, I hadnt realised I needed to disable the [local] module (done). And I havent installed the [beets] so thats not an issue. I have restarted and now I dont have access to beets library.

when I click on browse (using the default pimusicbox webclient) all I see is Spotify and Spotify browseā€¦ should I be seeing something there?

Here is the mopidy output during startup - does it look like [beets-local] is behaving normally?

2015-01-04 11:45:30,152 - INFO     Starting Mopidy 0.19.4
2015-01-04 11:45:32,045 - INFO     Loading config from: builtin defaults, /etc/mopidy/extensions.d/spotify.conf, /etc/mopidy/mopidy.conf, command line options
2015-01-04 11:45:33,658 - INFO     Enabled extensions: spotify, mpd, http, stream, spotify_tunigo, podcast-gpodder, local-sqlite, youtube, podcast-itunes, http-kuechenradio, softwaremixer, local-whoosh, simple-webclient, moped, musicbox_webclient, websettings, beetslocal
2015-01-04 11:45:33,661 - INFO     Disabled extensions: alsamixer, gmusic, scrobbler, subsonic, somafm, internetarchive, audioaddict, dirble, podcast, local, tunein, soundcloud
2015-01-04 11:45:33,664 - WARNING  Found alsamixer configuration errors, the extension has been automatically disabled:
2015-01-04 11:45:33,667 - WARNING    alsamixer/control must be set.
2015-01-04 11:45:33,670 - WARNING    alsamixer/card must be set.
2015-01-04 11:45:33,672 - WARNING  Please fix the extension configuration errors or disable the extensions to silence these messages.
2015-01-04 11:45:36,765 - INFO     Starting Mopidy mixer: SoftwareMixer
2015-01-04 11:45:36,772 - INFO     Mixing using GStreamer software mixing
2015-01-04 11:45:36,785 - INFO     Mixer volume set to 85
2015-01-04 11:45:36,788 - INFO     Starting Mopidy audio
2015-01-04 11:45:36,812 - INFO     Starting Mopidy backends: StreamBackend, BeetsLocalBackend, SpotifyTunigoBackend, YoutubeBackend, SpotifyBackend
2015-01-04 11:45:38,275 - INFO     Audio output set to "alsasink"
2015-01-04 11:45:39,277 - INFO     Mopidy uses SPOTIFY(R) CORE
2015-01-04 11:45:39,314 - INFO     Starting Mopidy core
2015-01-04 11:45:39,627 - INFO     Starting Mopidy frontends: MpdFrontend, HttpFrontend
2015-01-04 11:45:39,641 - INFO     MPD server running at [::]:6600
2015-01-04 11:45:39,663 - INFO     HTTP server running at [::]:6680
2015-01-04 11:45:43,170 - INFO     Connected to Spotify
2015-01-04 11:46:01,066 - INFO     Loaded 22 Spotify playlists
2015-01-04 11:46:08,490 - INFO     New MPD connection from [::ffff:127.0.0.1]:36800
7

Hi surfacescan,

When you say you dont have access to the beets library, does that mean your searches don#t return tracks from your local library?
If that is the case you should check

  1. your beets installation.
    Login (or su to) the user your mopidy process is running under. Than try a ā€˜beet list ā€™ and see if that returns the right results.

  2. You can initiate some debug output by starting mopidy with the verbose option ā€˜-vā€™ or put

    [loglevels]
    mopidy_beetslocal = debug

in the conf file. That should tell us some more.

That looks absolutely normal. But there is no query send to the backend.

Unfortunately i dont know much about the pimusic webclient. I will look into that.
Meanwhile can you test with rompr?
This is what i am using and it works for me.

HTH

8

Doesnā€™t look like mopidy-beets-local/mopidy-beetslocal implements the browse interface needed for this functionality.

9

Just installed musicbox_webclient and tried a search and it works for me.

Search query: {u'any': [u'T. Rex']} in uris: None
INFO     2015-01-04 22:57:35,490 [27167:BeetsLocalBackend-4] mopidy_beetslocal.library
:T. Rex
DEBUG    2015-01-04 22:57:35,490 [27167:BeetsLocalBackend-4] mopidy_beetslocal.library
Build Query "':T. Rex'":
DEBUG    2015-01-04 22:57:48,858 [27167:BeetsLocalBackend-4] mopidy_beetslocal.library
  Query found 450 tracks
DEBUG    2015-01-04 22:57:49,634 [27167:BeetsLocalBackend-4] mopidy_beetslocal.library
  Query found 35 albums

The result list though is under soundcloud entries only separated by a small grey bar.

Ah yes. I thought as much. This is something for the next release then.

mopidy-beets-local/mopidy-beetslocal

Yes, there i was not quite consistent when setting this up and now i have to pay the price.
Wonder which will be easier to fix.
And thanks for looking at my code.

10

I did a similar thing with TuneIn! And no worries.

Is is feasible / does it make sense to merge your extension with the other pre-existing beets extension? Having both of them is a bit confusing for someone who knows very little about beets - but arguably it therefore doesnā€™t matter as itā€™s not aimed at me.

11

Just uploaded a new version that fixed the Link Issue in the Readme.

There are already 3 separate extensions (local, local-sqlite and beets web) that provide access to local files and are to be used exclusively. When thinking about merging with an existing solution i could more easily imagine a merge with local-sqlite if @tkem is interested.
Beetslocal is pretty much about the same concept except it uses an already existing sqlite file.

OTOH i consider teaching new users about beets to be a good thing.
When it comes to scanning and organizing a music collection nothing beats beets.
Pun intended :slight_smile:

12

Hi @rawdlite,

when starting the sqlite back end, I first thought about using the beets DB schema or API myself. However, one of the primary goals of sqlite was and still is performance, especially when browsing or searching moderately large (about 10,000 tracks) media collections on a platform like the Raspberry Pi. Thatā€™s why sqliteā€™s DB schema closely resembles Mopidyā€™s data model, but doesnā€™t support multiple artists per track/album. Now, as far as I remember, the beets DB schema favors flexibility over raw speed (though still did not support multiple artists when I looked into it), and early attempts showed a substantial decrease in performance compared to a ā€œnativeā€ schema, especially with browsing. So unless this has changed in the mean time, Iā€™d prefer to keep these extensions separate. I also think thereā€™s room for more than one local library back end, since people tend to have strong, but different opinions on how such a thing should work :wink:

13

BTW, since BeetsLocal does not implement the Local Library interface, I donā€™t think itā€™s strictly necessary to disable the local back end, or is it?

14

Hi @tkem,

You are right on both accounts.

Strictly speaking it is not necessary, but you might end up with confusing double entries in the client.
Of course you could use beets to scan one part of your collection and local scan another part (like podcast or other tracks that are not autotaggable)

I am running on a hummingboard (ARM plattform similar to the Rasperry Pi) with about 35.000 tracks. For browsing i use direct selects on the tables (working on it right now) which is quite fast but not the best solution for maintainability.
When starting to code my main issue was the sheer amount of decentralized repositories on my system that had to be kept in sync. (beets, mopidy, rompr, mpd ā€¦)

That being said, i found myself in another extension project i started (a headless client) to be using my own sqlite repo for playcounts, tags, tracklist etc. Also translating mopidy queries to beets queries is not as trivial as it seems at first.
So yes, there are good reasons for a mopidy specific repository.

This is the pleasure of open source projects.
My opinion is use as much existing code as possible.
When using decentralized repositories make syncing painlessly.

Adding a beets scanner to mopidy-sqlite and a Requesthandler to trigger that scanner would pretty much fulfill my requirements.

15

Could you please elaborate on that, I donā€™t think I quite get what you meanā€¦

16

Well, let me try.

In the config a new optional entry beetsdb
[local-sqlite]
beetsdb = /var/lib/mopidy/beetslibrary.blb

When this is set use the beetsApi to collect all albums and tracks to fill the repo.
Including item_attributes and album_attributes.
Use beets mtime for updates. So only recently changed entries are updated.

Then have a http request handler for /scan_beets.
This can be called as

curl http://<server>:6680/scan_beets

from inside beets after any update, via a cron job or by the client.

17

Weā€™ll, I donā€™t see something like this being included in Mopidy-Local-SQLite any time soon. Adding a dependency on beets just for this is pretty much a no-go. Plus, I guess thereā€™s many other types of media libraries that people might want to sync this way, and supporting each of them within Mopidy-Local-SQLite is simply beyond scope.

However, if all you want to do is periodically sync an existing media library such as a beets DB, why not implement an extension that provides a command for that? I.e. similar to local scan, there could be a beets sync command. You could use the local backend interface, i.e. mopidy.local.Library.add() etc., to insert tracks pulled from beets, which would have the advantage that this should work with any local library provider - json, sqlite, whoosh, and whatever comes up nextā€¦

18

@surfacescan

I just uploaded a new version (0.0.6), that supports browsing in the musicbox_webclient.
Maybe you like to check it out.

@tkem

ACK.

That is indeed a promising approach. I will put it on the backburner though, for at the moment everything is working nicely for me and no syncing is required at all.
Also i like to tailor beetslocal to the requirements of the headless client i am building.
Having full control over the Backend is a huge benefit to the development process.
When things have more stabilized a consolidation along the lines you suggested might be an option though.

19

Thanks @rawdlite Iā€™ll take a shot at it this weekend and let you know how it works out! Thanks :smile:

20

Hi @rawdlite - so I turned on the debug and took a look at the log file and here is the output, including a connection refused error for port 8888. Is that the port that your Beetslocal plugin works on? Seems my musicbox config isnt allowing a connection.

any thoughts?

2015-01-09 19:46:09,395 - INFO     Enabled extensions: spotify, mpd, http, beets, stream, spotify_tunigo, podcast-gpodder, local-sqlite, youtube, podcast-itunes, http-kuechenradio, softwaremixer, local-whoosh, simple-webclient, moped, musicbox_webclient, websettings, beetslocal
2015-01-09 19:46:09,399 - INFO     Disabled extensions: alsamixer, gmusic, scrobbler, subsonic, somafm, internetarchive, audioaddict, dirble, podcast, local, tunein, soundcloud
2015-01-09 19:46:09,402 - WARNING  Found alsamixer configuration errors, the extension has been automatically disabled:
2015-01-09 19:46:09,405 - WARNING    alsamixer/control must be set.
2015-01-09 19:46:09,408 - WARNING    alsamixer/card must be set.
2015-01-09 19:46:09,413 - WARNING  Please fix the extension configuration errors or disable the extensions to silence these messages.
2015-01-09 19:46:12,039 - INFO     Starting Mopidy mixer: SoftwareMixer
2015-01-09 19:46:12,051 - INFO     Mixing using GStreamer software mixing
2015-01-09 19:46:12,070 - INFO     Mixer volume set to 85
2015-01-09 19:46:12,078 - INFO     Starting Mopidy audio
2015-01-09 19:46:12,108 - INFO     Starting Mopidy backends: StreamBackend, BeetsLocalBackend, SpotifyTunigoBackend, YoutubeBackend, SpotifyBackend, BeetsBackend
2015-01-09 19:46:12,202 - INFO     Audio output set to "alsasink"
2015-01-09 19:46:14,096 - INFO     Mopidy uses SPOTIFY(R) CORE
2015-01-09 19:46:14,172 - INFO     Connecting to Beets remote library http://127.0.0.1:8888
2015-01-09 19:46:14,217 - INFO     Starting new HTTP connection (1): 127.0.0.1
2015-01-09 19:46:14,224 - ERROR    Beets error: ('Connection aborted.', error(111, 'Connection refused'))
2015-01-09 19:46:14,261 - INFO     Starting Mopidy core
2015-01-09 19:46:14,567 - INFO     Starting Mopidy frontends: MpdFrontend, HttpFrontend