Browse artists/albums only shows a small number of artists

General
1

I’ve got 1700 local tracks loaded in mopidy, and when I go to ‘browse’, ‘artists’ in ncmpcpp or Android MPDroid, it only shows the first 10 or so artists. Same under ‘album’, just albums from the first 10 or so artists, and ‘genre’, the genres included in those artists. Like it gave up after scanning a few albums.

I’ve done ‘local scan’ and it shows all the music correctly under files/local media/artists, files/local media/albums etc, with artists and albums separated out nicely by the sqlite, it is just in the browse views where it doesn’t work.

Using the sqlite backend if that makes any difference. I’ve updated to the latest version just now, no fix. I’ll try json and see if that helps, but I’d rather use the sqlite because it’s better, so if there’s a fix for it I’d love to know.

Joe

2

Hi Joe,

you mention ncmpcpp and MPDroid - have you also checked with the MusicBox Web client?
And can you post /var/log/mopidy/mopidy.log to check for any errors during scanning or browsing?

3

Definitely something to do with sqlite, as json works fine.

I’ve removed the sqlite db completely and rescanned and it is still broken.

MusicBox web client doesn’t support direct browsing by artist or album, only going in via the ‘file system’ (and with sqlite that gives you artist / album folders that don’t really exist on the filesystem, which is nice, but not as convenient as them being in the normal ‘artists/albums’ bit in other software).

It is a pain because it means you can’t go straight to a list of artists or albums in mpdroid, so I’m going to have to move back to json till it is fixed

It appears to have albums for 108 tracks, I think that it only does the albums for the first batch of 100 files, and I have the batch size for scanning set to 100, ie. it is a bug in the batching of scans somehow.

No errors from the scan except a few timeouts and a stray .ini file that it sulks about.

2014-12-16 10:54:10,899 INFO [3695:MainThread] mopidy.__main__: Disabled extensions: none
2014-12-16 10:54:31,595 INFO [3695:MainThread] mopidy.local.commands: Found 1746 files in media_dir.
2014-12-16 10:54:31,619 INFO [3695:MainThread] mopidy.local.commands: Checking 0 tracks from library.
2014-12-16 10:54:31,632 INFO [3695:MainThread] mopidy.local.commands: Removing 0 missing tracks.
2014-12-16 10:54:33,350 INFO [3695:MainThread] mopidy.local.commands: Found 1746 tracks which need to be updated.
2014-12-16 10:54:33,356 INFO [3695:MainThread] mopidy.local.commands: Scanning...
2014-12-16 10:55:25,162 INFO [3695:MainThread] mopidy.local.commands: Scanned 100 of 1746 files in 51s, ~851s left.
2014-12-16 10:56:18,123 INFO [3695:MainThread] mopidy.local.commands: Scanned 200 of 1746 files in 104s, ~809s left.
2014-12-16 10:57:12,173 INFO [3695:MainThread] mopidy.local.commands: Scanned 300 of 1746 files in 158s, ~765s left.
2014-12-16 10:58:03,273 INFO [3695:MainThread] mopidy.local.commands: Scanned 400 of 1746 files in 209s, ~706s left.
2014-12-16 10:58:36,375 WARNING [3695:MainThread] mopidy.local.commands: Failed local:track:desktop.ini: Your GStreamer installation is missing a plug-in.
2014-12-16 10:59:06,043 INFO [3695:MainThread] mopidy.local.commands: Scanned 500 of 1746 files in 272s, ~679s left.
2014-12-16 11:00:04,652 INFO [3695:MainThread] mopidy.local.commands: Scanned 600 of 1746 files in 331s, ~632s left.
2014-12-16 11:01:03,218 INFO [3695:MainThread] mopidy.local.commands: Scanned 700 of 1746 files in 389s, ~582s left.
2014-12-16 11:01:16,210 INFO [3672:MpdSession-10] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48940
2014-12-16 11:01:16,408 INFO [3672:MpdSession-11] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48941
2014-12-16 11:01:35,165 INFO [3672:MpdSession-12] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48942
2014-12-16 11:01:54,229 INFO [3672:MpdSession-13] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48943
2014-12-16 11:02:03,857 INFO [3695:MainThread] mopidy.local.commands: Scanned 800 of 1746 files in 450s, ~532s left.
2014-12-16 11:02:07,417 INFO [3672:MpdSession-14] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48944
2014-12-16 11:02:11,601 INFO [3672:MpdSession-15] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48945
2014-12-16 11:02:12,269 INFO [3672:MpdSession-16] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48946
2014-12-16 11:02:13,099 INFO [3672:MpdSession-17] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48947
2014-12-16 11:02:18,815 INFO [3672:MpdSession-18] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48948
2014-12-16 11:02:19,534 INFO [3672:MpdSession-19] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48949
2014-12-16 11:02:20,224 INFO [3672:MpdSession-20] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48950
2014-12-16 11:02:20,825 INFO [3672:MpdSession-21] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48951
2014-12-16 11:02:31,086 INFO [3672:MpdSession-22] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48952
2014-12-16 11:03:06,294 INFO [3695:MainThread] mopidy.local.commands: Scanned 900 of 1746 files in 512s, ~482s left.
2014-12-16 11:03:22,060 INFO [3672:MpdSession-23] mopidy.mpd.session: New MPD connection from [::ffff:192.168.1.80]:48953
2014-12-16 11:03:32,597 INFO [3672:MpdSession-24] mopidy.mpd.session: New MPD connection from [::1]:49044
2014-12-16 11:04:06,702 WARNING [3695:MainThread] mopidy.local.commands: Failed local:track:Moby/Animal%20Rights/07-Soft.mp3: Timeout after 1000ms
2014-12-16 11:04:08,826 INFO [3695:MainThread] mopidy.local.commands: Scanned 1000 of 1746 files in 575s, ~429s left.
2014-12-16 11:05:07,760 INFO [3695:MainThread] mopidy.local.commands: Scanned 1100 of 1746 files in 634s, ~372s left.
2014-12-16 11:06:06,518 INFO [3695:MainThread] mopidy.local.commands: Scanned 1200 of 1746 files in 693s, ~315s left.
2014-12-16 11:07:17,197 INFO [3695:MainThread] mopidy.local.commands: Scanned 1300 of 1746 files in 763s, ~262s left.
2014-12-16 11:08:14,051 WARNING [3695:MainThread] mopidy.local.commands: Failed local:track:The%20Crocketts/We%20May%20Be%20Skinny%20%26%20Wirey/09-Six%20So$
2014-12-16 11:08:22,318 INFO [3695:MainThread] mopidy.local.commands: Scanned 1400 of 1746 files in 828s, ~204s left.
2014-12-16 11:09:18,605 INFO [3695:MainThread] mopidy.local.commands: Scanned 1500 of 1746 files in 885s, ~145s left.
2014-12-16 11:10:22,877 INFO [3695:MainThread] mopidy.local.commands: Scanned 1600 of 1746 files in 949s, ~86s left.
2014-12-16 11:11:24,213 INFO [3695:MainThread] mopidy.local.commands: Scanned 1700 of 1746 files in 1010s, ~27s left.
4

Going deeper into this, I’ve turned debug logging on, and if you send ‘list artist’ to the mpd server, as you might do if you want to list the artists, it gets into mopidy_sqlite as a search for all tracks.and uses the default limit for sqlite search of 100 replies.

DEBUG    2014-12-16 12:53:41,861 [3964:MpdSession-9] mopidy.mpd.session Request from [::1]:53540: list Artist
WARNING  2014-12-16 12:53:42,074 [3964:LocalBackend-3] mopidy_local_sqlite.libr$
  here, query: {}
DEBUG    2014-12-16 12:53:42,088 [3964:LocalBackend-3] mopidy_local_sqlite.sche$
  SQLite search query []: SELECT * FROM tracks WHERE 1
5

Okay, this is a bug in how mopidy responds to ‘list album’ commands (and all the other ‘list [blah]’ commands in the mpd protocol, in

The problem is that it has no real concept of albums in the underlying library code. So, rather than search for a list of albums, in mopidy/local/music_db.py/_list_album, it calls a completely blank query on the database, for all possible tracks, then adds each album to a big set. It is gross, and presumably if you get a decent sized library, makes it very inefficient.

There isn’t a simple fix to this, because it is a fundamental missing link in the library API, that there is no way to query for albums. I can up the number of results returned by the sqlite plugin (or make it return everything like json does), but that doesn’t fix the underlying problem.