Using Raspberry Pi? You could be bumping into the same problem as me:
The thing is that each playlist takes 1-5 seconds to load on my raspberry (can be seen in debug output). I don’t know what this means (is it loading the full playlists?) or if pyspotify/libpotify has to do this, but this has made mopidy kind of unusable to me the last months. I don’t have a solution unfortunately.