Sluggish library browsing with huge library

Engine DJ Hardware Media Players
,
1

Hi,

I’m writing this post to see if I can get any information on how to optimise the performance of my library (if possible at all) and to see if what I am experiencing is exceptional or is common behaviour. Also, I thought it could be useful simply to report to Denon my library usage needs, as most companies often welcome customer usage patterns when designing their products…

Before anything else, these are Denon’s official words, found in the SC5000s page FAQ:

We recommend limiting the total track count to under 10,000 songs on a media source. Larger track counts may affect load/search/sort times. We recommend that you only add songs to a drive that you need for the show/night/tour.

My library has about 50k songs, so I’m very well aware that this is 5 times bigger than Denon’s official recommendation. I’m not raising a bug or complaining here, just reporting my needs and the behaviour I observe, as maybe getting some tips from another users.

My scenario is:

  • two SC5000s
  • library of about 50k tracks, FLAC and MP3 files.
  • tracks are all properly tagged by Track Artist, Album artist, Track Title, Album Title, Genre, Comment, Year, and Record label (Publisher). Some of these tags are not being used my the decks, as we all know.
  • media is 1 TB SSD drive - Samsung 850 EVO 1 TB. According to some reviews and benchmarks available in the Internet this is a high quality drive, offering one of the fastest read speeds in the market.
  • drive is formatted in exFat
  • drive is connected with a USB 3.1 cable to the rear usb plug of one SC5000s. I think this is the fastest connection that can be achieved in the SC5000s.
  • the two SC5000s are connected using another USB 3.1 cable between them.
  • library prepared and analised on Engine. I have some crates, but most of the tracks don’t belong to any crate.

Unfortunately, browsing the library is very sluggish on the SC5000s. The behaviour in terms of performance is:

  • the library is functional (most of the time). I can browse it and load their tracks. It works. It’s just that it’s sluggish.
  • operations on the library such as searching or browsing tracks take about 8 seconds. Ie: when typing a character into the search box, switching between albums, navigating back and forth or changing the sort criteria will show the “Updating” message for about 7-8 seconds before showing results.
  • I don’t see a significant difference in behaviour when browsing the library from the deck that has the drive directly connected to it or from the other deck.
  • Subsequent execution of the same browsing operation is immediate. Ie: I navigate into an album, it takes 7 seconds. I navigate back and forth again, it takes 0 seconds. Obviously the result of searching and navigating operations are cached. I don’t know for how long or how many operations the units can retain on memory.
  • It looks like about 20% of the time it takes to perform one operation is access to the drive, and 80% processing. Ie: I navigate to one album. The read LED of the drive flashes for less than 2 seconds. It will then stop flashing but it will take about 5 seconds to show results.
  • I’ve tried the same library with one 3TB HD. I have not experienced a significant improvement in performance.

Apart from these performance issues, I see from time to time some functional issues:

  • sometimes the “Update” message just won’t disappear. The unit simply won’t show results or the tracks in an album. Only fix is rebooting.
  • I’ve seen the secondary deck (deck connected to the deck with the drive) rebooting without user interaction a couple of times.
  • I’ve seen a message saying “the library is corrupt” while using it a couple of times. Rebooted, message not shown again, unit could use the library again.

I’m sure these behaviour are 100% down to library size. When limiting the library to 1k songs everything works immediately fast. When limiting it to 10k I can observe some delay, but it’s more or less acceptable.

From all these observations it’s clear to me that cause of the behaviour is library size. It’s not media (drive, SSD vs HD) or USB connections. I think it’s more about how the library indexes tracks and in the database, and maybe navigation/searching algorithms.

Again, I insist that when keeping the library size under Denon’s recommendations everything works, and that I’m abusing the recommended size by 5 times. However, I’d like to express that a library this big is a real requirement to me and to many mobile DJs. These usage patterns are met by existing software such as Traktor, Serato or Rekordbox with ethernet Link. Unfortunately this is preventing me to completely drop the laptop, if that is one of the main premises of the system (although people reading Denon specs should be aware of the limitation, fair enough). Don’t get me wrong - the system works and it’s great, but if you really need a library this big and have that problem solved with Software but want to drop the laptop, unfortunately I think the Prime system is not offering it yet.

2

I think one way is not to have the entire 50k analysed. I have 63K tracks on a regular external HD that i plug in. Funny enough i have the entire thing analysed as the decks are for home use.

I read that somewhere.

Analyse your frequently played Crates only, then analyse on the fly if you need to play a song that you haven’t previously done. Such as that odd request by the brides mother.

I notice things move quicker if i allow both decks “update” to be completed after powering on.

3

When you mean not analysed, do you mean in library but not analysed (but searchable) or just not in library?

If they are not in library, sure, if would speed up things dramatically, but tracks but be really difficult to find without search and you would need to navigate through the folders structure.

Just to clarify, performance is degraded when the tracks are added to the library. If they are just in the drive but not in the library they don’t add up to loading time…

I have 63K tracks on a regular external HD that i plug in. Funny enough i have the entire thing analysed as the decks are for home use.

Could I ask you if you are also experiencing about 7-8 seconds upon search or navigation operations? I’ve seen another users in the forum reporting they are fine with their loading times with huge libraries.

4

Not analysed = Switch off Engine Prime Autoanalysis. Just analyse your “go to crates” only.

You can still search and play songs that were not analysed. Once you load the track on the deck it will analyse on the player.

Yes i do get a delay when moving out of a crate.

I use crates and subcrates a lot…so most time i just stay on the parent crate and i type a lot

I dive into a subcrate say im looking for track i cant remember the name.

1 Like
5

Mufasa said it. It was a workaround when Engine Prime database had to be under a certain limit to be shared between drives.

Also, this is another reason for Denon to finally add option to link SC5000 players with Engine Prime software with LAN connection that was present even in old Engine 1.5 for SC2900/3900 players

2 Likes
6

I would really see that happening. Some people would say that defeats the main purpose of the system: being just standalone and laptop free. But I don’t think that’s true. Even if one still needs some support from the laptop, the screens and controls layout of the SC5000 are great, and I’d rather spent 99% of the time just with looking at the screens the of SC5000 with the laptop connected and away that be forced to limit the size of my library.

That said, the SC5000s are just computers so I reckon they could ultimately perform as fast as a laptop. (This is simply a software development problem: the database may not be well indexed for text search and navigating through related items).

7

If you are using a large collection connected to the players via USB, ensure you are using a decent SSD with a decent USB 3.0 connection.

Currently I am using a 500GB Intel 545S with a Startek USB 3.0 to Sata cable. I never see any lag while browsing or loading tunes.

The entire collection was exported to the drive using Engine Prime.

8

Umm…no. They still remain standalone. You can put the music on laptop or connected into the players on usb/sd/hdd

9

Wasn’t aware this is still the case

Guess that’s the idea, but why would you want to take anything but your entire collection with you and have extra hassle with that.

I also like to use crates and subcrates a lot. Unfortunately crate management is atrociously slow with Engine Prime on any machine i tried.

10

This will be ideal if one was just hired to do a DJ Set.

How do one pack for a wedding or private function or a bar with demographic of 16 to 99 year olds, different taste in music.

I just take my entire collection (serato + controller of choice)

Working DJ woes

:man_technologist:t5::headphones::microphone::level_slider::notes:

1 Like
11

THis has been something I have noticed in the Engine Prime software. Creating and modifying/sorting crates takes an exorbitant amount of time.

3 Likes
12

I suspect it’s the library size, not the number of tracks analyzed.

1 Like
13

I suggested something related before that would be ideal:

  • Automatic background search database build (when plugging in unanalyzed card/drives tags will be read in the background, quickly make the search function available to all tracks. Further analyzation could be done once the track is loaded)
14

I wonder is this is something Denon is aware of and sees as an issue, or is it just me, @paul_denondj ?

With the Prime4 having a bay for a drive I guess users will expect to truly use the space of their 1TB drives with proper navigation, searching and loading times.

Thanks

15

The library size you mention is well beyond our suggested library size recommendations. We set these to help avoid instances like this. Please see below:

16

Someone has suggestions on how to improve the library in another thread

Bye bye laptop? Am I ready?

17

Just to be clear, that was done using the PC version of the database - I’ll check the USB version of the database probably later today but I don’t expect to see much difference.

Also @paul_denondj I really don’t think that’s a good enough answer - clearly there are optimisations the developers can do on the database side. For example, a project I was working on recently was running a version of Windows CE and had over 50,000 users on the SQLite database - I don’t know the exact hardware but it was nowhere near as powerful as the SC5000 hardware. Doing a search for username (which searched first name, last name, email address) took 2.81 seconds on average. The main reason for this was good indexing and views.

You can also structure queries to only bring back the information in each table that you actually use, rather than all of the fields - this especially helps if the fields contain lots of long strings or other long datatypes (images, blobs etc).

The SC5000’s taking 7-8 seconds to search through 50,000 records with the hardware available is therefore pretty unacceptable in my opinion.

Additionally, you guys are recommending a 1TB SSD for the Prime 4 - do you really think people are going to limit their library to 10,000 tracks on a 1TB SSD? That’s just not realistic.

2 Likes
18

Fair comment and good info, thanks Jonny

I’ll ask our software Dev team expert to chime in with more insight…

Best

Paul

1 Like
20

Even with extended mix wav files, you’ll easily get over 50000 tracks on 1TB

21

Well, there are conscious people and there are stubborn people. Whether they put a limit to themselves or not it is their choice. What matters is the fact that every piece of equipment has different specifications and limitations and they should be used accordingly. What we want is not always what we can obtain from a given equipment.

2 Likes