Managing a massive collection with Engine Prime

Legacy Products Engine PRIME
, ,
1

I’m the kind of DJ that likes to have my whole music collection available to me when I DJ. I have a massive music collection, approx. 2TB on a 5TB portable drive, separated into folders based on top level genres, i.e. Techno, Hip-Hop, 80s, etc. In each of these top folders are hundreds of subfolders sorted by artist, and within those artist folders are all the artist albums and tracks (not too different from iTunes organization). I end up with something like 60,000 tracks across several thousand folders.

As a Traktor Pro DJ for many many years, my workflow was very simple:

  1. Plug external HD into laptop or NI controller.
  2. Browse through the folders to find the track I want to play.
  3. If the specific folder I’m in is not already analyzed, I right-click on it and select Analyze Folder to get BPMs and other metadata right there in that folder view, which helps me pick the track I want. (Example: I recently added a new Underworld album (folder) to my music drive. I’m DJing and want to play a song from it but I don’t know any of the BPMs or keys because it wasn’t analyzed yet. While still DJing some other song, I analyze that new Underworld folder on the fly, get the metadata, and then choose which track from the album I want to play next.)

This process allowed me to access my entire collection and analyze folders in real time while DJing. However it seems this process is not doable in Engine Prime and my P4 controller. While I can browse my external hard drive’s folder structure easily enough, no metadata is displayed in any folder - BPM, key, genre, etc., nor do I have the ability to analyze those folders from this file browser mode. Instead, the only solution I can see is to add all these thousands of folders to my Collection first.

The best workaround I’ve come up with in is to create Crates that exactly mimic the folder structure on my external drive. So under Collection, I’ve got Crates for Techno, Hip-Hop, 80s, etc. and I drag and drop folders from my drive to the corresponding Crates. This works… but presents a lot of problems:

  • Denon does not recommend having a massive Collection like this, especially on a spinning hard drive (and I’m not forking over the money for a 4TB SSD).
  • I have to remember to manually add new music from my drive to the Collection. With the constant updating on my drive, it’s near impossible to keep the drive and the Collection in sync. If there was an auto-sync feature where EP looked for changes to my external drive and automatically analyzed anything new it discovered, it would help a great deal in terms of taking out the guesswork. Of course, I’d still be dealing with a massive Collection which presents its own problems.

Bottom line, I don’t want to have to cull down my music collection, change the way I organize things, or have two sources of truth (the external drive and the EP Collection). The goal all along when I bought the P4 was to have everything in my existing collection available to me, just like it was in Traktor. It’s this whole “add to the Collection first” mechanism that is really throwing a wrench in the way I DJ. If I could just keep everything out of the Collection to start with, and then analyze folders as needed from the file browser view (thus adding them to the Collection), the problems would be solved.

Does anyone out there have a better workflow for DJing “live” from a massive music library?

4 Likes
2

Looking forward comments from the great community. Interesting to me also. My plan is to buy a new SSD for the Prime 4 and start migrating content.

By the way, I also come from Traktor. Maybe you are interested in supporting this

3

Although the processor inside the P4 is (was said to be) very powerful, it’s not an i7 quad-core MacBook Pro with 32gb ram, that can do all you ask of it & keep the music running smoothly. Selecting an entire folder to analyse whilst doing a gig is really asking for trouble In my opinion. You also can’t honestly expect the P4 to instantly read the tags of all the un-analysed songs in a folder on the fly whilst doing a gig. And to put the cheery on top, expecting this to work flawlessly using a 5tb mechanical drive with 60000 tracks scattered across several thousand folders?? I don’t want to sound like a dick, but honestly, what you are asking for is totally unrealistic. Save up & get 2 x 2tb SSD’s, copy your music across to those, add it all to the EP collection, analyse it all (It’ll take a couple of days) and your stress will be over. Then you’ll be able search your entire music collection and not even worry about where & what folder it’s in. Yes you’ll still have to add your new music to your collection and analyse it, but so does everyone else. This is a standard thing that “most” dj’s do when they get new music, whether they are using Rekordbox, EnginePrime, Serato or Traktor. Preparation is key. “Failing to prepare, is preparing to fail”. Regarding seeing the metadata of tracks that have already been analysed in the folder view. I’m totally all for that. I think that will need to be a feature request though.

3 Likes
4

I don’t think you sound like a dick at all, quite the opposite, I find your response very insightful. It may be unrealistic to expect the P4 to match the capabilities of a laptop. I did as much research as humanly possible before buying the P4 in terms of library management and how large libraries are handled. Unfortunately as a brand new platform there wasn’t much practical information availanle, so it was a little bit of a gamble for me to sell off my whole Traktor rig and embrace the Denon platform. I don’t regret it, but this is one of those key things I wish I knew about ahead of time.

I’m fine getting everything into the collection first. That method works, it’s just super time consuming, and lacking automated functionality is a bummer too. I’m going to feature request both an auto-analyze tool on a monitored drive (whether internal or external) as well as displaying BPM and key in the file browser view. Both of those would help DJs like me big time.

PS- I already have a 2TB SSD inside the P4 but I found it really unmanageable to keep that and my primary external drive in sync. I realize I’ll probably need to go 4TB external SSD at some point to do exactly what I want and have good performance, but I’m going to ride out this cheap spinning drive for as long as I can first :wink:

5

Your pain is honestly why stand-alone mode on my Prime gear is my backup. My laptop dies/gets stolen/catches fire I can still do a gig. All the added prep if I want to run standalone requires more time than I have in regards to EP. It was a sweet thought, but just not for me. But that’s the beauty of the Prime system. You can use it however is best for you! Sorry for your struggles though.

6

And that basically sums it up - the standalone mode might simply be too limited for me, despite my best efforts to emulate what I was used to with Traktor. I do think if Denon implements the two feature requests I mentioned above that it would become a lot more attractive. Regardless, I’m going to power forward with this massive collection & genre-based Crate approach, as it is working well so far, even with the slow hard drive.

7

I’m glad I didn’t offend you with what I said. Sadly most of our frustrations all point to Engine Prime and it doesn’t seem to be progressing the way we had all hoped it would. I also have a fairly large collection of music ±40K and it’s also arranged in my own specific folder structure (not as in depth as yours though). I have also had to make crates in EP to mimic my folder structure (pain in the ass), but not all the folders, just the one’s I know I use the most. This is something I learnt to do, moving from Traktor to Serato, it was the most difficult part of the transition! Serato doesn’t have a proper folder/file browser (it has a shitty little file browser above the library view). So I had to make crates just like I’m doing now again in EP. If we could see the metadata in the folder view (like you said) it would be so much easier. Anyway, it took about a full day to add & analyse about 40K tracks. Once that was done though, it really hasn’t been difficult to keep things updated in EP. I buy maybe 5-10 tracks per week, Every friday afternoon, I add my new tracks to EP which takes about 5mins, then Im done. Once a month I make a backup of the Engine library folder just in case something goes wrong. I guess it really depends on how much new music you’re adding per week or month. But once the main work is done, it’s relatively easy to maintain.

8

Even though I have expressed my gripes before with Engine Prime, I have high hopes that Denon DJ developers will eventually come out with a better solution for library management, data transfer/sync to/from Prime Series units, beat grid analysis and everything else people have been complaining about it over the last 2 years. I just hope when they do, it won’t be too little to late, because it would be a freaking shame to bury such nice hardware because of such lousy software.

1 Like
10

While the EP discussion will rage on for a time to come I guess, in essence - at least for me - dumping the extra hassle of a laptop (no matter how much others don’t find it a hassle, I did) and still having all this information handy in a way that beats “the old day” of vinyl is what makes the P4 work for me.

Sure, I too have gotten spoiled by having sync buttons and such. And yes, I understand there are definitely situations where all these things need to be flawless to function properly in your kind of DJ-ing. That said, nothing has changed when it comes to picking the correct controller/gear for you:

  1. Set a budget (don’t fall for the shiny lights lure & take into account you need decksaver, flightcase, etx.)
  2. Determine your current/desired workflow
  3. See what software (external laptop or internal standalone controller) best fits your workflow
  4. See what hardware best fits your workflow, your preferred software and your budget
  5. Try the top 2-3 picks hands-on
  6. Decide and go for it!

Imho, that is still how you decide what to get. For some still a laptop, others can work fine with the P4 right out of the box (although I do admit to using third party software to get my music ready for the P4 and don’t juse EP natively for track prep and such).

There is no wrong choice except the one you make because not following the above decision-making process.

Sure it would be nice if someone invented the ultimate DJ machine that would fit EVERYONE’s desires and needs, but that is not gonna happen anytime soon I think.

Yes, things need to improve at various levels in the software, wrongly detected BPM’s are a pain, messed up beatgrids don’t help anyone. Personally, again, I haven’t noticed that as much, or have not been bothered by it, I can easily default to manual handling things. It’s a perk of being an “old” hand at this and having come up with gear that did NOTHING automatically :-D.

Whether it’s gain-staging, bpm-counting and beat matching, it was once all done manually. And even the best DJ software gets it right less than 90% of the time, leaving you with a healthy 10%+ tracks that need human interference/input to behave as you like it. So, you either learn to do that manually or you forgo that 10%+ tracks and just don’t use them. I am not saying it’s a perfect world, but better to count your blessing as to what IS rather than keep focussing on what ISN’T.

As for the OP’s point, I could argue (have before) that it just isn’t wise to have such an extensive collection (other than for requests, which have a totally different set of requirements), but hey, everyone should do what they feel comfortable with.

At the end of the day, YOU have to be happy with your gear, software & workflow. Only then can you be the best DJ you can be.

So, keep on spinning YOUR way!

My 3 cents as usual

6 Likes
11

This thread has me worried because I have a large collection of Serato playlist and it was my understanding that they would be recognizable once loaded into Engin Prime. I brough my P4 three days ago and it’s been tough getting set up correctly. My previous controller recently broke down so i purchased the Prime 4 in haste because I have a whole lotta christmas gigs coming up. I am now struggling to get my crates anywhere near to what they looked like in serato and i know this will affect my ability to find the music I want. Don’t get me wrong the P4 is a great unit, but I know I will be DJing under stress over the next few months as I try to get my crates sorted to how I need them. I can’t seem to get my serato crates visible. Have followed the video but still not there. I’m therefore having to build all my crates from scratch. Is there someone I can pay to help me online with this as im getting stressed!

This is a major disappointment but I am willing to persevere.

12

I will advise just using it as a controller with serato. Get through your busy period first. You don’t need stress at your gig.

But if you insist on migrating completely…what issues are you having ?

It’s almost straight forward

  1. Get a SSD, same size or larger than your Serato music folder

  2. Format it to ExFat

  3. Fire up Engine Prime

  4. Click on the Serato tab

  5. Refresh library

  6. It will take a few minutes

  7. Once that’s complete, CMD + A All tracks in the Serato tab

  8. Right click and Add to Collection

  9. Now all the tracks are in EP collection

  10. You can go back to The serato tab in EP side panel and add each crate to collection as well

Now all tracks and crates are in EP

  1. Plug in drive you intend to use with your prime hardware

  2. Now start dragging each crate to the destination drive

  3. You will see the drive in the drive panel in EP

That’s it

13

I too learned on vinyl and still love spinning it - I will never sell my 1200M3Ds! Having a foundation in actual beatmatching I think should be a requirement for any self proclaimed DJ :wink: (I’d make the same argument about sports car drivers learning how to drive stick, haha)

Despite my love of vinyl, I was an early adopter of digital mixing, starting with the original Final Scratch and moving on through FS2, Traktor DJ Studio, Traktor Pro (and several different NI controllers) and now Engine Prime and the P4. Throughout my analog and digital journeys I’ve learned what I want most in a DJ platform. And that is to have all pertinent information at my fingertips, and at a moment’s notice - I DJ “live”, I don’t methodically plan sets out ahead of time. I may have a group of say 100-150 tracks that I’ll pick for a gig, but I vibe off my audience and play what I feel like in the moment, in no predetermined order.

And that’s where I’m running into problems with the current Denon platform. Without the ability to analyze just a handful of tracks on the fly, and having no metadata displayed in my file browser view (which is where I spend most of my time, due to the size of my music collection and how difficult it is for me to find stuff in the other views on the P4), I’m forced to become that “plan everything out in advance” kind of DJ which is counter to how I’ve always been.

Another example for how this is a problem for peeps like me: Let’s say I have another DJ friend over who brings their thumb drive with a bunch of new tracks they just downloaded from somewhere like Beatport. We’re playing b2b and now it’s his turn up and he plugs in his drive. Cool, there are all his tracks, but no metadata. OK, so now what? He has no idea what BPM or key they’re in, so he’d like to analyze them. The onlyway to do this on the P4 directly is to manually load every song into a deck and wait for each to be analyzed - and then, the only way to actually see this analyzed metadata in a list view is to go to the History view, because there’s no way to see that info while browsing his drive directly. But that History view includes my track history too, muddying the waters and making it harder to find what he’s looking for. The only other alternative in this scenario is for him to add his tracks to my Collection, but obviously that’s a painstaking process with having to copy his music over to my drive first, launch EP, and analyze that new music - all of which requires a laptop and a lot of time and planning ahead of time.

Bottom line, as revolutionary as the P4 is (and trust me I do love it), it still doesn’t allow for any kind of on-the-fly DJing in terms of file analysis, and the various views are not consistent with displaying metadata. Yes, us oldschool DJs can still beatmatch by ear and do not have to have BPMs displayed ahead of time, but my goodness it certainly makes things easier!

1 Like
14

I understand what you are saying here and agree with a couple of things, but what is different between using the P4 and going back to back on a set of cdj’s?

15

Regarding your friend bringing a rekrordbox memory stick over to play… there’s nothing stopping him installing engine prime on his computer at home & prepping a memory stick up before he leaves.

16

This Link will sort you out. Even very good for Serato & I Tunes users too.

[Massive Collection Help For Denon Prime]

(https://www.youtube.com/watch?v=py68W-SIXys&feature=youtu.be&fbclid=IwAR0ZphF5fs2s_4Ec_Y6hAajX70uICddaF4KZS7VUSEYPDWqWouKtX7mstLM)

17

A whole hell of a lot -

  • CDJs (at least the Pioneers) understand NTFS and Mac formatted drives so it’s easier for DJs of different varieties to start DJing immediately
  • CDJs do not have comparable processors or memory like the P4 and thus have no capability to do some of the things the P4 can do - especially in regards to searching and sorting library content in different ways
  • A CDJ setup requires at least 2 separate CDJs and a separate mixer, with all the cabling and power that goes along with that - nowhere near as integrated and streamlined as a P4, so it’s not something that’s as easy to take over to a friend’s house or bring to an impromptu gig

I could go on… but the point I was making is that because the P4 is so advanced, I was hoping it would allow me to mix in the same way Traktor on a laptop does. I realize that currently the answer is that it cannot - and that’s fine, I will get used to a new style of mixing and see how I like it. I just hope that one day it can better emulate the style I’m looking for with this kind of controller based setup.

18

if like me the problem is i have a 2tb drive which is full of folders arranged by label, so therefore the drive has over 1700 folders on the top layer and then theres folders within folders within folders etc, so to add and anylyze my collection will take days, but theres a way. you will need an empty drive the same size formatted in exfat/fat32, open your drive and your source drive in windows explorer and on the source drive window search mp3 wav flac etc then add the files without all the folder business so you can add the drive to your collection and scan them in EP in 1 big hit, or you could break it down how you want if you have genre’s in folders you could do the same on the folders and create folders on the new drive how you want it without too many sub folders then use EP to sort by bpm/key/genre etc and put into crates as you wish

19

Don’t use Fat32 for a big collection, coz the database that Engine creates will reach the Fat32 4gb file limit.

20

Ok, I think I misunderstood what you were saying. The reason I compared playing b2b on the P4 or cdj’s was to point out that you wouldn’t get the functionality you wanted from using cdj’s either. Currently the Prime series is the closest you will get to what you are looking for. Hopefully things you are after will gradually get implemented with updates along the way.

21

What about having two or three 2/5 TB external SSD drives plugg into the laptop and or one also plugged into a USB hub? I have 12 terabytes total hooked up with two or three drives for my Traktor S 4 and 8. It works well as long as I have the computer doing dj stuff not office email etc. NTFS Windows pc 2 years old 16 gigs of RAM i5 processor nothing special not a gaming computer. I’d like the prime 4 and to keep my music…?? Any input would be greatly appreciated.