Yeah, I was reading some of your older posts about this too, due to this posts. That, or a dropbox library (which is worth the investment if you care for backups anyway)
Still no regrets or downsides about having your main collection on external drives? It feels a bit scary to have your main collection floating around on an external drive, but it solves a lot of portability problems, and separates the whole shazam from iTunes (which keeps doing funky stuff anyways). You could insert it into your Synology from time to time for backups…
Wow, your comment comes off as negative and judgy. This isn’t a democracy and we’re not voting on features here. I have no idea how valuable this would be to the wider audience, but sounds like you probably don’t have the information to make a ‘99.9%” claim.
I buy a lot of music on a computer which is not parked at my DJ setup.
I still have a bit to learn about the best practices here. Mostly, I’m just surprised that simple stuff like preventing duplicate tracks from syncing isn’t a thing. Since these two machines are in the same house, I’ll probably just rsync the database. Hell, if the app lets me pick a location, I can just select the iCloud drive I want to be using. As long as only one Engine Dj app is open at a time, should be fine.
Does this really work? In the past it was recommanded not to change the engine library location. If yes, it would be great to use the 1tb OneDrive for it
I can´t understand, why it´s still impossible to add services like Onedrive, iCloud and / or Googledrive to Engine.
So many Users requested it several times.
Dropbox is much to expensive. Especially since it’s completely unnecessary when you already have subscriptions to onedrive 6* 1tb (Office 365 family 6 user, by far the best and cheapest solution), icloud 2tb and gdrive. I’m not going to pay for another storage provider like dropbox anyway.
This topic is one of the reasons I decided to have my file structure from my PC synced to a T7 and then Engine builds the database on that external drive. I can use synchredible to sync over any new files, drag and drop them (most recent changed) to Engine and everything is all on the one drive (which can then be cloned or backed up as necessary).
Saves a whole lot of hassle and is definitely the best way to run Engine just now (if you don’t want the hassle or cost of using 3rd party stuff).
I understand everyone works differently but the basic fact is that Engine in it’s current state is very poor.
STU has his writing style, but in the end he is right: there isn’t much software with built in syncing features. Syncing often leads to conflicts, which leads to unexpected situations: you don’t want to know how often I shut down my work computer too soon, continuing work at home, not waiting long enough after booting my laptop, just to learn a few days later that my Dropbox didn’t sync fast enough, ending up having to join 2 excels. Nobody wants to actively support this…
There are a lot of ways around this, and actually, if you play around a bit with Engine, there is some kind of beauty in the external drives library. You can even use an external drive you drag around, circumventing sync needs, you can use use Dropbox, Remote Desktop, build your own solution using rsync or symbolic links, each with up- and downsides.
I advise the external drive option, after reading this and that…
Can you elaborate on how you process your files? I have a really hard time working with the strange “collection” system that EnginDJ designed. I have a completely fine folder/directory structure on my PC, with all the audio I need to have on my Prime4+ under one directory and all sorted in appropriate subdirs. When I drag that dir (that has all my subdirs) from the Drive onto “Collection”, it creates that same dir structure. But as soon as I want to add another dir or bunch of files to my own structure on disk, I need to entirely RECREATE that Collection, there is no way to simply add them to the collection, because when I drop it on top, it will create another entire collection and renames is with a “2” added to the name. Such a PITA! Whoever thought this was smart or handy should be fired. How on earth do I simply add a few files that reside in my own directory structure to a collection without having to select each file and add it to where I want it to reside in the ‘collection’ structure that EngineDJ created?
As far as I see it, EngineDJ could have skipped their entire “Collection” invention. We all have collections on disk already, we don’t need an extra virtual folder-structure for this. Also, engindj does not have to put the files in its own dir-structure on the disk/storage inside the Prime4+ either, they’re already sorted just fine where they are. Super annoyed by this. And I did not expect this from a brand like denon.
I see a lot of misconceptions in your post. Let me try to help:
At first, the structure under “collection” arent folders, these are playlists. The concept here (and with almost all competing brands), is
that you have a bunch of MP3 files
your library software adds these into a large database of tracks, pointing to the physical file using the path
and then you can add your tracks into playlists. Playlists can be viewed as “crates”, but have the advantage that you can have one physical file, with one set of loops, cues, etc, appearing in multiple playlists. So you can have a bunch of playlists showing these tracks by genre, or another bunch that show tracks by energy, or another that shows them by era. 1 file can appear in multiple playlists.
When you drag a folder structure to the collection however, Engine will put the tracks you added in playlists resembling this folder structure. Thats more of a convenience, Engine doesnt have to do this, but these playlists are no folders!
the path is a unique value in the track table of Engine. So one MP3 file can only be added once to the “collection”. Within that collection it can ofcourse appear in multiple playlists, but that is still one track. If that track appears multiple times in the collection, then there are 2 physical copies references.
Engine puts a database on each disk it encounters, and a track is a member of the collection on the disk it resides itself. Meaning that when you add a track which you put in you disk yourself, Engine will just reference it, and leave the file in your folder structure. If you however have this track on another disk, but add it to the collection of your disk, Engine will first physically copy the file to the disk of the collection you are adding it to, and that will be in a folder structure within “Engine Library/”. If you change the original location, and then drag this file again, duplicates will appear.
Hope this was helpful, and that you can track down the problem in your workflow…
That’s not solving the problem though. The issue is: Adding my dir and subdirs of already sorted audio-tracks to Prime4+ internal SSD can only be done using this route. I have to first add them as a Collection in order to sync them with the Prime4+ storage. This is why I’m even using this method. If I want to physically add music to the internal or USB storage of the Prime, and I do it the normal MTP or USB-PC route applicable for normal external storage devices, the denon prime does NOTHING with the files I’ve put on its storage. This is really super-annoying. Denon decided to process tracks only in virtual format, not actually based on disc/storage location. So all of you have loads of useless doubles on its internal storage, because every time you add a new file, if it already resides on the internal storage, denon creates a new soft link to a new file, never the existing exact same file already there. Either way, the design of their ‘sync’ interface is simply horrible. You only sync virtually, never the way rsync or actual true syncing would. And this virtual syncing is useless for people who already have their own folder/directory structures maintained in TOC physical format on storage media.
In addition, what if my playlist already exists on disk? What if I simply want to add the playlist based on my already sorted tracks in a directory? I can drag a dir (also quite strange, why not just move the dir directly to the Prime4+ storage?) to ‘Collection’ so it creates a playlist with that name, but then, whenever I change or add or remove tracks from the source dir, like I’ve been doing for years on my PC, it never reaches this ‘Playlist’ in EngineDJ, hence it never reaches the internal storage of the denon prime, hence everything I’ve done, cue points, phrase-matching etc. gets lost, because I have to re-create the virtual Collection based Playlist out of the existing dir with every addition or change. It’s insanely silly.
These are 2 functions that serve different purposes: you maintain your own folder structure and library on a storage device, OR you use “sync manager” to make a copy of your main library to a secondary storage device. Mixing these 2 workflows inevitably gives unpredictable results.
Dragging files or directories is a “one time” import. Engine happens to replicate your folder structure, but it was not designed to sync a folder into the library. If you add files, you will need to add these to the library yourself, like with every other DJ software.
Because Engine, like every other DJ software, has a database which serves to maintain metadata like cues, loops, playlists. If you don’t want to use this feature, you can however just drag files to a storage device, and browse it’s folder structure on a Prime 4 or SC6000. But you’ll notice some features aren’t present in this browsing mode…
Again, a lot of misconceptions. I suggest a good read of the manual
Again, they are NOT “misconceptions”. You seem to miss the entire point I’ve made earlier many times in other threads as well: A TOC for files and dirs on disk already IS a database, denon does not need to re-invent the wheel and think up a virtual TOC on top of the existing one, which it now does. It’s completely useless and very inefficient use of CPU and storage IO, and it adds a heap of extra problems from there. All the metadata you could want to add to audio-files on disk can be added in any database, as long as it points directly to the file. That’s NOT what denon does, denon re-invents a table of content structure on top of the existing one, accesses its own TOC and then ruins all references to the original location on disk as soon as you ‘sync’ this file/dir to your denon device.
This is not how all DJ software works, in fact quite the opposite. Most databases reference files directly, so that when you move it or alter it, it is also moved and altered in your DJ controller/software.
Just look at your internal storage of your Prime 4+, if you have it. You’ll see how denon has decided to store files on it, based on how it decided to recreate the wheel for you, completely ignoring ANY prior organisation you’ve done beforehand. The only organisation and metadata and playlist-content exists virtually, for denon’s own reference table, not any existing file/dir structure. Which means that you can never change anything to audio on your storage without ruining denon’s meta/database contents.
Really, all software I ever used uses the file path to reference a file. Traktor, Rekordbox, Serato, Ableton, Logic, Photoshop, Aperture, Lightroom, Xcode, should I stop here? If you move the file it is missing by definition for any application that references that file. Rekordbox is the only other competitor that has hardware that uses USB sticks as a database, and copies the files to the USB storage device itself, your USB stick is a completely closed system in Rekordbox. But it still uses it’s own database format on top of a regular FAT filesystem. Engine gives you a bit more freedom, but with freedom comes more possibilities to break things.
You can NOT add random data to a system database like the filesystems TOC. a filesystems TOC has a strict format so it can be read by any operating system supporting that filesystem. Randomly adding fields to the filesystems TOC would break any support by such operating systems, and therefore would be a new database format on it’s own. It would very much likely break the ability to boot from this volume too. Any bug would further risk catastrophic data loss. your FAT/TOC is off limits for user space! A TOC bridges the gap between user space software and kernel space, and there is a good reason there is such separation. If not we would be catapulted back into the MSDOS era of single-user/single-tasking operating systems, were full control over hardware was handed over to the application.
PS, from the other thread:
Thats another thing you have backwards: scanning a filesystem, opening each file, and reading the metadata from it is very CPU intensive. If you want a load time of a few seconds instead of a few minutes up to a few hours (depending on database size) you really need a database centralizing all this info. 1 specialized database file is faster than parsing this info from 10.000 files, containing mostly information irrelevant for browsing purposes (99.9% is audio, 0.1% is ID3 tags). You would have to read a few gigabytes of data instead of a few megabytes… On top of that, tracks and playlists have a many-to-many relationship, while tracks and folders have a one-to-many relationship. Not using a centralized database would mean you have 2 or more hard copies of one file if you want it to appear in more than one playlist, wasting disk space. Smartlists would be impossible.
Really, with all due respect, @DeJulius, if you think that you can just add columns with information to a filesystems TOC then you do have a lot of misconceptions on how computers work…
You totally seem to miss the points that are important here.
Let’s say you’ve created a heap of playlists on your EngineDJ desktop app. ALL tracks in those lists reference audio-file locations on local storage/HDD/SSD (if you’re lucky, because whenever you link EngineDJ with your controller, this all gets even messier, because some files will be referenced to the local storage ON the controller, and some not, but that’s a different issue altogether).
If I or other software then change a directory/folder-name, content, or location of the audio-file(s) on local storage, suddenly all playlists have become useless, can’t find the files. Not even when I purposely select “Auto relocate files” EngineDJ seems unable to even search the same HDD/SSD for the changed file locations, which is pathetic all by itself; Even a monkey is better at relocating files on the exact same storage medium. Just try it, change the foldername holding your entire audiofile collection one character and ALL of EngineDJ turns red and is unable to relocate the files. ALL playlists have become moot.
All this is caused by the fact that denon has decided to virtualize your file-system database with overlayed database.
A much simpler way to process audio-files from and to your denon prime 4+ storage would be to mirror the file/dir structure of source and target from both ends;
If you have a music collection on PC under "M:\All-my-DJ-tracks" you could select that folder/dir in EngineDJ config so it knows what to keep synced with your P4+, always assuming that changes to the files on PC are master over those on your controller. THAT would be actual syncing, and that’s how all syncing works with services like dropbox or google or onedrive etc. Not with the current EngineDJ.
The storage in my P4+ invents a new directory structure that has literally nothing to do with the source dir on local PC HDD/SSD. Making it impossible to ever know what is synced or not, without first having to check a virtual reference for every file. This slows things down and is a huge point of failure, as described above; Change one character of that huge folder and ALL references are broken.
All that EngineDJ and the Prime4+ have to hold in their own database is the additional data, like key, BPM, and history when used/loaded, and keep playlists in sync, which all can simply reference the audiofiles in the exact same folder/directory structure on both ends. Location of source audio is separate from all other datapoints belonging to those audiofiles. It makes no sense whatsoever to link the two together and re-invent the wheel for file-locations/dir-names and mess up syncing that way.
The problem would persist, because if you move/rename a file/folder in that folder on your M drive, tracks would still go missing…
Also, your reference to Google Drive or Dropbox is a bit disputable, because such cloud storage services don’t care about the contents of a folder structure. They just sync the folder structure, because that’s all there is to sync… DJ software retains metadata, making an XML or SQL database, referencing the filesystem, necessary. Engine does this, Traktor does this, Virtual DJ does this, Rekordbox does this, Serato does this, even iTunes does this…
Dont move or rename files in bulk, and that goes for any music software!
