Engine is really starting to get to me now. i have added 2 new folders, scanned a, analised and stemed them but when trying to sync JUST THESE 2 folders engine does a totall collection estimation and leave me hanging for hours. i dont need it to calculate the whole bloody collection again. i just want to add 2 folders to my drive. i cant even just unselect everything else as it atutomattically selects everything when i launce sync manager. i just want to quickly add new music. this is making it almost impossible to prep sets. any advice? i have a good pc, ryzen 5800xed, 32 gig ram and a gen 4 ssd. just moving files from my pc to the same drive takes seconds, its just moving via sync manager thats taking ages. its defenatly not a sepped issue with either the source drive or the destination drive.
What media are you using to store your collection for performance? SD cards are generally very bad at random read/writes and depending on your collection, it may be part of your bottleneck.
Good quality SD cards are very good. I use SD cards since 2013 - never had a smallest issue.
i have multiple media and it dosent matter which i use i have the same issue. i have a usb 3.2 external ssd and a high speed sd card as mains and some usb sticks. i like to over prepare and have multiple backups on hand. its defenatly not a media issue as ive said, normal transfers work really fas. its just sync manager thats slow or hangs
Are you using Dropbox to back up your pc music folders
no. im just using sync to make my play drives. its a big collection as im an open format dj with a huge spread of people lol. the initial export took overnight and that was fine. i understood moving a large collection would take tme but its the fact that just adding another 2 folders with 40 tracks takes hours. as soon as i open sync manager it starts calculating and then it calculates the whole libary. i dont need the whole libary calculated. i just want to add those 2 folders.
So glad you have not experienced any type of corruption. you’re pretty lucky.
You say this but I think your understanding of media and computers may be flawed.
There are plenty of things to consider when it comes to your issues:
- Storage type and its ability to read/write data randomly (sustained write speeds are not the best benchmark for this).
- Interfaces: you can say USB 3.2 but are you using the right cables and actual interfaces on these devices.
- Are you using USB Hubs?
- Are your devices connected to the same USB controller?
- How is your media formatted? FAT32 or exFAT? The latter is much faster.
- The number of files in your collection that are actually loaded in Engine. The more files loaded into the collection, the slower things will be. Engine will need to read and write from all of the databases for all of the connected media at some points in your workflow. Upon export, there is a lot of little read/writes that are happening all the time and media like SD cards will work really slowly because of this.
I’ve done a crap load of tests and writing my collection to a v60 SD card is orders of magnitude slower than writing to a 500MB/s USB 3.1 SSD, and that’s all even slower than writing to a USB4 NVME drive.
All of this stuff matters and without having much detail, it’s hard to help you. Please share exactly how you’re using Engine =).
i have…twice…hense the backups lol
and ,respectfully, i dont think you understood my inital post explanation. my poor typing wont have helped. using windows explorer i can move files from my internal gen 4 ssd to my usb type c usb 3.2 drive in about 55 mins. doing the same using engine sync manager takes around 15 hours. the way ive been successfully making backups is to leave the sync manager running overnight and then just copying the created folder from the target sync drive to my other backup drives via windows explorer instead of syncing each drive via sync manager. this is done with each main drive in under an hour instead of 15 hours via sync manager.
the point of this post origonally however was that i just want to add 2 folders of music to my existing libary. if i was just copying to a drive that would take seconds. but running through sync manager it takes hours because syncmanager decides to recalculate the whole libary when i only want 2 folders moved, not the whole libary. i know the rest of the libary works fine and the backups work fine. even if i have made no changes to the libary (cue poins or beat grid fixes for example ) it takes hours.
this is without changing any cables or how its interfaced. windows explorer is faster by a huge margin.
so this is my workflow.
- the full libary is already analysed and songs requiring stems are analysed for stems. the libary clean up has been run and libary is backed up.
- i connect my gen 3.2 external ssd to my motherboards type c connector. the host drive (that is my c:drive) is an nvme gen 4
- i export my libary via sync manager (taking 15 hours for 580 gig, i know , its a big lib) to my external ssd (d: drive)
- i then exit engine dj
- i connect my backup ssd,s and use windows explorer to copy the folder created on the d: drive to them via usb 3.2 ports on the motherboard. not i havent disconnected the d: drive at any point. This transfer usually takes less than an hour (not 15)
- i decide to add my 2 new music folders (aprox 400 tracks) to engine. i analyse them and run stems analysis on those tracks i want stems for. again on my c:drive
- i now want to sync those 2 files to my d: drive so i open sync manager and wait up to 2 hours for sync manager to calculate. it gives me the warning that its doing a whole collection estimate and it may take some time.
this is where the main problem is. i dont need to estimate the whole drive again. i havent done anything with that drive . i just want to add the 2 folder that i have scanned and analysed. ones it finishes estimating it moves those 400 tracks in about 30 mins. not the few seconds it takes in explorer
hope that makes it a little clearer. i apreciate everones help. im just getting frustrated by this. i adore the hardware. ive been a denon fan from the cd days. my controller for years was an mc3000 that i refuse to get rid of because its a tank. my sc live 4 has made djing fun for me again but the software is frustrating me sooo much. i was spoiled by being used to virtual dj where the folders are just there lol. but engine desktop seems to be my weak link here
oh and exfat formatted for all my target drives
No, I just know what I’m doing. Choosing the right sd cards for their purpose. My cameras sd cards also never missed a single frame. My performance sd cards never had a corrupted content. All works as it should. Once in my whole life I encountered a corrupted sd card. After formatting it, it worked fine till it’s end.
mine was because of a power cut at the club and then because i was new and just took card out without letting it fully ejecy. dumn i know lol
my typing is getting worse…bloody fat fingers and old eyes lol
You’re doing fine 
Thank you for taking the time to share this information @Jason_Cassidy ! I too get very frustrated with Engine DJ, though the dev team doesn’t seem to prioritize making it much better. That’s what it seems, at least, where there are bugs that are years old. 15 hours is not entirely acceptable and I can understand how one can blame Engine for being slow, however I like to look at the entire system and the DJ’s workflow as part of the equation. There are so many factors that play a role into how well Engine works, including Engine’s own limitations.
Let’s tackle a few things.
Copying music from one drive to another using Explorer:
When you copy files using Windows Explorer (via drag and drop, etc.), the operating system is basically writing the files from the source storage to the destination. It typically does this at the fastest possible bandwidth each device can allow. Some of the overhead in copying files includes updating the filesystem allocation tables (This is where FAT comes from - File Allocation Table). And when you copy music, there are tiny periods where little writes happen (allocation table updates), and longer periods where large writes happen (writing the data for the songs). Copying files with the operating system is typically fine tuned and as fast as possible, full stop.
Exporting using Engine DJ: This is a lot more complicated of a process and is way slower than the above, for a lot of reasons. When you export, there are a lot of database reads and writes that need to happen for each of the file transfers, in addition to the OS file copy process like stated above. It’s for this reason, exports are generally slower.
If you have bad media, the above processes can take exponentially longer, as reading/writing cells that are going bad (but not entirely bad according to the S.M.A.R.T. specs) will increase latency.
Some questions for you:
- What make & model are your external drives?
- How large is your host computer’s main Database
C:\Users\[Your Username]\Music\Engine Library\m.db.
- How large is your main performance Database?
D:\Music\Engine Library\m.db
- When is the last time you have run a Database Cleanup (with your performance drive connected)?
- When you do an export, have you looked at Task Manager to see if CPU utilization is high or perhaps Disk Utilization for
D:\is above normal? - Would you be willing to share your host computer’s Main database with me? Don’t worry, this won’t leak your music, just metadata about your music. I wonder if there is a limitation you may have run into?
Lastly, you’re not the only person experiencing slow exports. If you search this forum you’ll find posts like this:
ill be happy to share my database my friend. im not precious about my music lol. im working the next few days so will sort all of that early next week.
sd card i use is a lexar play sdhx 1tb (i now not the fastest but i do a windows explorer copy to this)
an orico 1tb portible ssd
a crucial ct2000p3 nvme 2tb in a thunderbolt 5 enclosure
and a crucial mx500 2tb in a usb c 3.2 thunderbolt enclosure.
boot drive is another a crucial ct2000p3 nvme 2tb
but even though whole collection export is still a problem its not the MAIN issue im having
thats trying to just add 2 folders to my performance (target ) drives without engine having to recalculate or estimate the entire collection. i just want to add 2 folders.
oh and the 3 performance drive are all less than 2 months old. only got the unit in november and got these speciffically for the denon. the sd lives in the unit unless i reallly need to do a major update to the collection . the denon finds and playes everything perfectly and quickly. not as fast a search as vdj but i wouldnt expect that anyway.lol. im actually very surprised how fast and robust the unit is. its a dream to work with. ive been djing since i was 13 and am now 52, and i used to do design and installation of sound and lighting systems for pubs and clubs. i wish this gear had been around 20 years ago. would have made my life so much easier (apart from the engine issues lol)
Engine DJ sync organizes your music on the performance drive by Artist and Album in the \Engine Library\Music folder. Therefore, your music isn’t just being copied as stored on the source. This is part of what “more complicated of a process” means. I imagine when you start sync it checks everything done previously, with a large library that will take time.
You could deselect everything via the “Playlists & Smartlists” checkbox at the top of the source sync drive. BUT, you don’t want to do that because you will be telling sync not to sync those playlists and to delete them. At least, that is how I understand sync.
Of course, these comments don’t remedy your situation.
I use Mac, but I have discovered a few changes with Engine DJ and sync v 4.1.0.
- For context, my Engine DJ library is only a small curated selection of my collection, 2500 tracks, much smaller than yours.
- When opening Engine DJ desktop, it freezes with a rainbow wheel for a minute or less. I use a lot of smartlists, so I assume it is checking those and hope it doesn’t become a bottleneck.
- Syncing now appears to be much faster. But, in actuality, even though sync says it is completed, there are processes (file copying) still going on in the background. If you try to eject the performance drive, Engine DJ and FInder will both say that the drive cannot be ejected because it is being used. You can see music files being added to the performance drive music folder. Therefore, it is imperative that you wait for all of these processes to be done. Technically, Engine DJ should show that they are running and a progress bar would be helpful.
Same here. I have >50k files and >100 playlists. For performance I only need some of them, never all and depending on the type of event, I need different sets of playlists.
Exporting takes very long, due to the bookkeeping necessary for every file. We have 2 databases to update, and the file to copy. In the beginning, Engine must count all files and add their sizes to be able to determine whether the target drive has enough space. It does this again for every click to include/exclude a playlist in the export set …
This is unsatisfactory. But with the concept of performance drives and the proper management of such drives it seems to be unavoidable. My solution to this is the following:
I do not export/sync any longer. My files do not live on the main computer but directly on (several) USB sticks. I can plug sticks into my main computer and edit everything with Engine OS. When finished, I simply plug in the sticks to my performance hardware (Prime 4 and Prime Go), and everything works flawlessly. No exporting, no syncing, no nothing. Simply works.
Te only thing I need to be aware of are the drive letters of the USB sticks. A stick ALWAYS should have the same drive letter, otherwise Engine OS gets confused. I manually assigned drive letters starting from P what avoids any conflicts.
Another benefit I enjoy is that I can work with my main computer in the studio as well as with a laptop. I just plug in the sticks and I can work remote.
I do not have this issue with Engine.
- My main library contains approx 80k tracks.
- Any time I add new tracks/playlist to my main collection, it takes a few minutes to update my performance drive with the new tracks/playlist.
note
- I don’t pre-compute stems
- Main collection = Samsung T7 (APFS) via USB.
- Performance drive = Samsung T5 and Sandisk Portable drive. Both ExFat.
- My library is synced to engine from Serato DJ. I am still actively using Serato.
Did you try a new collection database?
Before you proceed Back up your current database folder.
TLDR
Even if I wipe my performance drive and do a full export, it’s a 80 mins job.