Skip to content

BFR: Big Refactor (new scanner and DB schema changes) #2709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 562 commits into from
Feb 20, 2025
Merged

BFR: Big Refactor (new scanner and DB schema changes) #2709

merged 562 commits into from
Feb 20, 2025
+16,535 −5,801

Conversation

deluan
Copy link
Member

@deluan deluan commented Dec 15, 2023 •
edited
Loading

⚠️ If you want to try it, be sure to backup your DB first! ⚠️

This is the branch/PR that holds the new Scanner implementation and changes in the Database schema. This will require a big refactor in large chunks of the codebase.

Issues that are being tackled by this PR:

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 15, 2023 •
edited
Loading

Download the artifacts for this pull request:

@deluan deluan force-pushed the new-scanner branch 4 times, most recently from c53d1bf to f68f9b7 Compare December 16, 2023 18:41
@deluan deluan mentioned this pull request Dec 16, 2023
Open
3 tasks
@deluan deluan changed the title New scanner WIP: New scanner Dec 18, 2023
@deluan deluan force-pushed the new-scanner branch 8 times, most recently from f5e98f0 to a7af130 Compare December 20, 2023 23:34
@tty228
Copy link

tty228 commented Dec 21, 2023 •
edited
Loading 

INFO[0127] Scanner: Starting scan                        fullRescan=true numLibraries=1
DEBU[0127] Scanner: Found folder                         _path=/home/****/GitHub/navidrome/music/1 audioCount=1 folder=1 hasPlaylist=false images="[]"
DEBU[0127] Scanner: Found folder                         _path=/home/****/GitHub/navidrome/music audioCount=4 folder=music hasPlaylist=false images="[]"
DEBU[0127] Scanner: Found folder                         _path=/home/****/GitHub/navidrome/music/2 audioCount=1 folder=2 hasPlaylist=false images="[]"
INFO[0127] Scanner: Finished scanning all libraries      duration="713.296µs"

1234

I attempted to scan songs, and the logs indicate that music files were found, but the song list is not displayed in the UI.

Version
0.50.1-SNAPSHOT (e8adcda) from https://github.com/navidrome/navidrome/actions/runs/7281968279

How Navidrome is installed?
navidrome_linux_amd64_linux_amd64_v1

Environment

  • OS: pre-built binary (ubuntu-22.04.3)
  • Browser: Firefox
  • Client: Web interface

@deluan
Copy link
Member Author

deluan commented Dec 21, 2023

Hey @tty228, thanks for trying it out, but... WIP = Work In Progress 🙃

The work is far from being in a usable state. I created this branch to have a docker image available for my own tests. When it is usable, I'll change the description and will ask feedback in our Discord dev channel.

@deluan deluan force-pushed the new-scanner branch 2 times, most recently from 4f61c0a to ef51ff6 Compare December 22, 2023 00:27
@tty228
Copy link

tty228 commented Dec 22, 2023 •
edited
Loading

Thank you for your reply. I was just curious and gave it a try. I mistakenly thought this was a version that required testing and feedback. 😆

@deluan deluan force-pushed the new-scanner branch 9 times, most recently from 56e2c95 to 89b0e9e Compare December 27, 2023 17:54
deluan and others added 4 commits February 16, 2025 20:20
@deluan
add catalogNum to AlbumInfo
02bfe88 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan
remove dependency on metadata_old (deprecated) package
5204aa2 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan
add RawTags to model
2188c9e 
Signed-off-by: Deluan <deluan@navidrome.org>
@kgarner7 @deluan
support parsing MBIDs for roles (from the https://github.com/kgarner7…
097fec7 
…/picard-all-mbids plugin) (#3698)

* parse standard roles, vorbis/m4a work for now

* fix djmixer

* working roles, use DJ-mix

* add performers to file

* map mbids

* add a few more tests

* add test

Signed-off-by: Deluan <deluan@navidrome.org>

* try to simplify the performers logic

Signed-off-by: Deluan <deluan@navidrome.org>

* stylistic changes

---------

Signed-off-by: Deluan <deluan@navidrome.org>
Co-authored-by: Deluan <deluan@navidrome.org>
This was referenced Feb 17, 2025
Closed
Closed
@deluan
remove param mutation
a025d27 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan
run automated SQLite optimizations
0d7c6fc 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan
fix playlists import/export on Windows
f330648 
* fix import playlists

* fix export playlists
@deluan deluan marked this pull request as ready for review February 17, 2025 20:22
@deluan
Merge branch 'master' into new-scanner
d42efb5
@deluan
better handling of Windows volumes
4b2e728 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan
handle more album ID reassignments
1fd121b 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan deluan changed the title WIP: Big Refactor (new scanner and DB schema changes) BFR: Big Refactor (new scanner and DB schema changes) Feb 17, 2025
deluan and others added 6 commits February 17, 2025 20:47
@deluan
allow adding/overriding tags in the config file
e545908 
Signed-off-by: Deluan <deluan@navidrome.org>
@kgarner7
fix(ui): Fix playlist track id, handle missing tracks better (#3734)
94daca8 
- Use `mediaFileId` instead of `id` for playlist tracks
- Only fetch if the file is not missing
- If extractor fails to get the file, also error (rather than panic)
@deluan
optimize DB after each scan.
ec4d2ac 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan
remove sortable from AlbumSongs columns
b9fbe6f 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan
simplify query to get missing tracks
9c2de68 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan
mark Scanner.Extractor as deprecated
daf128e 
Signed-off-by: Deluan <deluan@navidrome.org>
@deluan deluan mentioned this pull request Feb 19, 2025
Open
18 tasks
@deluan deluan merged commit c795bcf into master Feb 20, 2025
34 checks passed
@deluan deluan deleted the new-scanner branch February 20, 2025 01:35
@deluan deluan mentioned this pull request Feb 20, 2025
Closed
@thebiblelover7
Copy link

This is exciting! Can't wait to try it all out! Thanks for all your hard work @deluan !

@furudean furudean mentioned this pull request Feb 20, 2025
Open
1 task
@BrewTestBot BrewTestBot mentioned this pull request Mar 10, 2025
Closed
@github-actions GitHub Actions
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 21, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
2 more reviewers

@mintsoft mintsoft mintsoft left review comments

@deluan-tw deluan-tw deluan-tw left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
bfr Issues with the BFR version frozen-due-to-age
Projects
Navidrome roadmap
Status: Released
Milestone
Big Refactor
Development

Successfully merging this pull request may close these issues.