I work a lot with on collaborative projects which have loads of files, but I rarely care about the majority of them and I therefore leave the majority of the project’s folders collapsed.
I’ve noticed that the folder refresh always happens in breadth-first order, and takes a fair amount of time refreshing folders I don’t care about.
Please consider prioritizing refresh of uncollapsed folders over collapsed ones.