Note that the sidebar is for the files you’re working with in your project and it not meant to be a full on filesystem brower; as noted above, when you open a folder, Sublime spiders the entire content of the folder structure to discover all of the files and folders, so that it can display them in the side bar and show them to you in the file list for Goto Anything
(and they are also indexed so you can jump to symbols in them.
Quitting Sublime and restarting it by default brings back the state that it was last in, which includes the files and folders that were open previously.
So, if you accidentally open a folder that you don’t want to be open, you need to take action to either remove the folder from the list of open folders, or open a new Sublime window and then close the first one (which is itself "removing the folder’).
If you don’t, every restart of Sublime will keep restoring the same state.
This includes if you uninstall and reinstall it, because that just removes the application, but your configuration and state is kept safely untouched in your home folder.