I have since found a plugin that mostly does this, called "Build Switcher".
It is limited by a few things however:
There is no API support to get the build system list for the plugin to function properly, so you have to manually define the build system names. (at least according to the plugin, has support for that been added since? The plugin doesn't appear in the package settings list so it may be old).
There seems to be a limit of 8 build systems you can assign to a project. More than that does not appear in the build system list and is not switchable. This is unfortunate because I have 20 different build targets I need to switch between. (Developing a C++ application that supports 4 target platforms and debug/release/profiling/final support for each of them).
I've also since discovered that I can delete the sublime-build files to remove the default ones, but you are still restricted to 8 build systems in a project.