Yes, I was also referring to the line numbers and not the entire lines. The idea is that git could color the line numbers (as mentioned, red for deletion, green for addition etc) and other plugins such as BracketHighlighter could use the gutter to place an icon there (as they do right now). Alternatively, the line numbers could be colored by the linter.
This approach would at least provide two ways to show things on the side, which is a bug progress. Plugins could provide an option to choose which one is used.