Sublime Forum

Retrieving inserted and deleted text

#15

as long as the assumption holds true that any command run will affect text only besides a cursor, I don’t think a full diff would be necessary

  1. store the cursor positions when the selection changes
  2. when the document is modified, get the cursor positions and compare them to what was stored
  3. the changes will be between the differences, whether it was a deletion or an addition

i.e.

  1. cursor at line 1 column 2
  2. user runs a command to paste ‘hello\nworld’
  3. cursor is at line 2 column 5. between these 2 positions is the inserted text

obviously it becomes more complicated with multiple cursors but you can take into account the position differences of the cursors before the one you are analysing

or am I missing something?

2 Likes

#16

That sounds like it would definitely work for inserts… what about deletes? Is it possible to know when we have an insert or a delete?

All I need is the positions of the deleted characters because I will be storing a shadow copy of the insert events but I don’t know how I would differentiate between when code is being added and when it is being deleted. Perhaps I can look at the size of the buffer to tell whether there was an insert or delete???

1 Like

#17

I would just go with if the cursor begin position is before the previous end position, then there was a delete :slight_smile:

ofc, if there was a non empty selection and it has been replaced with something else (which could be the same size, bigger or smaller), it would be a delete and an insert. But you can identify this by comparing your shadow copy to the buffer between the old cursor position and the new to see what is different and where…

I might try to build a reusable proof of concept when I get chance :wink:

1 Like

#18

What about this scenario:

  • insert some text on line 10 (of 100 total lines) cursor position is on line 10 and whatever column
  • user goes to line 90 and deletes some code

In this scenario the new cursor position is after the latest insert, which would imply an insert but it is really a delete. I am also worried about replacing 10 characters of code with a different 10 characters- the size of the buffer is the same but there were some changes.

A proof of concept would be great if you have the time!

1 Like

#19

Also check out Modific

They have a pretty well functioning model that works asynchronously.

1 Like

#20

Nice, any reason why this works with lambas but not regular functions?   If I move self.view.run_command ( "timeout_insert" ) to its own function & call the function instead of a lamba, it just jumps to the final output without any delay.

When I changed it back to inserting @ the view, the edit object expired after the first loop.

I tweaked it a bit & got this:

 

import sublime, sublime_plugin,re

global string, stringIndex

class TestCommand ( sublime_plugin.TextCommand ):

	def run ( self, edit ):

		global string, stringIndex

		string = "Testing delayed output."
		stringIndex = 0

		for index in range ( 0, len ( string ) ):
			sublime.set_timeout_async( lambda: self.view.run_command ( "timeout_insert" ), 100 * index )

class TimeoutInsertCommand ( sublime_plugin.TextCommand ):

	def run ( self, edit ):

		global string, stringIndex

		self.view.insert ( edit, self.view.sel()[0].a, string[ stringIndex ] )

		stringIndex += 1
1 Like

#21

I took at shot at this using the ideas mentioned from @fico, @kingkeith, and I. I am not a python programmer so I may be doing some things incorrectly:

import sublime, sublime_plugin

class StorytellerListener (sublime_plugin.EventListener):

	def __init__(self):

		#do we need to call the base class constructor???

		#a shadow copy of the text in the buffer (this will be changed to event 
		#objects with timestamp, dev group, etc. later)

		#this object will have a 2D array for each buffer (keyed by the buffer id)
		#there will be an arry for each line and events will be added to a line
		#newlines will be added to the end of the line they are on
		#file 1:
		#123
		#456
		#789
		#[["1", "2", "3", "\n"]["4", "5", "6", "\n"]["7", "8", "9"]]
		self.allInsertEventsByFile = {}

		#stores the size of each buffer (keyed by the buffer id)
		self.allBufferSizes = {}

	def on_new(self, view):

		#all new files get an empty 2D array to hold insert events
		self.allInsertEventsByFile[view.buffer_id()] = [[]]
		
		#all new files start with a buffer size of 0
		self.allBufferSizes[view.buffer_id()] = 0

	#view is the changed file
	def on_modified (self, view):		

		#get all of the cursors (this only works with one)
		updatedRegions = view.sel()

		#go through each of the regions (usually only one- multiple regions if 
		#there are multiple cursors)
		for region in updatedRegions:

			#get the current size of the text inside the active buffer
			newBufferSize = view.size()

			#get the id of the buffer for a multiple file project
			bufferId = view.buffer_id();

			#holds the number of characters that were either inserted or deleted
			numCharsChanged = abs(newBufferSize - self.allBufferSizes[bufferId])

			#if the current buffer size is exactly the same as the last edit 
			if(newBufferSize == self.allBufferSizes[bufferId]):

				#I'm not sure when this happens!!!
				print("Size the same-- When does this happen????")

			#new buffer size is smaller than the old buffer size, this must be a delete
			elif(newBufferSize < self.allBufferSizes[bufferId]):

				#remove some data from the shadow container
				self.removeEvent(numCharsChanged, region.a, self.allInsertEventsByFile[bufferId], view)

			#buffer is larger, must be an insert
			else:
				self.insertEvent(numCharsChanged, region.a, self.allInsertEventsByFile[bufferId], view)

			#store the new buffer size for next time
			self.allBufferSizes[bufferId] = newBufferSize


	def printEvents(self, view):
		
		print("buffer id: %i" % view.buffer_id())

		#get the events for this buffer
		allEvents = self.allInsertEventsByFile[view.buffer_id()]

		#go through all of the rows
		for row in allEvents:

			#print each row
			print(row)


	def insertEvent(self, numCharsChanged, cursorPoint, bufferOfInsertEvents, view):

		#get the region with the new text
		insertedRegion = sublime.Region(cursorPoint - numCharsChanged, cursorPoint)

		#get the text from that region of the view
		insertedText = view.substr(insertedRegion)

		#go back to where the insert started and get the row and column
		beginInsertRow, beginInsertColumn = view.rowcol(cursorPoint - numCharsChanged)

		#used to specify where to insert
		insertRow = beginInsertRow
		insertColumn = beginInsertColumn

		#for each of the new characters, insert into the shadow buffer
		for newChar in insertedText:

			#if the new character is a newline
			if newChar == "\n":

				#get the rest of the current line so we can add it to the next line
				restOnLine = bufferOfInsertEvents[insertRow][insertColumn:]

				#insert the newline event where it occured in the current line
				bufferOfInsertEvents[insertRow].insert(insertColumn, newChar)

				#insert a new line and add the rest of the previous line  
				bufferOfInsertEvents.insert(insertRow + 1, restOnLine)

				#remove all of the characters we already added
				for x in restOnLine:

					#remove from the end of the line
					bufferOfInsertEvents[insertRow].pop()

				#start back at the beginning of the next line
				insertRow = insertRow + 1
				insertColumn = 0 
			
			else: #the new character is NOT a newline

				#if the insert is on a new line (happens after deleting all the 
				#chars on a line and then adding some back)
				if insertRow == len(bufferOfInsertEvents):

					#add the new line
					bufferOfInsertEvents.append([])

				#add the character event to the current line
				bufferOfInsertEvents[insertRow].insert(insertColumn, newChar)

				#move the column index forward
				insertColumn = insertColumn + 1						

		#print the new state of the events
		self.printEvents(view)


	def removeEvent(self, numCharsChanged, cursorPoint, bufferOfInsertEvents, view):

		#get the row and column of the cursor after the delete (at the beginning of the delete)
		deleteRow, deleteColumn = view.rowcol(cursorPoint)

		count = 0

		#start removing events, once for each of the chars deleted
		while count < numCharsChanged:

			#if we are removing a newline
			if bufferOfInsertEvents[deleteRow][deleteColumn] == "\n":
				
				#remove the newline from the end of the line
				del bufferOfInsertEvents[deleteRow][deleteColumn]

				#if there are more rows underneath
				if deleteRow < len(bufferOfInsertEvents) - 1:

					#grab the next line
					nextLine = bufferOfInsertEvents[deleteRow + 1]

					#add the characters from the next line to the current line
					for char in nextLine:
						bufferOfInsertEvents[deleteRow].append(char)

					#remove the next line since we copied it over
					bufferOfInsertEvents.pop(deleteRow + 1)

			else: #removing a non-newline

				#all removes happen at the cursor point since and events slide back
				del bufferOfInsertEvents[deleteRow][deleteColumn]

				#if the line is completely empty now AND it is not last line
				if len(bufferOfInsertEvents[deleteRow]) == 0 and len(bufferOfInsertEvents) > 1:

					#remove the empty line
					bufferOfInsertEvents.pop(deleteRow)
				
				#if we just removed the last character on a line
				elif deleteColumn == len(bufferOfInsertEvents[deleteRow]):

					#move on to the next row
					deleteColumn = 0
					deleteRow = deleteRow + 1

			count = count + 1

		#print the new state of the events
		self.printEvents(view) 

Currently, it only tracks new files. To run it, open up a new file using sublime and make sure the console is open.

It seems to work with single cursor inserts and deletes (typing and cutting and pasting). I am using the buffer size to determine if there was an insert or a delete. I am not sure when the file would change and the buffer size would be exactly the same size as before but it does appear to happen occasionally. Does anyone know when/why that happens?

I am also not sure about find/replace scenarios, multiple cursors, etc. If anyone with a lot of sublime experience can guide me that would be much appreciated.

2 Likes

#22

Lookin good :grin:

I messed around with it for a few minutes, didn’t figure anything major out but did manage to take out a traceback caused by allBufferSizes being uninitialized.

Code @ Gist

 



 
I wanted to figure out the multi-caret thing but need to get back to working on another project.

I think the math involving numCharsChanged & cursorPoint needs to be re-worked.   Currently it is lumping all of the modifications to a single cursor point.   In addition to numCharsChanged, offsets of each cursor need to be taken into consideration.
 



 
You should host this on GitHub.   I’d definitely follow it & contribute a bit when I have some extra time.

It would be fun to get that set_timeout_async script working as a ST player for your data files.

1 Like

#23

You don’t need to use lambda, but to call a function with arguments, you need to do something special to wrap the function call in another function, so that it will remember the parameters without immediately executing the function. i.e.

import sublime, sublime_plugin

global string, stringIndex

class TestCommand ( sublime_plugin.TextCommand ):
    def run ( self, edit ):
        global string, stringIndex
        string = "Testing delayed output."
        stringIndex = 0
        
        def run_timeout_insert():
            self.view.run_command ( "timeout_insert" )
        
        for index in range ( 0, len ( string ) ):
            sublime.set_timeout_async( run_timeout_insert, 100 * index )

class TimeoutInsertCommand ( sublime_plugin.TextCommand ):
    def run ( self, edit ):
        global string, stringIndex
        self.view.insert ( edit, self.view.sel()[0].a, string[ stringIndex ] )
        stringIndex += 1

or, more concisely, as the insert command is built in and wrapping a function (to “create a closure” I believe is the term) is a common operation, Python has a built in method for it:

import sublime, sublime_plugin, functools

class TestCommand ( sublime_plugin.TextCommand ):
    def run ( self, edit ):
        string = 'Testing delayed output.'
        
        for index in range ( 0, len ( string ) ):
            sublime.set_timeout_async( functools.partial(self.view.run_command, 'insert', args = { 'characters' : string[index] } ), 100 * index )

see this post on SO for more details

3 Likes

#24

My proof of concept, will log changes to the console: (EDIT: just edited it to work with more than one cursor)

import sublime, sublime_plugin

def get_cursors(view):
    return [cursor for cursor in view.sel()] # can't use `view.sel()[:]` because it gives an error `TypeError: an integer is required`

class RecordSessionListener(sublime_plugin.EventListener):
    prev_cursors = {}
    
    def on_new_async(self, view):
        self.record_cursor_pos(view)
    
    def on_activated_async(self, view):
        self.record_cursor_pos(view)
    
    def record_cursor_pos(self, view):
        cursors = []
        for cursor in get_cursors(view):
            if cursor.empty() and cursor.begin() > 0: # if the cursor is empty and isn't at the start of the document
                cursors.append((cursor, view.substr(cursor.begin() - 1))) # record the previous character for backspace purposes
            else:
                cursors.append((cursor, view.substr(cursor))) # record the text inside the cursor
        
        self.prev_cursors[view.id()] = cursors
    
    def on_selection_modified_async(self, view):
        self.record_cursor_pos(view)
    
    def on_insert(self, view, cursor_begin, cursor_end, text):
        self.log('insert', 'from', view.rowcol(cursor_begin), 'to', view.rowcol(cursor_end), '"' + text + '"')
    
    def on_delete(self, view, cursor_begin, cursor_end, text):
        self.log('delete', 'from', view.rowcol(cursor_begin), 'to', cursor_end, '"' + text + '"')
    
    def log(self, *values):
        print(type(self).__name__, *values)
    
    def on_modified_async(self, view):
        offset = 0
        for index, cursor in enumerate(view.sel()):
            prev_cursor, prev_text = self.prev_cursors[view.id()][index]
            prev_cursor = sublime.Region(prev_cursor.begin() + offset, prev_cursor.end() + offset)
            if not prev_cursor.empty() or cursor.begin() < prev_cursor.begin():
                self.on_delete(view, prev_cursor.begin(), prev_cursor.end(), prev_text)
            if cursor.begin() > prev_cursor.begin():
                region = prev_cursor.cover(cursor)
                self.on_insert(view, region.begin(), region.end(), view.substr(region))
            offset += cursor.begin() - prev_cursor.begin()

works with cuts, pastes etc. although it doesn’t seem to handle undo very well at the moment, it calls it the opposite of what it is… but should give you a clear starting point, you can just create a class that extends RecordSessionListener and make use of the on_insert and on_delete methods :slight_smile:

3 Likes

Clear non-saved state when changes are reverted manually
Count the no of times a user used the suggestions given by the autocomplete
#25

 
Awesome, thanks for clarifying!

The functools.partial method is too intense for me :dizzy_face: lol.   I like my whitespace & verbose declarations, keeps me from having to think too much while scanning through code.

1 Like

#26

I made a slight change to the on_modified this morning that should handle multiple cursors:

	def on_modified (self, view):		

		#get all of the cursors (this only works with one)
		updatedRegions = view.sel()
		
		#get the current size of the text inside the active buffer
		newBufferSize = view.size()

		#get the id of the buffer for a multiple file project
		bufferId = view.buffer_id();

		#if we are tracking this view
		if bufferId in self.allBufferSizes and bufferId in self.allInsertEventsByFile:
			
			#holds the number of characters that were either inserted or deleted
			numCharsChanged = abs(newBufferSize - self.allBufferSizes[bufferId])

			#MM- change here!!
			#if all of the changes are exactly the same length we can handle
			#them all
			numCharsChanged = numCharsChanged / len(updatedRegions)

			#go through each of the regions (usually only one- multiple regions if 
			#there are multiple cursors)
			for region in updatedRegions:
				
				#if the current buffer size is exactly the same as the last edit 
				if(newBufferSize == self.allBufferSizes[bufferId]):

					#I'm not sure when this happens!!!
					print("Size the same-- When does this happen????")

				#new buffer size is smaller than the old buffer size, this must be a delete
				elif(newBufferSize < self.allBufferSizes[bufferId]):

					#remove some data from the shadow container
					self.removeEvent(numCharsChanged, region.a, self.allInsertEventsByFile[bufferId], view)

				#buffer is larger, must be an insert
				else:
					self.insertEvent(numCharsChanged, region.a, self.allInsertEventsByFile[bufferId], view)

			#store the new buffer size for next time
			self.allBufferSizes[bufferId] = newBufferSize

This works only if all of the changes being added are exactly the same length (like in a multi-cursor situation). I take the difference in buffer size and divide by the number of cursors.Three instances of inserting the word ‘cat’ would have a difference of 9 characters. Divide by the three cursors and each cursor contributes 3 characters. Send the position of each cursor to the insert/delete function and it seems to work.

This will not work if unequal sized chunks of text are ever entered at multiple points by the tool. I’m not sure if that ever happens in sublime. Some other editors are more transparent about which changes are occurring.

I also checked and found that find and replace text of exactly the same length does not work… still thinking about how I might solve that problem.

Yes, if I end up going with sublime I will put it on github soon. I am still checking out a few other editors.

1 Like

#27

Very nice!

1 Like

#28

Nice!   Glad I asked what you were using that code for, this is a really cool project :smiley:

1 Like

#29

@fico @kingkeith I decided to go with visual studio code for my more complete version. Sublime is awesome but I wanted to reuse what I had in js. Here is the project page: https://github.com/markm208/storyteller#storyteller

I am curious for your input!

3 Likes

#30

wow, that is really cool! I’m exclusively an Sublime Text user, so I haven’t tried the plugin itself, but the playback mode is awesome! great work! I’m jealous that you didn’t write the storyteller extension for Sublime Text :wink:

1 Like

#31

Works great, nice job man! :grin:

 

Suggestion:

  • Add an icon-based UI group for controls

 

The first time I tried it out, I couldn’t tell that there was a speed setting. Then when I skimmed through the documentation, I noticed the keyboard commands.

A control panel can add visual indication of the current speed, and IMO looks better & is more intuitive than text buttons (paired with hover-tooltips for context).

 

Here are a few mockups showing potential states of said UI:
#@Imgur

 


 

Edit:

I didn’t notice the StepBackward & StepForward buttons when I made the mockups, so here’s an adjusted one:

1 Like

#32

Thanks for posting this. How did you discover those sublime_plugin.EventListener.on_insert, on_delete methods? This is not documented, is it?

1 Like

#33

the on_insert and on_delete methods are not a built in part of sublime_plugin.EventListener - I had to do some coding wizardry to determine what text was inserted and deleted and where - if you look closely, the code I posted uses on_selected_modified_async and on_modified_async together to determine this information - the idea being someone could extend my RecordSessionListener class which has already done all the hard work :wink:

1 Like

#34

Hey thanks for the answer. I should have seen that. Didn’t pay enough attention.

1 Like