Sublime Forum

CodeMap - interactive code tree plugin

#1

Hi all, just wanted to share my ST3 plugin for showing interactive code map for the content in the active view: https://github.com/oleg-shilo/sublime-codemap.

It currently supports building the code tree for Python syntax. Support for C# is in the pipeline. You can also add custom parsing algorithm to build the code tree for any alternative syntax.

Enjoy.

6 Likes

Acecool Code Mapping System - previously XCodeMapper - Developmental Name - Nearing Version 1 - addition to [ CodeMap by oleg-shiro ]
#2

Looks pretty cool, but I am wondering why don’t you use the symbols as parsed by ST to build the map - if the syntax definitions are all consistent enough (which they should be for color scheme purposes) then you could support all languages at once without any extra work :wink:

1 Like

#3

Yes it is an attractive option. However it would mean that the code tree would reflect not the content itself but its decoration (highlighting). And while indeed this approach would cover all languages it would be limited by the nature with no room for extendibility.

Instead of this, I wanted to give a completely working infrastructure with the default mapping that anyone could integrate with a custom tree building algorithm even for the content that has no associated highlighting.

The plugin comes with:

  • Python mapper. Very simple (~70 lines) indent based mapper that deliberately limits mapping depth to 2-levels of nesting. While syntax highlight would go all the way, producing extra mapping noise.

  • Text mapper. A sample for building a pure content based mapper, which has no associated highlighting. This mapper builds the map of text lines. Similarly one can adjust it to map the paragraphs or logical sections (header vs footer)

  • C# mapper. A comprehensive mapping that goes beyond regex analysis and relies on true Type System analysis (Roslyn). This mapper as an adapter only (integration layer) and will fully function only when I release CS-Script plugin (in a week or so). This plugin is a port of Notepad++ plugin for C# intellisense and C# script execution.

Anyway, ideally I see CodeMap as a shell, which is used by others to integrate other custom (e.g. domain specific) mappers. One of the best examples of which would be a universal mapper based on your idea - “the symbols as parsed by ST” with a possibility to switch to dedicated mappers via settings when required.

I would be happy to include any mappers contributed by the community directly to the package.

A mapper is an incredibly simple generic Python routine that takes a file path as an input, analyeses the file content and returns a map text where each line represents a map item in the following format:

<item title>:<item position in source code>

This is the mapper sample: https://github.com/oleg-shilo/sublime-codemap/blob/master/custom_mappers/code_map.txt.py

0 Likes

#4

This plugin is great, but it has a problem with some python files not showing the map (try commands.py from sublimelinter package)

Also, I prefer a more compact view, I added these in the codemap.py (class show_code_map) to make it look better

code_map_view.settings().set("line_numbers", False)
code_map_view.settings().set("gutter", False)
code_map_view.settings().set("fold_buttons", False)
code_map_view.settings().set("font_size", 8)

Maybe you could add some configurable settings for how you’d like the map to look. It would be even cooler if it created the map on a new vertical group of its own, instead of adding the tab to the current group.

0 Likes

#5

An excellent suggestion.

  • Not needed visual - removed
  • Individual column placement made default and settings controlled.

The release is published: v1.0.6

And I will have a look at parsing problem. Thank you for your feedback.

0 Likes

#6

Fixed too. It was just the lack of utf-8 support on reading the .py file

0 Likes

#7

Very cool, I also like that it closes the pane if you close the map, not so obvious :).
I still can’t get the map for that file (my other UTF-8 files show the map). Also, about my suggestion I found out later that disabling just gutter would also disable line numbers and fold buttons, so maybe it’s better just

code_map_view.settings().set(“gutter”, False)

so one can enable/disable them in one go, if preferred.

Thanks and good work!

0 Likes

#8

…it closes the pane if you close the map…

Yes I thought it would be a good user experience. It’s all controlled by these settings:

{
    "close_empty_group_on_closing_map": true, 
    "show_in_new_group": true
}

I decided that maintaining font size different to other views is a bit an overkill. But the rest makes perfect sense.

…disabling just gutter…

Will consider putting this one into settings file too. Currently hiding is hard-codded.

As for that “naughty” codemap.py, can you just sent it to me. May be mine somehow is different to yours. I will have a look at its encoding.

0 Likes

#9

This looks very similar to something I’ve seen in another editor. It appears to only map for the current file. Would it be readily extensible to following structure across several files?

0 Likes

#10

The plugin is extensible, meaning that you can easily supply your custom routine (mapper) that would parse a file (or files) and produce the map. It actually comes with the mapper for MARKDOWN syntax. Thus showing the multi-file map is not a problem at all.

Navigating to the clicked item is a challenge.

The current implementation simply opens the document that is associates with the active map and navigates to the line with the same number as the last word in the clicked map item (line). Hooking into on_click event is not a problem. I can extend it and the custom handler can be taken from the same custom mapper module.

However, if you want to achieve multi-file model you will need to think how to encode multiple documents paths into the map item. This is something that you will need to solve if you are keen to get it working.

0 Likes

#11

Thank you. There is another outliner style plugin that I discovered when pointed to a feature request for permitting API access to the sidebar. Once I’m finished with my current plugin development phase, I may take up the lance and attempt to extend this for a full source tree.

0 Likes

#12

Thanks, this is similar to what I’ve been looking for…

I’ll need to do some reading in order to create an option for GMod Lua - does is the code_map..py where syntax is supposed to be the extension or the name… ie I have it as Lua but the syntax is GMod Lua - and from what I can see you load the language file ( and I pointed to that ).

Bookmarked…

For those having trouble

This is what I have so far for adding GMod Lua to CodeMap:

Note: For some
user\codemap\custom_languages\lua.sublime-syntax:

     %YAML 1.2
---
# http://www.sublimetext.com/docs/3/syntax.html
name: CodeMapGModLua
file_extensions:
  - lua
scope: text
contexts:
  main:
    - match: \b(break|do|else|for|if|elseif|return|then|repeat|while|until|end|function|local|in|continue)\b
      captures:
        1: keyword.control.lua

  functions:
    - match: \b(function)\s+([a-zA-Z_.:0-9]+[.:])?([a-zA-Z_]\w*)\s*(\()([^)]*)(\))
      captures:
        1: keyword.control.lua
        2: entity.name.function.scope.lua
        3: entity.name.function.lua
        4: punctuation.definition.parameters.begin.lua
        5: variable.parameter.function.lua
        6: punctuation.definition.parameters.end.lua

Basically reading the Lua.tmLanguage and altering the format to match the sublime-syntax style… Not done yet, but it is a start. and I’m using code_map.py.py with a few changes for now… renamed to code_map.lua.py

There are a lot of improvements that could be made the the mapper system - ie you have the python one hardcoded, just as the md one is hardcoded… It could be altered to create a nice output by using classes, etc… This way you can really control the output and you wouldn’t need to create the same script for each and every language ( unless you want to do something different… )

But, as it is, with some work this looks like what I’m looking for… If I create a dynamic mapper, I’ll probably submit it to the github project or post it here… Then you can define the language in python by calling the class and a few additions, then you don’t have as much code being repeated…

0 Likes

#13

When you are happy with you mapper you can share it and I can include it into the default CodeMap package.

0 Likes

#14

Sounds good - I’m making a lot of changes - this one will be for GMod Lua, although it’d work for Lua too - I’m slightly altering how you have it set up, helper functions / classes, etc…

Also, because I created my own coding standard, I have predefined variable names which always mean something specific so I replace _p with _p as helpful output in the function args list, for example…

Things like that will be easily definable…

I may add a different system for adding, ie supporting categories ( so instead of everything added to one single table - added to table[ category ][ type ] or simply table[ category ][ entry key ] = entry or similar with sorting options…

Additionally, because I use “namespacing” in my Lua code the output is quite large…
ie:

Classes / function Acecool.C.
  function Acecool.C.timekeeper:New( <N> _duration, <Timestamp> _start, <N> _delay, <*> _mode ):150
  function Acecool.C.timekeeper:FromTable( <*> _data ):201

would automatically remove the prefix so they’d show up as:

Classes / function Acecool.C.
  function timekeeper:New( <N> _duration, <Timestamp> _start, <N> _delay, <*> _mode ):150
  function timekeeper:FromTable( <*> _data ):201

to shorten the data being displayed…

The goal in the actual mapper script is to get rid of all of the repetitions:

				# -- This is here so I can have a separate category for functions defined in my namespace sub Classes rather than it be under functions..
				if code_line.startswith( 'function Acecool.C.' ):
					last_type = 'function Acecool.C.'
					last_indent = indent_level
					_entry = ( line_num,
							'Classes / function Acecool.C.',
							line.rstrip( ),
							indent_level, 1 )

				# --  Different category than above
				elif code_line.startswith( 'function ' ):
					last_type = 'function'
					last_indent = indent_level
					_entry = ( line_num,
							'function',
							line.rstrip( ),
							indent_level, 1 )

				elif code_line.startswith( 'local function ' ):
					last_type = 'local function'
					last_indent = indent_level
					_entry = ( line_num,
							'local function',
							line.rstrip( ), #
							indent_level, 1 )

would become

   SetupNewWatcher( string_search_for = 'function Acecool.C.', string_category = 'Classes / function Acecool.C.', boolean_strip_search_pattern_from_line = 1, string_substitute_replaced_search_pattern_with = 'function' )

or something similar… ie all of it becoming 1 line definition ( and then that defs file could be read from, in the following order fall-through: Project Root Folder > User Language Folder > User CodeMap Folder > Default CodeMap Folder > Default Language folder - or similar…

which would make using this plugin that much easier for people that aren’t familiar with python, or other languages but use ST3 for other purposes, etc…

I have this project on high priority because it’ll increase my efficiency on the other projects I’m working on so you can likely see a post soon…

Oh, question… At the top of the code_map..py file, you declare ignored, installed, and map_syntax but they aren’t used anywhere in the file… Do you use it in another file? I haven’t looked through the entire plugin yet - but I’m guessing you do use it to load certain data…

Current example of output: https://www.dropbox.com/s/oejte8c9vjed26y/sublime_text_unregistered__screenshot_20170914003027_7648.png?dl=0

0 Likes

#15

Is there a way to hide the minimap for the CodeMap group? My function definitions with args are longer than I’d like the group to be - and I’ve done things to shorten it for example the primary classes group Acecool.C. is removed by adding it as a category and nested components are indented… But I added data-types to the arguments field ( although I may alter that too )…

Additionally… Is there a way to add icons ( specific ico files or png etc… ) to the gutter, or to the file output? And, since the gutter is hidden, is there a way to enable it? I have yet to look at the main plugin code, so it could all be right in front of me but since you created it, it’d take less time for you to simply reply than it would with me familiarizing myself with it…

The mapper is almost ready - I’m still putting a few things in place ( for example in GLua there are realms - ie where code is executed… Clientside or Serverside or shared / both which is typically known by if statements but some also add the designation to the file name or folder-name… Right now for REALM switches in the context of the code I add a new category and inject additional data before the prefix so if specified in a file, it is known for those lines )…

I still need to add built-in support for detecting localized functions and so on and use a special character to show it belongs to a parent instead of simply being tabbed - I will probably shorten the realm system too so it doesn’t take up important real-estate by having the category show the realm and have the same L char pointing that it is owned by a parent… I am going to re-add Alias support but possibly integrate them into the output where the original definitions are which would mean changing the storage system from the original mechanism… among other things…

Here’s an example of the output:


Basic idea of alias support

0 Likes

#16

I am a bit confused about your post. CodeMap is designed in such a way that it is completely decoupled from the mappers. Thus the level of consistency (or inconsistency) between coding styles in both a mapper and plugin is completely irrelevant.

The same comes for your preferred naming conventions. And the ability of people to understand CodeMap internals.

The plugin extensibility model is extremely simple. You don’t need to know anything about CodeMap but just implement an analyzer for the syntax of your choice, with a very simple set of responsibilities. That analyzer is expected to produce a list of the file “bookmarks” in the form of <bookmark_name>:<line_number>. That’s it. Nothing more nothing less.

Feel free to format the whole entry as you want or to control the overall width of the entry. You can even place some visual separators:

Category A
  function_1: 20    
  function_2: 21

Category A
  function_1: 30    
  function_2: 31

-----------------

Globals
  function_1: 40    
  function_2: 41
 

All this is up to you and all this has no affect on CodeMap as double-clicking in the map view will ignore any decoration line and only navigate on clicking a valid bookmark entry <bookmark_name>:<line_number>.

May be I misunderstood your post but I interpreted it as a prompt for changing rather the actual plugin implementation than the developing Lua mapper. Please correct me if I am wrong.

0 Likes

#17

Forgot…

Is there a way to hide the minimap for the CodeMap group?

Yes, I would live to be able to do that. Though don’t know if it is possible.

Additionally… Is there a way to add icons ( specific ico files or png etc… ) to the gutter, or to the file output? And, since the gutter is hidden, is there a way to enable it? I have yet to look at the main plugin code, so it could all be right in front of me but since you created it, it’d take less time for you to simply reply than it would with me familiarizing myself with it…

This goes well beyond the intended functionality of the plugin. Sorry.

The ST3 philosophy is to keep views extremely simple and to prevent developers from creating any complex (non-Text) views. I personally would prefer ST to allow me to do complex views as I can do for VSCode:

This way CodeMap would be much better suited for the purpose, but… we have what we have. Despite multiple requests ST team does not want to allow custom views or custom toolbars. And I am not sure it’s gonna change.

That’s why any attempt to mimic comprehensive visual experiences with pure text is going to be extremely costly. And, just considering a simple “effort vs. benefit” ratio makes me think that evolving CodeMap visual complexity is very unlikely.

Ironically the CodeMap itself is an attempt to mimic a TreeView with a pure text. Yes it works but there are only so much that can be done. Just compare it to my CodeMap for Notepad++. It’s so much better visually:

1 Like

#18

The minimap can only be hidden on a per-window basis.

2 Likes

#19

Thanks - I’m aware of the

<bookmark name>:<Jump-To-Line>

format - and the mapper I’m writing is a simplified way to add search / replacers and methods to format the output to make it cleaner… Think - a mapper could be made as a definitions file which includes a base mapper - ie

# Create the Categories we want to show ( Category Name, Category-Wide Actions to be applied to the entry, other flags )
<ENUM / Key> CATEGORY_FUNCTIONS = CodeMap.AddCategory( "Functions", { LINE_STRIP_RIGHT }, { CODEMAP_FUNCTIONS_CATEGORY } );
<ENUM / Key> CATEGORY_ENUMS = CodeMap.AddCategory( "ENUMeration" );

# Set the order in which the categories show in the output
CodeMap.SetCategoryOrder( CATEGORY_ENUMS, CATEGORY_FUNCTIONS );

# Other Config

# Indentation will be 2 spaces despite what is used in the file
CodeMap.SetIndentationChars( '  ' );

# Should we hide functions, etc... declared within other functions or show them indented?
CodeMap.HideNestedDeclarations( False );

# How should we display nested declarations? Examples of using the CharacterMap tree characters to clean up the look - Obviously there will be more config options, ie which char to show when it is the last in the list, the first, or in-between, and which to show when there are others ie double-line with 1 line going to it and the left one going underneath and so on...
CodeMap.SetNestedDeclarationsChars( ' ∟' );
CodeMap.SetNestedDeclarationsChars( ' ┗' );
CodeMap.SetNestedDeclarationsChars( ' ╚' );
CodeMap.SetNestedDeclarationsChars( ' ╙' );
CodeMap.SetNestedDeclarationsChars( ' ╘' );

# May set up args to look like: ( FirstWithoutOthers, FirstWithOthers, Center, End ) which would look like:
CodeMap.SetNestedDeclarationsChars( '└', '├', '├', '└' ); # But it could be shorted to Multiple, End ie the last 2 which would work with end being first without others and end, multiple being first with others and center meaning others...

# Adding a way to match code to a category ( Category Key, Match String, Match Start of Line or end of line or whatever ENUM so MATCH_LINE_REGEX MATCH_LINE_STARTSWITH MATCH_LINE_ENDSWITH, Flags varargs which can be MATCH_ or it can be things such as STRIP_LINE_LEFT STRIP_LINE_RIGHT STRIP_LINE LINE_TO_LOWER LINE_TO_UPPER ) or ( Category Key, Matching Text, Matching Options as a single option or an array, Entry Options as a single option or an array )
CodeMap.AddCategoryMatch( CATEGORY_FUNCTIONS, 'function ', MATCH_LINE_STARTSWITH, { LINE_STRIP_RIGHT, LINE_STRIP_MATCH_TEXT ( This will remove function  from the entry although this option could be applied to the category add code } );

# So we know to skip things in comments - more specifically to skip things in block comments where string.startswidth would be "fooled"...
CodeMap.AddCommentLineMatch( '//' )
CodeMap.AddCommentLineMatch( '--' )
CodeMap.AddCommentBlockMatch( '--[[', ']]' )
CodeMap.AddCommentBlockMatch( '/*', '*/' )

Etc… In short, what I am doing is making it easier to create mappers for new languages - right now it doesn’t seem you can create the lua.sublime-syntax and have it do the work for you - it seems like you need to create the mapper… And the mappers, right now, is a lot of repetitive code… if code_line.startswith: Add it elif re.find … elif blah blah blah…

What I am doing simply makes CodeMap available to a wider audience, quicker than would otherwise it’d be available to them because it saves them the time of creating a long / elaborate or repetitive code mapper by letting them make a few simple function calls and options to define or by making it easier for you to add languages…

Additionally, because there are built-in markers to detect when you’re in a comment-block, for example, it won’t wrongly match functions within those blocks and it has built in functionality to enable those features ie what do you want to do when you come across a nested function… or what do you want to do for nested whatever… what about classes… etc… In the category definitions function you simply define the behavior…

Everything to make things easier - so far I am enjoying CodeMap…

IE some basic features in my eyes would be a way to sort the function list by name, ensure anything categorized is held together so if functions is a category name, and enumerations is another, then you can always force all enumerations to display at the top ( I code in that way, but if someone else doesn’t and they want to show that category at the top, then it’d be available to them )…

Things of that nature is what I’m doing… Also the data-typing for languages which don’t have strict declarations in the function declaration - and for things that do such as C++ if a declaration is made at the top, and the real function is at the bottom then they can optionally be hidden or shown differently than in the functions category… etc…

A few questions: Is it possible to have the CodeMap in a separate window ( I prefer to have more room to code and I’ve shortened a lot of the definitions quite a lot [ Although I’ll probably add other keywords so function can be shortened to func and the rest of the code be properly highlighted, and so on to get everything shorter ] but it isn’t short enough - When I move CodeMap to a different window it isn’t updated and when I move it back, it’s fine )…?

In Notepad++ I had my functions list ( and the snippets panel ) on my right monitor docked to the left side of the monitor, and on the left monitor I had my file-tree docked to the right side, and so on…

This is one of the main reasons not being able to have separate panels in ST3 is such a drawback - but if you know a way where it can be updated in a separate window, that’d be fantastic ( to simplify behavior, the last file opened / selected, as it currently is, would be what CodeMap displays if it has a mapper for it )…

I agree - being able to have panels would increase efficiency as sometimes complex panels are needed… Having code analysis graphs generated would be nice, ie how much of the file size is from spaces, tabs, strings ( which should be in a language definitions file ), etc…

We’re incredibly limited - but ST3 is so much faster than Notepad++ because of the properly done threading system, and pre-indexing projects and so on… But panels are a requirement… They need to be added… because being limited to one window for your project is hell because the file-tree can easily take up 25 to 50% of a 1920x1080 monitor, then CodeMap can take up 15 to 50% easily ( I’m trying to shave off as much as possible while providing all of the necessary information I need at my fingertips ) and not being able to limit the search / replace thing at the bottom makes stretching the window across multiple monitors impossible and non-usable - if CodeMap and the file-tree could be extended all the way to the bottom, then using multiple monitors for 1 window wouldn’t be that bad but panels would solve all of these issues…

I would switch to Atom but it has the same problems as Notepad++ so it’s slow and so many things were done incorrectly…

This is the current lua.sublime-syntax in user/codemap/custom_languages/ for GMod Lua - a few custom things marked Acecool can be removed along with the _ variables ( Should be everything starting at line 258 AcecoolDev_Framework Namespace

But even if the language is defined, it doesn’t seem to do anything - but again I haven’t looked through the package code to see what is done with it…

0 Likes

#20

Thank you, I will try to find the time to go through your code with the attention it deserves. Though, did you also want to share the mapper?

And the mappers, right now, is a lot of repetitive code… if code_line.startswith: Add it elif re.find … elif blah blah blah…

Agree 100% :slight_smile: And it is exactly why I am reluctant to see any “code parsing” in CodeMap regrdless how good or bad it is. I want to push it completely to the mappers. Even if the parsing is done in some generic and very dev-friendly way. My argument is a pore IoC reasoning.

I would rather have some generic parsing util/module available for mapper creators instead. But it needs to be completely decoupled form the CodeMap codebase, even if they are distributed together. Though, as I said, I do not want to dismiss it completely without giving it a chance.

As for panels, you will not find a bigger supported then me. After creating quite a few plugins for Visual Studio, VSSCode, Notepad++ and ST3 I struggling to see how such a beautifully crafted creature, that ST3 is, can continue dismiss the request for custom panels support so arrogantly.

Is it possible to have the CodeMap in a separate window…

I understand and fully agree your motivation for this feature. I think it makes sense to make this option available. Can you please log the feature request on GitHub: https://github.com/oleg-shilo/sublime-codemap

0 Likes