Sublime Forum

Sidebar with large file tree crashes (maybe not indexing related)

#1

Hi,

I am facing the infamous “high memory usage” / indexing huge folders. I’ve read many (if not all :P) online solutions and am familiar with the case, but this looks different.

When a folder with many files is open, sublime mem usage grows “indefinetly” (~215k files, hitting 40GB mem until OS crashes it).

  • "index_files": false is active
  • Problem happens even when subl --safe-mode
  • "folder_exclude_patterns" do fix the issue, but with undesired effects: I really need to browse through the files, source drivers, etc (eg. linux kernel work).
  • Help -> Index status always shows idle
  • Console log shows no corrupted file signal.

I’m on a i7 13th gen, 32GB RAM (128 GB swap to debug this), fast ssd, etc… 215k of kernel/rootfs isn’t that huge (VS Code handle it easily… with many plugins, goto x, etc)

Maybe it’s something not really related to indexing, but instead the display of the tree in the sidebar? When the folder is not listed there, the issue is gone.

The folder is a nvidia jetson rootfs and similar:
https://developer.nvidia.com/downloads/remk-463r32releasev73sourcesubuntubionic-l4tldeaarch64srctbz2
https://developer.nvidia.com/downloads/remetpack-463r32releasev73t210jetson-210linur3273aarch64tbz2

that has links (maybe in loop?) that could be causing this?

I don’t mind if the indexing took long once in a while… but it seems to not even be running;
and sublime always crashes after ~3 minutes, so it’s unusable.

Other settings that I tried:

  "show_git_status": false,
  "index_files": false,
  "index_exclude_patterns": ["*.*"],
  "show_git_status": false,
  "index_exclude_gitignore": false,
  "goto_anything_exclude_gitignore": true,
  "ignored_packages": ["Vintage", "Diff", "CSS", "HTML", "ShellScript"], 
  "folder_exclude_patterns": ["Linux_for_Tegra/rootfs", "ubuntu_bionic-l4t_lxde_aarch64_src", "ubuntu_bionic-l4t_lxde_aarch64_src/usr", ".svn", ".git", ".hg", "CVS", ".Trash", ".Trash-*"],

If any of those Linux folders is removed from ignore and appear on the sidebar tree, the issue is triggered.

Anyone has any idea how can I try to fix this?

Thanks

Sublime Text 4 build 4143 
Linux - Fedora 38 - 6.2.15-300.fc38.x86_64 - Gnome 44.1 - X11
Installed from the stable development repo (dnf)
0 Likes
#2

Downloading to attempt to reproduce

sublime_text:
  build: 4150
  channel: dev
  platform: 
    system: MacOS
    arch: arm64
    portable: false
  env_vars: '/bin/zsh -l'
system_info:
  hardware: MBP 14inch 2022, 10-Core M1 Max, 64 GB, 512GB, Apple M1 Max
  software: macOS 13.3 (22E252)
  kernel_version: Darwin 22.4.0
  open_gl_context_information:
    gl_api_version: 4.1 Metal - 83.1
    glsl_version: 4.10
    vendor: Apple
    renderer: Apple M1 Max
0 Likes
#3

I did another test here, on another huge folder tree.

image
Basically the linux kernel cloned inside itself, multiple times.
That’s 15x more in file count, 10x total size and deeper trees.

Sublime works fine. Memory consumption is stable around 300-400MB, opens and navigates deep into the tree fast, no indexing take place, etc.

Also, the issue is not there for Jetson-210_Linux_R32.7.3_aarch64.tbz2;
only for Tegra_Linux_Sample-Root-Filesystem_R32.7.3_aarch64.tbz2

Steps to reproduce:

curl -L -o Tegra_Linux_Sample-Root-Filesystem_R32.7.3_aarch64.tbz2 https://developer.nvidia.com/downloads/remeleasev73t210tegralinusample-root-filesystemr3273aarch64tbz2
mkdir rootfs
sudo tar -xjf Tegra_Linux_Sample-Root-Filesystem_R32.7.3_aarch64.tbz2 -C rootfs

sudo is required to proper set some special file permissions in rootfs. When extracted without sudo it works fine (tree opens fast, mem usage stable at ~500MB)

This makes me guess that some sort of link/hardlink is being created with special permissions and causing some sort of recursive loop in the tree parsing?. Mem usage grows at +1GB/s in that case, until sublime crashes.

Strangely enough I’ve navigated through other rootfs’ses using sublime; a fairly common task in embedded work.

0 Likes
#4

Is there any way to prevent sublime from following links in the sidebar tree?

Maybe one relative link in the rootfs tree is causing an infinite recursion.

image
image.png1199×553 84.9 KB

0 Likes
#5

I confirmed that a recursive link exists in that file tree: output/Linux_for_Tegra/rootfs/var/run/host/

ll output/Linux_for_Tegra/rootfs/var/run/host 
lrwxrwxrwx. 1 root root 3 ago  8 08:33 output/Linux_for_Tegra/rootfs/var/run/host -> ../

fzf works fine, but the folder above points to my /, that eventually points again to the same folder and so on.

There should be a way to not follow links when displaying the sidebar file tree.

0 Likes
#6

ST has loop detection both by reading symlinks and by fingerprinting directories. Perhaps there’s an issue due to the link being relative.

0 Likes
#7

@bschaaf due to the link being relative.
Could you explain how a relative link could be different than a regular link?

Isn’t most “regular” links relative?
Ex:
lrwxrwxrwx. 1 root root 5 jul 26 21:00 yum -> dnf-3

Relative to the current folder

0 Likes
#8

I can’t reproduce this with the “Tegra_Linux_Sample-Root-Filesystem_R32” folder that you were able to - I’ve also extracted using root. Seeing stable memory usage around 60MB and no loops detected on build 4156.

0 Likes