Sublime Forum

Faster find in files with ripgrep

#1

I have generally been happy with the find in project search speed, but recently my work repo must have grown significantly, because searches are now taking a very long time. It shows that it’s searching 15,429 files, and it’s taking around 5 seconds.

When I do the same search using riprgrep, it takes less than half a second. I’m sure it’s not a fair comparison because it’s not doing the same filtering in both cases, but I’m confident that using ripgrep would be universally faster.

Is there any chance of switching or making this configurable? Thanks!

1 Like
#2

It should be noted that ripgrep’s goal is to be fast, inasmuch as they don’t support multiline regex, lookarounds, etc. It all depends on what your doing. The fastest search isn’t always better if it doesn’t do what you want it to do, but I did find the write up of comparing regex engines interesting: http://blog.burntsushi.net/ripgrep/.

I’d say, if your happy with ripgrep, keep using it on the side of the editor. There’s nothing wrong with using a separate search tool that fits your needs better.

1 Like
#3

That’s true, it doesn’t support more advanced regex. If we need to support that, then there are other options like sift.

I strongly disagree with your suggestion to use ripgrep outside of the editor. The reason we have things like find in files and the build system is so we don’t have to keep jumping over to a terminal to run something. Having to switch apps, run a search, find a match, and then navigate to the file in Sublime is the height of inefficiency. The point of my request was to point out that the existing find feature seems to be significantly slower than alternatives, so it’s worth considering them.

0 Likes
#4

That’s fine, but as someone who uses an external search tool, I’ll agree to disagree.

In some ways, but I’ve always found project search lack luster. Using the same line for filters as the location is cumbersome. Replace across multiple files opens each file in a tab unsaved. I’m just not a fan. But single file search is great.

As someone who feels Sublime is the best editor out there, I’m also not afraid to notice its shortcomings, and project sort is meh. For me, it isn’t the speed that is the downside. You could make it super fast, and I’d still find the project search awkward.

I’ve always been of the mind use the best tool for the job. Though I do diff files in Sublime, I also use external diff tools for serious diffing. Sublime can’t be the best at everything, but it is best at editing.

1 Like
#5

I know there is a plugin: Search in project, which (should) allow to use external tools to search, inside Sublime. It currently supports ack, pt, ag, and more.

The prolbem is that I found some issues in using it (at least, with the latest Sublime version…), and I would prefer a search result the same way Sublime presents it (new window, with all the search results).

2 Likes
Plugin for RipGrep
#6

Yeah, since it is a cli tool, I can see it would be a little awkward to jump straight to file. You’d have to either modify an existing plugin to fix/add the support you want, or add your own.

I think Sublime took the right approach with engines though. You may not please everyone, but they picked a decent capable one that a lot of people should be familiar with. Had they picked a niche one like ripgrep, I suspect they would get a lot of complaints about why feature X doesn’t work in the search bar. Sublime themselves realize that specialized regex engines are good for tailored tasks as they use their own for syntax parsing. But when you’re talking about a general purpose user facing engine, boost is pretty decent.

0 Likes
#7

The search functionality within Sublime Text uses some API features to perform jumping to results. So it shouldn’t be too hard for an enterprising soul to create a package to run ripgrep and display the results in a a buffer or output panel like the build in Find in Files does.

If you are interested, you could check out the source of the file Default/exec.py to see how the file/line regex patterns are applied to the view settings to get next/prev result and double-click actions to work.

2 Likes
#8

Yeah, I might investigate making a plugin, though I don’t have a lot of time at the moment :unamused:. I was just hoping that it might be possible to swap out the existing implementation, which would benefit everyone. Maybe ripgrep was a bad suggestion if it’s important to support advanced regex, but there’s definitely something out there that’s faster.

0 Likes
#9

@wbond I can’t find that file anywhere in my Application Support/Sublime directory. Where should it be?

0 Likes
#10

It’s packed inside the Default.sublime-package that ships with Sublime and isn’t directly visible.

The easiest way to look at it would be to use PackageResourceViewer to open it.

If you’re using one of the recent development builds you can also select View Package File from the command palette to look at it.

1 Like
#11

Awesome, thanks!

0 Likes
#12

create a package to run ripgrep and display the results in a a buffer or output panel like the build in Find in Files does

… which makes sense only, if such a package could use the Find And Replace panel frontend. Don’t think it makes much fun to toggle the options using a quick panel.

0 Likes
#13

In response to those saying that it doesn’t support advanced searches… It doesn’t matter… Because you can have more than 1 search algorithm running or usable…

If the search is basic - use the fastest algorithm possible… If the search is advanced - use the fasted algorithm possible… If one happens to use the one suggested in this thread and the other uses a different one - the default perhaps - then it’ll still be a bonus…

It’s no different than using BubbleSort with QuickSort and other sort algorithms to make something faster because BubbleSort is faster than QuickSort for SMALL samples… I did a paper a while back showing the gains - I combined 3 separate algorithms ( Quick, possibly merge? and Bubble ) and instead of something taking say 10 seconds, it took 3 ( Used on nodes with many different tests and sorting options and it was always such a huge gain on speed ).

Granted that example isn’t exactly what this would be doing because in mine One to all algorithms are used based on the sample-size - with this only one search algorithm would be used at any given time… But still - if it provides a gain in speed then it is worth looking into… Overhead shouldn’t be a problem…

0 Likes
#14

When we’re talking about replacing the default regex engine, it does matter. But if you’re talking about using a plugin with different algorithms, then it doesn’t matter.

It’s not practical for Sublime to replace it’s regex with multiple engines. It is also confusing to people who don’t know the difference about what they are using. But, if you put it in a plugin for people who do know what they are getting in to, then it’s fine.

The original question talked about replacing the current by default, which is much different than a plugin allowing you to swap out different engines. It was never about ripgrep being bad, just not a suitable default.

2 Likes
#15

Issues of which regex engine aside, I am surprised your searches are so slow. I regularly search a mixed C++/python project containing 8000 files in a fraction of a second (on a fairly fast iMac with SSD and plenty of RAM). So I’m suggest spending a bit of time on diagnostics to see if you can improve the situation on your computer. Some thoughts:

  • Check your exclusion list to make sure object code and such isn’t also being searched.
  • Do you have enough free space on your drive, and plenty of RAM?
0 Likes
#16

I’m talking about a switch which occurs in the back-end to use the fastest search engine based on what is available… ie if using the engine from the thread starting post makes sense - use it… if using the default system is faster then use that… It really isn’t difficult to code it in the back-end…

Requiring the user to make that switch manually wouldn’t help things where-as an automated selection would.

0 Likes
#17

Again, that isn’t trivial. Sublime imports boost regex and is coded to use that. It would be pain to support multiple regex engines on the backend from external processes. If I was coding Sublime, I wouldn’t do that either. I’d tell the community to write a plugin. Sublime provides a decent general purpose regex engine, and honestly, that is all I expect from them.

Now could they improve the plugin API to give plugins better control of the search panel? Yes, that could be cool. If they allowed plugins to completely override the search panel’s buttons and provide a way to populate and control all the input boxes, then you could create a native feel with your external regex process of choice. I wouldn’t expect them to waste time creating a plugin(s) to specifically setup and use all popular external engines; their time is better spent elsewhere.

0 Likes
#19

Apologies for digging this up, but I’d just like to add that ripgrep has added support for multi-line searches now.

0 Likes
#20

It also has an optional pcre mode with full forward and back reference support. It also has a backing library that I think could be used directly rather than by shelling out. I think all the earlier concerns have been addressed.

0 Likes