Building with Make, Regex and F4

[Lozminda]

@OdatNurd, thanks once again. I’m gonna have to let that stew a bit as on first reading it’s gone a bit over my head…I shall ponder…ATB

[Lozminda]

Some thoughts (if I’e understood you correctly):

That means when I compile with the verbose option, the compiler/linker is outputting messages to a different working directory depending on whether the verbose flag is set ?

I’ve got to find out where make (and maybe some other processes) send their output too (???) and change the working dir accordingly (???)

Or am I totally off piste here ?

L

[Lozminda]

Thanks, nice to be appreciated ! (even if it is for moaning )

[OdatNurd]

That means when I compile with the verbose option, the compiler/linker is outputting messages to a different working directory depending on whether the verbose flag is set ?

Here’s an example:

{
	"shell_cmd": "g++ \"${file}\" -o \"${file_path}/${file_base_name}\"",
	"file_regex": "^(..[^:]*):([0-9]+):?([0-9]+)?:? (.*)$",
	"working_dir": "${file_path}",
	"selector": "source.c++",

	"variants":
	[
		{
			"name": "Run",
			"shell_cmd": "g++ \"${file}\" -o \"${file_path}/${file_base_name}\" && \"${file_path}/${file_base_name}\""
		}
	]
}

The file_regex here is what allows Sublime to detect the file, line, column and error message. The only things that are going to match are lines that match this regex. Anything that doesn’t match is entirely ignored.

So, vitally important step #1: if you want to navigate between errors, this regex needs to be able to match them. If passing extra command line arguments to the tool you’re running changes the way the error messages are presented, then you also need to change this, or it’s not going to match, and you’re not going to be able to navigate.

Now, presume an error message like this:

/home/tmartin/test.c:8:13: error: expected ‘;’ before ‘}’ token

The name of the file is absolute; it is always possible to know exactly what file on my hard drive is the one that triggered the error.

On the flip side:

test.c:8:13: error: expected ‘;’ before ‘}’ token

Ok, a file named test.c has an error. Which one? There can be arbitrarily many of them. I could be some sort of freak with a test.c in every folder on my entire hard drive. Which one is it referring to? Sublime can’t know, it’s not the thing that generated the error message, the external tool is. This is the inherent problem with relative paths. They’re relative to some known location, but what is that location?

	"working_dir": "${file_path}",

This part of the build system says “while you are executing this tool, as a first step please change the current working directory to be the path in which the currently active file is stored”.

Given this, Sublime can presuppose that if it has a relative filename, it has to be relative to something, and the only information it has is the directory that was currently active while the tool was running. So, it will stick the relative filename onto this working directory path and see if that opens the file.

Vitally important step #2: If you don’t set the working directory, OR the tool that you’re executing changes it, AND the error message is a relative filename, there’s no way to find and open the file.

The import of the my original statement is something like this:

(master *%>) tmartin:dart:~/local/src/dragonWarrior> make
make[1]: Entering directory '/home/tmartin/local/src/dragonWarrior/external'
make[2]: Entering directory '/home/tmartin/local/src/dragonWarrior/external/lua'
make[2]: Nothing to be done for 'install'.
make[2]: Leaving directory '/home/tmartin/local/src/dragonWarrior/external/lua'
make[2]: Entering directory '/home/tmartin/local/src/dragonWarrior/external/mersenne'
make[2]: Nothing to be done for 'install'.
make[2]: Leaving directory '/home/tmartin/local/src/dragonWarrior/external/mersenne'
make[2]: Entering directory '/home/tmartin/local/src/dragonWarrior/external/md5'
make[2]: Nothing to be done for 'install'.

This build is recursive; every time it goes into a folder, it’s changing what the currently active directory is. If your build does that, AND your tool outputs a relative filename like hello.c, then you’re screwed and you can’t open the file because Sublime has no way to know what file it could possibly be; the external tool changed information out from under it.

[Lozminda]

Thanks, A Fantastic Reply, it’s a bit late where I am, I’ll get into in the next day or two…Awesome., great work Mr Martin !

L

[Lozminda]

Some cross over from this post

I notice that you have this as you working dir:

	"working_dir": "${folder:${project_path:${file_path}}}",

Would that fix the issue I’m having in this post I wonder…

Ok I think I’ve understood (think being the word here)

I saved this project I’m testing in the sae folder as the source code.

Here’s output from building using clang and the build file:

As you can see F4 works. Notice the error format WLM007.cpp:37:4: error: blar

Here’s the build file:

	"cmd" : ["clang++-3.8 -std=c++14 -stdlib=libc++ -lSDL2 -lSDL2_image -lgmpxx -lgmp -O02 -pthread -w ${file_name} -o ${file_base_name}"],
"selector" : "source.c",
"shell": true,
"file_regex":"^(|..[^:]*):([0-9]*):?([0-9]*)?:? (.*)$",
"working_dir" : "$file_path"
}

So now building with the makefile:

The error format is the same as above, so I’m guessing that means your first point isn’t what’s happening here:

00000000000000000000000000000000000000000000000000000000000000000000

The name of the file is absolute;

to

So, it will stick the relative filename onto this working directory path and see if that opens the file.

00000000000000000000000 END QUOTE 000000000000000000000000000000000

Here’s the build file for the makefile:

{
  "cmd" : ["make --file=makeOdatNurd.mk -B TARGET=\"$file_base_name\""],
  "selector" : "source.c",
  "shell": true,
  "file_regex":"^(|..[^:]*):([0-9]*):?([0-9]*)?:? (.*)$",
  // "working_dir" : "$file_path"
  "working_dir": "${folder:${project_path:${file_path}}}",
  "syntax": "Packages/Makefile/Make Output.sublime-syntax"
}

The make file makeOhDatNurd (hope you don’t mind !) is in the same directory as the source code. I thought changing the working dir to the above might help. It’s given me some nice colouration…And I thought the syntax might help…

Surely if ST is displaying the error info ? There clearly something I’m missing. I’ve read your post several times, I believe I understand whats going on… Ha ha belief.

Cheers, Loz
Ps I changed

“working_dir”: “${folder:${project_path:${file_path}}}”

to (and I then get no error messages and F4 doesn’t work as opposed to error messages and F4 doesn’t work…)

“working_dir”: “${project_path:${file_path}}}”,

[Lozminda]

How d you get make to printout which directory it’s “going into” ? The makefile I’m using is in the same directory as the source code and all the object files too. But just in case.

Also there’s something else that makes me think it isn’t reason 2 you gave:
If I keep pressing F4 nothing happens in the panels where my code is until I get to the end and then it opens this funny dir tab with nothing in it…

[OdatNurd]

How d you get make to printout which directory it’s “going into” ?

My make is doing that automatically because my Makefile looks something like this:

install clean release::
    @cd external && $(MAKE) $@

Building with this Makefile tells make to go inside of the external folder and recursively call make again to do the build in there (and exernal also has a Makefile in it). Since make knows that it’s invoking itself again, it tells you when it’s entering and leaving folders like this so you can track what it’s doing.

For a build that’s not doing this, the most expedient thing would be to add pwd (or if you’re on windows, cd) as a command under the make rule, which should display the current folder while that rule is executing.

[Lozminda]

I’ve printed out the directory from the makefile, it’s the same as the source file and error messages, so it remains a mystery !
(I split my recipe with ; pwd)

[OdatNurd]

As a ridiculously simple test;

test.c

---

#include <stdio.h>

int main (int argc, char **argv)
{
    printf("Hello, World!\n")
    return 0;
}

Makefile

---

all: test
	./test

test: test.o
	clang -o test test.o

test.o: test.c
	clang -c test.c

Lozminda.sublime-build

---

{
  "shell_cmd" : "make",
  "selector" : "source.c",

  "file_regex":"^(|..[^:]*):([0-9]*):?([0-9]*)?:? (.*)$",
  "working_dir": "${folder:${project_path:${file_path}}}",
  "syntax": "Packages/Makefile/Make Output.sublime-syntax"
}

The folder structure has the Makefile and test.c sitting in the same folder; executing the build with the build you have here produces this output:

clang -c test.c
clang -o test test.o
./test
Hello, World!
[Finished in 0.1s]

All as we would expect. Now, remove the semicolon on the printf and build again:

lang -c test.c
test.c:5:30: error: expected ';' after expression
    printf("Hello, World!\n")
                             ^
                             ;
1 error generated.
make: *** [Makefile:8: test.o] Error 1
[Finished in 0.0s with exit code 2]
[shell_cmd: make]
[dir: /tmp/bob/test]
[path: ... ]

An error is detected by clang, which reports it. Also because make returns a failure, Sublime displays a diagnostic that tells you what command it executed, what directory was the current directory, and the path at the time (which I have clipped just because my path is excessively long).

Seems like things work on the face of it, based on the output.

The file_regex is what tells Sublime that a line contains a result to be navigated. Yours looks like this:

^(|..[^:]*):([0-9]*):?([0-9]*)?:? (.*)$

The first capture group is the name of the file:

(|..[^:]*):

So, the filename is either any two characters (..), followed by 0 or more characters that aren’t a colon ([^:]*), followed by a :. OR, due to the alternation operator, the filename can be nothing at all.

Then, there’s this output:

[dir:

So, [ matches the first ., and d matches the second ., and then ir matches [^:]*, the next character is : which closes off the capture group, and what was captured was [dir as a filename.

What you expect? No. What you told it was acceptable? yep.

Your regex looks like it’s based on the one from C Single File.sublime-build (or one of my examples perhaps, which come from there), except that the filename field has the alternation in it.

That one is meant to parse gcc output, but clang produces different output than gcc does. I’d try something like this:

  "file_regex":"^(..[^:]*):([0-9]*):([0-9]*): (.*)$",

Similar to the one you’re using, but the row and column for the error are now not optional (and the filename isn’t allowed to be empty), which should match a bit better based on my sample of this case right here.

[Lozminda]

I’ll give it a go, just to say the regex I’m using works on clang fine, it’s when error come via clang via a makefile that I’m having trouble…

Also if I build with the same makefile using a different compiler option (g++) F4 works.
If just compile with clang++ (no makefile) F4 works
If i use a makefile which compiles with clang F4 doesn’t work

Same results as above with new regex…

[Lozminda]

The folder structure has the Makefile and test.c sitting in the same folder; executing the build with the build you have here produces this output:

Up to here I’ve done everything you say, test.c and make as you descrided in the same folder. When I hit build

make: *** No targets. Stop.

Maybe this is a clue ?

[OdatNurd]

Worked when I did it; you may want to make sure that your paste is using tabs for the rules; some versions of make get cranky about space indented rules and this is the message you see when that happens.

[Lozminda]

Yep checKed them

tried cmd “make all”. The error is make: *** No rule to make target `all’. Stop.

Bizarely I get an error banner in my make file.

I’ll try again with out make all…

No joy. Here’s a screen shot of both make and test.c in the same dir

[Lozminda]

Have done some jiggling of my own, we have landed, there’s an error F4 works…

  "shell_cmd" : "make --file=make.mk",
  "selector" : "source.c",

  "file_regex":"^(|..[^:]*):([0-9]*):?([0-9]*)?:? (.*)$",
  "working_dir": "${folder:${project_path:${file_path}}}",
  "syntax": "Packages/Makefile/Make Output.sublime-syntax"

and obvs changed the name of the make file to make.mk…

It seems to be when clang runs through make it’s initial output before it gets to the errors is slightly larger than when it’s not running through make…

What does this text highlighting signify (in the commandline ouptut panel) ?

Everything but the last three lines is highlighted. The highlighting is different depending on make and compiler options…

[OdatNurd]

Can I ask why you keep making the capture for the filename include an empty capture? Are you trying to capture lines that start with :line:col or something?

[Lozminda]

Morning/Evening. When you say capture, I’m assuming you’re talking about the regex ? I’m not trying to capture anything. The regex’s that I’ve used are only the ones (I think it’s just you) have suggested.
I don’t know anything about regex. Why I keep sending you a screenshot is that the highlighted pattern of text is different depending on what build system (make,g++,clang++ or combinations) i use (even though the regex is the same except for the different ones you suggested in this post) thus if I/we knew what the purpose of the highlighted text was (because I’m not highlighting it ST3 is) it might give us some clue as to what was going on…? (Or not, I have no idea how ST3 works, again I could take stab, but that’s it)
So far our diagnostic tools aren’t working I thought maybe look at some edge cases. Now I could be barking up the wrong tree completely, which was why I wanted to know what the text highlighting is in the “error box” (sorry don’t know correct name for it couldn’t find it).
If this is pointless so be it. What is the text highlighting showing ?

Ta

[OdatNurd]

The file_regex is used to match lines that have error/message output, and in order to do so it has to capture the information out of the line. In particular, at a minimum it needs to capture the name of the file, or it can’t open it. Then, if it also captures a line, it can open the file and go to the correct line. If it also captures a column number, it can focus on the error directly, and if it ALSO captures the message text, it can display it to you.

What a regex is and how they work is out of scope for the conversation here, but the parts of the regex wrapped in () are the captures; they tell the regex engine to store the text that was matched for use later.

One of the constructs in a regex is the | which means “or”. So for example, a regex of this|that will match either the word this OR the word that. In your regex above, you have (|..[^:]*): which is meant to match the filename. However it starts with a | which means “match anything, OR whatever this second match is”. Unsurprisingly, when given the choice to match nothing, there’s a lot of places in the file to choose from that satisfy that.

I’m pretty sure that regex didn’t come from me (if it did please trace where I said it so that it can be fixed). As I mentioned above, that’s almost certainly not a regex that’s going to work for you for clang (it matches incorrectly here even in the simple test I outlined above), while the fixed regex I provided works for clang just fine.

[Lozminda]

Clicking on Build Errors doesn't work & other stuff

@OdatNurd thanks for you reply to my last post. I thank you here because as I’m going to use one of your videos to help elucidate (maybe) my issue. Also I looked at this post: At about 10.40 ish in this video: the host (@OdatNurd) says if you get an error in your code you can double click on it. You’ll get a marker in your code too. I get neither, I noticed a few days ago, but I’ve finally got round to finding and checking out the vid. Am I missing something ? L

Re reading that post it looks like I might hve got it from one of your vids…
BTW the regex with the ‘|’ has worked for the last 18 months, it’s only since using make to build files has that regex stopped working (and agin it does work with make and g++). TBH I don’t think it’s the regex as I’ve replaced the old regex, with the one you’ve suggested…

Cheers

[OdatNurd]

If I put a broken regex in one of my videos I’d love to know about it; if you happen to run across it, please let me know. If I had any regex related to this in a video, it should have been copied directly from the C Single File.sublime-build file, which does not look like that.