LaTeXTools: Smart auto completion of "cite[tab]" to insert Bibtex cite key

[Wheeler]

I have tried the same tex file and the same bib file (the one without the #), and I am still getting the same behaviour with authors cut off after the line break.

This also seems to be independent of the bibtex backend, same behaviour for bibtex and biber.

I am using Sublime Text Build 4126, Stable Channel, with recent installs of LaTeXTools and LaTeXYZ, running on MacOS 12.3.1 Monterey.

It seems that the program parsing the .bib file stops reading after a line break.

If this would only be a display problem, I’d accept it. One could argue that truncating the list of authors even makes sense, to have more space for the title. However, since the authors are truncated I also cannot search for names of 2nd, 3rd, … authors, which is really a problem because often I do not remember the name of the 1st authors, but that a paper was co-authored by someone, and thus this makes LaTeXTools on Sublime Text not well usable for me…

Does anyone have an idea where this parsing of the bibtex file happens, so that I can check the code there?

[rcopat]

I wonder if it is having issues reading line endings in MacOS to display in the auto-complete panel. This is the last thing I would try. Go to Preferences -> Settings, and add "show_line_endings": true,. Open the BibTeX file. On the status bar, you should see “CR”. Click on it and select “Windows”. Save the file. Try the \cite{} command one more time.

[Wheeler]

Thanks for all the time you have invested. I tried it, the status bar first said “Unix Line Endings (LF)”, I tried both Windows (CRLF) and MacOS (CR), each time saving and even restarting Sublime text, but with no effect, the issue remains.

However, i have found something else. When using Bibtex, the parsing seems to happen in the file

˜/Library/Application Support/Sublime Text/Packages/LaTeXTools/bibliography_plugins/traditionalBibliography.py

And this parsing seems to happen line-by-line (see line 66 of the code, “for line in bib_data:”).

In line 94 there is even a comment:

# Note: we capture only the first line, but that's OK for our purposes.

Unfortunately, I disagree and this is not ok for my purposes. Or actually the purposes of anyone else that wants to search for other authors than the first author, and that is using Bibtex files using line breaks in the authors field. Note that all Bibtex entries from DBLP, which are most widely used in Computer Science, are of this form, here is a random example: https://dblp.org/search/publ/bibtex/?q=stream%3Astreams%2Fconf%2FecAgents%3A.

This is really a major limitation of LaTeXTools for Sublime Text, which forces me to consider switching back to Emacs. Unfortunately, because otherwise I like Sublime Text, but not being able to search for non-first authors is a major issue for my work.

I am still wondering why it works for you, @rcopat.

I noticed that there is another parser in file

˜/Library/Application Support/Sublime Text/Packages/LaTeXTools/bibliography_plugins/newBibliography.py

which parses Bibtex files in a different way and which seems not to have this issue. However, this other parser seems to be a different plugin and is not used on my machine, not even when using the example LaTeX files provided by @rcopat. Is there a way to configure LaTeXTools to only use the new parser in newBibliography.py?

[Wheeler]

Ah, found it. I can set

// "new"				a newer parser which supports more complex
//						formatting and additional fields, but may
//						be slower
"bibliography": "new",

in the LaTeXTools user configuration. However, in this case slower means 30+ seconds for each \cite command, and even though the parsers seem to buffer the parsed file this doesn’t help, I still have to wait for more than 30s until the selection panel shows up. This is unusable.

Very unfortunately, this means that unless something surprising happens now, I’ll cancel my Sublime Text and -Merge subscription and return to Emacs, until this issue is resolved. In Emacs Bibtex files are parsed correctly and instantly, without any noticable waiting time.

Thank you very much @rcopat for helping me to figure out the cause of this issue, this is highly appreciated!

[Wheeler]

For what it’s worth, here is a fix that replaces the code of class TraditionalBibliographyPlugin(LaTeXToolsPlugin) in the traditionalBibliography.py file. However, it is a quick and dirty fix, probably not very reliable, and of course needs to be fixed manually again with the next LaTeXTools update, which is why this is not a convincing solution to me. However, others might find it helpful, therefore I am sharing it here:

def get_entries(self, *bib_files):
    print("Here!!")
    entries = []
    for bibfname in bib_files:
        bib_cache = bibcache.BibCache("trad", bibfname)
        try:
           cached_entries = bib_cache.get()
           entries.extend(cached_entries)
           continue
        except:
           pass

        try:
            bibf = codecs.open(bibfname, 'r', 'UTF-8', 'ignore')  # 'ignore' to be safe
        except IOError:
            print("Cannot open bibliography file %s !" % (bibfname,))
            sublime.status_message("Cannot open bibliography file %s !" % (bibfname,))
            continue
        else:
            bib_data = bibf.readlines()
            bib_entries = []

            entry = {}

            i = 0
            while i < len(bib_data)-1:
                i += 1
                line = bib_data[i-1]
            # for line in bib_data:
                line = line.strip()
                # Let's get rid of irrelevant lines first
                if line == "" or line[0] == '%':
                    continue
                if line.lower()[0:8] == "@comment":
                    continue
                if line.lower()[0:7] == "@string":
                    continue
                if line.lower()[0:9] == "@preamble":
                    continue
                if line[0] == "@":
                    if 'keyword' in entry:
                        bib_entries.append(entry)
                        entry = {}

                    kp_match = kp.search(line)
                    if kp_match:
                        entry['keyword'] = kp_match.group(1)
                    else:
                        print(u"Cannot process this @ line: " + line)
                        print(
                            u"Previous keyword (if any): " +
                            entry.get('keyword', '')
                        )
                    continue

                # Now test for title, author, etc.
                # Note: we capture only the first line, but that's OK for our purposes
                multip_match = multip.search(line)
                if multip_match:
                    key = multip_match.group(1).lower()
                    value = codecs.decode(multip_match.group(2), 'latex')

                    if key == 'title':
                        value = value.replace(
                            '{\\textquoteright}', ''
                        ).replace('{', '').replace('}', '')

                    if key == 'author':
                        while value[-4:] == " and":
                            i += 1
                            value += " " + bib_data[i-1].strip()
                        if value[-2:] == "}," or value[-2:] == "\",":
                            value = value[:-2]
                    entry[key] = value
                continue

            # at the end, we have a single record
            if 'keyword' in entry:
                bib_entries.append(entry)

            print ('Loaded %d bibitems' % (len(bib_entries)))

            try:
                bib_cache.set(bib_entries)
                fmt_entries = bib_cache.get()
                entries.extend(fmt_entries)
            except:
                traceback.print_exc()
                print("Using bibliography without caching it")
                entries.extend(bib_entries)
        finally:
            try:
                bibf.close()
            except:
                pass

        print("Found %d total bib entries" % (len(entries),))
    return entries

[rcopat]

You are right. I have in my settings "bibliography": "new",. When I comment it out, I can replicate the issue. I am glad you were able to make it work with the plugin.