Porting Word citations to latex with \cite{}

Alastair Heffernan

188BET靠谱吗I was trying to convert a word document with citations created using the Zotero plug-in to a Latex format that I could just copy-paste across.

I opened an arbitrary .csl file just to see if I had a chance of doing it.

My question: is there a "name variable" in that style editor that would allow me to put the BBT "Citation key" field in the in-text citations (for reference, other examples are things like "author" and "title" and "editor" etc, there's a whole list here page 30: https://media.readthedocs.org/pdf/citation-style-language/1.0-20100321/citation-style-language.pdf)?I tried loads of things to see if I got lucky.I have a feeling this is impossible but thought I'd ask.

One possible solution I don't know how to implement is to batch convert the citation keys of BBT to an unused field that is common to most formats, e.g.short title or something like that - which I could then reference in the csl file.Any ideas?

Cheers.

adamsmith

Maybe @emilianoeheyns188BET靠谱吗, who created BBT, has an idea, but I'm not aware of any way to do this until Zotero has proper citekey support:
- BBT citekeys are not mapped to CSL
188BET靠谱吗- There is no way to access them via Zotero's server API that I know of (though this is the part that I may be wrong about)
188BET靠谱吗- There is no reasonably accessible local API for Zotero

188BET靠谱吗If there is a way to access them via the server API, it would be possible to copy them over to a different field using pyzotero
188BET靠谱吗https://github.com/urschrei/pyzotero

Alastair Heffernan

Thanks for the response.I figured it was a lost cause.Got to transplant my phd thesis from word to latex in three days so not got time to investigate much myself - but if anyone does find out how to do this I'm sure it would be very useful in the future...

adamsmith

188BET靠谱吗I'm expecting citekeys in Zotero 5.1 (or 6.0 depending on versioning) and then this will be trivial.Until then any solution will be rather involved.

bwiernik

My recommendation for your in your current situation would be to convert your citations/bibliography to regular text (with the Unlink Citations button in Word), then use Pandoc ( https://pandoc.org/) to convert the Word to LaTeX.188BET靠谱吗AS adamsmith says, there isn't currently a very reasonable way to get from Zotero's Word citations to a stable BibTeX citekey.

emilianoeheyns

BBT citekeys are not mapped to CSL

But they can be via a BBT trick:
188BET靠谱吗https://retorque.re/zotero-better-bibtex/configuration/#citeprocnotecitekey

188BET靠谱吗There is no way to access them via Zotero's server API that I know of (though this is the part that I may be wrong about)

This is correct.BBT keys are not available via the API, nor can they be made available.188BET靠谱吗BBT relies on a running Zotero to do its work

188BET靠谱吗There is no reasonably accessible local API for Zotero

BBT does have some APIs, but I'm not sure how you'd want to use it.

edit: ugh it would be so nice to have markdown editing here.

bwiernik

Wow.That's incredibly helpful.

emilianoeheyns

The post-edit is probably a lot more helpful than the single link I accidentally submitted while meaning to preview :)

laurence80386

Sorry to revive this thread but I too am trying to work in the space between Word and LaTeX...

Given that BBT has citekeys, is it not then trivial to create a CSL that wraps the references in '\cite{' and '}'?I notice the 'BibTeX generic citation style' *almost* does it, but just spits out the citekey instead of wrapping it.

Am I right in thinking that's all it would take, or am I missing something?188BET靠谱吗Does one even need BBT for this, given Zoteros built-in citekeys are sufficient for most purposes (the changes of a conflicting author/year are quite slim)?

adamsmith

188BET靠谱吗You do need BBT because Zotero doesn't have built-in citekeys, so the citekey in the CSL style and in Zotero's exported bibtex isn't necessarily the same.

You could indeed easily modify the existing style, yes.BBT actually offers an option linked on the description of the citeproc functinoality:
188BET靠谱吗https://retorque.re/zotero-better-bibtex/installation/preferences/hidden-preferences/#citeprocnotecitekey
also not quite what you want, but likely easier to work with then the regular CSL one.

laurence80386

Ah great, thanks @adamsmith

188BET靠谱吗One further question -- if I receive a document from someone containing Zotero references to sources I don't have in my local library, is that a problem?(I'm trying to create a publishing workflow for a journal, and I won't obviously have all the sources that authors use).

adamsmith

yeah, that won't work well, I'm afraid.
In that case you'd have to just work with the CSL Bibtex style entirely, which is not without its flaws.
188BET靠谱吗To get high-quality bibtex, you need the items in Zotero (from which you can then export them) but they'd have a different internal ID and wouldn't be linked to the citations in Word.

emilianoeheyns

Would the reference extractor be capable of extracting data of sufficient quality to build a toolchain around?

adamsmith

the data quality is fine, but I can't think of a way to update the citation metadata in the document?You'd have to do:
- extract data
188BET靠谱吗- import into Zotero
- add citekeys
- replace existing citations with newly imported ones
- Apply bibtex citation style with BBT citekeys

It's the penultimate step that I think would be very tricky to do.

emilianoeheyns

Yeah, you're right, that would easily get into fuzzy matching and that kind of ugliness.Not a great fan other than for tech proof of concepts.

BTW wrt @laurence80386

the changes of a conflicting author/year are quite slim

when you say "putting together a workflow for a journal", do you mean you writing for a journal, or you working at a journal and managing the influx of documents?Because if the latter, at some volume "slim chances" become near-certainties.

laurence80386

I'm meaning publishing.Basically we're (notionally) accepting both Word and LaTeX docs, but I want to use LaTeX for typesetting so need to copy the Word documents' text into LaTeX for that purpose -- which means using BibTeX to ensure the links between references and the bibliography are automatically generated.

Once the PDF is camera-ready the originals can be archived!the only place where a conflict is possible is within a single document, and if Smith has two publications in 2009 (i.e.two separate instances of \cite{smith_2009} then that can be detected at proofing (I guess?)

laurence80386

188BET靠谱吗For more context (again sorry for hijacking the thread...) I'd like to receive Zotero-based Word files, convert the CSL to BibTeX (including the '\cite{...}'), copy that text into LaTeX, then compile the PDF.Does that sound plausible?

My main concern (but there may be others) is if I don't have the references in my local library -- do they travel with the Word document, in at least sufficient detail to be able to change CSL for the purposes of the workflow I've just described?

emilianoeheyns

the only place where a conflict is possible is within a single document, and if Smith has two publications in 2009 (i.e.two separate instances of \cite{smith_2009} then that can be detected at proofing (I guess?)

Not sure that's correct.An article could cite smith_2009 twice, or cite two separate smiths that both published in 2009, or one smith that published two articles in 2009.

188BET靠谱吗the references do travel with the Word document (which is why the ref extractor can extract them) and you could, while extracting, save the item ID or key (I think one of them is in there) in the extra field somehow, replace the cite in the doc with \cite{key or item ID}, and then stitch that together somehow (whether that goes through Zotero or not is a different matter).

As an author, I'd be pretty nervous that either I or someone not-I would have to check that all cites remain in order, that the required sentence-case-to-title-case translation has been done properly (not a trivial matter to automate, trust me), some people post-edit the in-doc cites to get parencite-equivalent output, CSL and bib(la)tex don't have a perfect mapping between fields which means compromises or even data loss during conversion...

adamsmith

188BET靠谱吗the references do travel with the Word document (which is why the ref extractor can extract them) and you could, while extracting, save the item ID or key (I think one of them is in there) in the extra field somehow, replace the cite in the doc with \cite{key or item ID}, and then stitch that together somehow (whether that goes through Zotero or not is a different matter).

Getting the item ID out of the citations is actually going to be quite messy.It is indeed in the document, but it is not typically extracted by the reference extractor (we're referring to this app: https://rintze.zelle.me/ref-extractor/) so that would require even more scripting.But on the positive side, all citeable data is included in the document and can be used to switch between citation styles.

Given what you describe, I'd start by simply testing out two different and easy-to-implement workflows:
1.Completely rely on CSL, including for the formatted bibtex.This will correctly match citekeys and bibtex, but the bibtex itself may have problems.Modifying the existing CSL style to include the cite command will indeed be trivial.
2.188BET靠谱吗Rely on CSL (again, modified) for the cite command in text, the use the ref extractor to import into Zotero and export using Zotero's (or BBT's) bibtex, set to just include author_year as the citekey.This is more likely to mismatch citation and key, but will give you better/correct bibtex.

If either of them is good enough for you, you're set with minimal effort.If you need something that performs better, you can follow some of Emiliano's thoughts further -- I think with those, it may be possible to get to a 100% reliable solution (at least for all standard item types) but that'll require much more significant up-front scripting.

laurence80386

@emilianoheyns

Not sure that's correct.An article could cite smith_2009 twice, or cite two separate smiths that both published in 2009, or one smith that published two articles in 2009.

I think the first of these problems isn't really a problem, particularly since I'm using ACM referencing (which is just a number referencing an item in the bibliography, e.g.'[23]').Multiple references to the same bibliographical item are simply multiple instances of this, i.e.multiple insertions of `\cite{smith_2009}`.188BET靠谱吗You're right that there's a potential conflict if Zotero's author_title_year key applies to two items, but that seems extremely unlikely does it not?Two authors (or the same author) would need to have the same name, have written about the same subject, within the same year.The likelihood seems remote to me.Even when the same author does have >1 publications in the same year, I'm finding the default title key is reasonable, e.g.`hildebrandt_smart_2015` and `hildebrandt_radbruchs_2015`.

But maybe I am missing something...?

laurence80386

@adamsmith

Unfortunately your first option (relying entirely on BibTeX) won't work because we're soliciting submissions from both computer science and the humanities and the latter simply won't be willing to learn how it works, unfortunately.

As to your second suggestion:

2.188BET靠谱吗Rely on CSL (again, modified) for the cite command in text, the use the ref extractor to import into Zotero and export using Zotero's (or BBT's) bibtex, set to just include author_year as the citekey.This is more likely to mismatch citation and key, but will give you better/correct bibtex.

Can I clarify the process you're suggesting:

(i) use a BibTeX CSL to format the citations in the document once received from the author (they will use ACM while writing!I will convert to BibTeX for publishing)
(ii) extract the citations from the document with the ref extractor to get a BibTeX file
(iii) copy the body text into LaTeX and create the relevant separate BibTeX file
(iv) link the two and compile

I'm not sure what you meant about including author_year as citekey.188BET靠谱吗Can this be chosen in Zotero's bibtex?I notice the default is author_titlekeyword_year, as per the examples I gave above -- I presume, per the above, that I can stick with the default?

188BET靠谱吗Is there any need for step 2 (extracing the refs and importing them in Zotero)?If I insert a bibiography at the end of the authored document, this will already be in BibTeX, which I can then copy and paste into the bibtext file in LaTeX.Is that right?(This is why I was asking if the authored Word document includes all the metadata about the inserted references, without which this step will of course not work -- but it sounds from what you're saying that this will not be a problem).

188BET靠谱吗Strange request, but in order that I can test this, would one of you mind uploading a Word document with Zotero references somewhere on the web?This will let me emulate receiving an independently-authored document with references that have no connection with my own personal library, to see if the above process is feasible.

Thank you for your help so far!!

emilianoeheyns

I can't help with a Word document -- I don't use Word unless I absolutely must (and I am one of those rare exceptions who writes mostly in humanities, but uses latex for everything).

WRT extracting bibtex using the ref extractor -- I have looked at what I think is the source of the ref extractor, and I *think* it uses the bibtex-csl style ( https://github.com/citation-style-language/styles/blob/master/bibtex.csl) to convert the CSL to bibtex, but if that is correct, it turns an item with title

JR3: paper not in a group & other

into

@article{Author,
title={JR3: paper not in a group & other},
}

and that, aside from the problem that it doesn't title-case the title, simply isn't valid bibtex.And I don't see how CSL could do a better job than this, given what I know about converting text to valid latex and what little I know about what CSL stylers can do to the text.

188BET靠谱吗For conversion to valid latex, you're looking at either passing it through Zotero, or the closest competition I know, astrocite (even though astrocite doesn't pass my full test suite, and does not title-case).

laurence80386

@emilianoheyns

That's not my experience with BibTeX CSL -- I get things like this:

@article{rawls_two_1955,
title = {Two {Concepts} of {Rules}},
volume = {64},
issn = {0031-8108},
url = { https://www.jstor.org/stable/2182230},
doi = {10.2307/2182230},
number = {1},
urldate = {2019-02-13},
journal = {The Philosophical Review},
author = {Rawls, John},
year = {1955},
keywords = {website-bibliography},
pages = {3--32}
}

which looks pretty comprehensive, no...?

adamsmith

My option 1) was rely entirely on CSL, not rely entirely on Bibtex.

What you have in your example, though, is not Bibtex produced by the CSL style.188BET靠谱吗It's Bibtex exported by Zotero (which is generated by fairly elaborate javascript.It's absolutely crucial that you understand that distinction because it is at the heart of what makes this difficult.

One of the problems of using the CSL style is, in fact, that you can'tuse short titles for the citekey, since CSL can't modify individual elements and you'd end up with spaces in the citekey, so you're stuck with just author(s)_year and you'd have to adjust the export (by either using BBT or modifying the javascript) accordingly.This also, of course, makes citekey issues much more likely.

adamsmith

I would recommend that rather than doing this in the abstract, you spend 2hs or so actually experimenting.You'll likely run into many of the issues we're trying to raise here, and it'll be much easier to discuss if and how they can be solved once you have a clearer sense of how this would work for you.Getting to a set-up with a couple of Word document to test with shouldn't take you very long, so we don't have to do all of this as a whiteboard coding exercise.

laurence80386

Apologies @adamsmith, I misunderstood your previous message.Yes I think I need to do some more experimentation to better understand what's happening under the hood -- I'll create a 'clean' file with external citations and see what happens (and doesn't happen) when I follow the procedure I have in mind.

emilianoeheyns

As @adamsmith188BET靠谱吗points out, that's not produced by the CSL style, so either I'm mistaken that ref-extractor uses the CSL style (it could technically use an online Zotero translator server, but I see no evidence of that, and it claims everything happens in the browser), or you are indeed passing it through Zotero.

But {Two {Concepts} of {Rules}},is still very likely wrong -- it's much more likely that it should be {Two Concepts of Rules},.This may sound nit-picky (OK, it *is* nit-picky), and I don't mean to be belligerent, but if you're the accepting journal, and you're going to potentially change the submitters' bibliography...I don't know, that sounds tricky to me.

I must in this admit that I have a pretty narrow focus in bibliography production that can not easily be accused of being pragmatic.

Rintze

188BET靠谱吗that's not produced by the CSL style, so either I'm mistaken that ref-extractor uses the CSL style (it could technically use an online Zotero translator server, but I see no evidence of that, and it claims everything happens in the browser), or you are indeed passing it through Zotero.

Reference Extractor doesn't use CSL to generate BibTeX output.It feeds extracted CSL-JSON to the Citation.js library ( https://citation.js.org/), and I'm pretty sure its BibTeX generator ( https://github.com/citation-js/citation-js/tree/master/packages/plugin-bibtex) doesn't rely on CSL.

I also just discovered that Citation.js has the (rather hidden) option of using the original item ID as BibTeX citekey ( https://github.com/larsgw/citation.js/issues/181), which should be easy to enable in Reference Extractor.

emilianoeheyns

Well there you go, option #3 I wasn't aware of.

I haven't played with citation.js so I don't have an opinion on the conversion it offers.188BET靠谱吗I do know that conversion between CSL (which is what can be extracted from a word doc with zotero references) and bibtex is a lot more complex than most people appreciate.I for one did not know what I was getting into when I started.

Rintze

By the way, this might be a good feature request for pandoc ( https://pandoc.org), which should be able to convert .docx files to LaTeX, and already can use CSL-JSON for citation rendering.

emilianoeheyns

It's not just that it can use it, it's the primary path for most (but not all) pandoc output!for most conversions, if you offer it bibtex, it first converts that to CSL, and that conversion is not lossless.It mostly works reasonably well, but if you can use a pure-CSL path through pandoc (unless you're basically use it as latexmk), you get the best results.

Other things to look out for are cross-references (specifically, but not exclusively, xrefs that point to a page number).