Exporting metadata AND (cached) full-text in some format

sdspieg
188BET靠谱吗I've (sort of) asked this before and we are (sort of) able to do this now, but still: does anybody know of an elegant way to export Zotero libraries/collections in a way that includes both the metadata AND the full text in a single file (JSON, csv, ...) that can then be read by third-party tools (for textmining, corpus annotation, machine learning purposes, etc.)?Thanks!
  • sdspieg
    Nobody?We are really struggling with this...
  • bwiernik
    188BET靠谱吗The Zotero Beta now has a way to pass arbitrary JavaScript code to Zotero so that you can access the JavaScript API.You can use that API to access a collection, its items, and their full text content.
  • sdspieg
    Thanks.I'm happy to hear it is possible, even though I still have no idea how exactly to do it .Could you provide an example?Or even refer us to some documentation on how to do this?Please also note that I have had other people (that I don't know personally) approach me on this - also on the forum!188BET靠谱吗so we're not the only ones who'd love to find out how to export our collections from Zotero and then import them in tools like Voyant or ITMS or the new Thresher Quickcode or...
    I apologize for asking this, and we MAY be able to figure this out.But it would take us days or even weeks.With no guarantee of success.
  • bwiernik
    The JS API is described some here 188BET靠谱吗//www.brodersterzo.com/support/dev/client_coding/javascript_api

    Look at the code for retrieving a list of items in a collection, and the code lower down on retrieving item attachments.
  • sdspieg
    If anybody else is interested in this and has some js-coding skills: please do contact me.188BET靠谱吗We are still extremely eager to find an elegant way to export both the metadata AND and the full-text (cached) text from Zotero libraries/collections into a format that can be processed by textmining tools.Thanks!
  • dstillman
    188BET靠谱吗By full text, you mean the contents of the .zotero-ft-cache file?
  • sdspieg
    Yes I do
  • ntrlshrp
    @sdspieg188BET靠谱吗, I'm not sure if you're still seeking a solution 10 months later, but to future visitors who want to "export both the metadata AND and the full-text (cached) text from Zotero libraries/collections into a format that can be processed by textmining tools":

    If you're willing to use R, you can do it like this:

    #####
    # install.packages(c("magrittr", "DBI", "RSQLite", "quanteda", "readtext"))
    library(magrittr)
    library(DBI)
    library(RSQLite)
    library(quanteda)
    library(readtext)

    188BET靠谱吗# connect to Zotero's SQLite database
    con = dbConnect(drv = RSQLite::SQLite(),
    188BET靠谱吗dbname = "~/Zotero/zotero.sqlite")

    # get names of all tables in the database
    alltables = dbListTables(con)

    # bring the items and itemNotes tables into R
    table.items <- dbGetQuery(con, 'select * from items')
    table.itemNotes <- dbGetQuery(con, 'select * from itemNotes')

    188BET靠谱吗# bring in Zotero fulltext cache plaintext
    188BET靠谱吗textDF <- readtext(paste0("~/Zotero/storage", "/*/.zotero-ft-cache"),
    docvarsfrom = "filepaths")

    # isolate "key" (8-character alphanumeric directory in storage/) in docvar1 associated with plaintext
    textDF$docvar1 <- gsub(pattern = "^.*storage\\/", replacement = "", x = textDF$docvar1)
    textDF$docvar1 <- gsub(pattern = "\\/.*", replacement = "", x = textDF$docvar1)

    # bring in itemID (and some other metadata) and that's all
    textDF <- textDF %>%
    dplyr::rename(key = docvar1) %>%
    dplyr::left_join(table.items) %>%
    dplyr::filter(!is.na(itemID), !itemID %in% table.itemNotes$itemID)
    #####

    188BET靠谱吗For a full example, you can see how I used this to calculate tf-idf for each indexed Zotero item and then added new tags based on those results to the database here: https://ntrlshrp.gitlab.io/post/zotfidf/

    188BET靠谱吗*** NB: This directly accesses Zotero's local SQLite database, which is considered programatically more brittle / fragile than working with the local JavaScript API (see 188BET靠谱吗//www.brodersterzo.com/support/dev/client_coding/javascript_api).

    Hope that is helpful!
  • adamsmith
    This is cool.To be clear, the code posted above is completely fine to use.188BET靠谱吗There is no issue with reading from the Zotero database (you'll want Zotero closed while running the code so it's not locked).It might stop working if the database structure changes, but I'd actually expect it to be fairly stable (and easy to fix if it does).In any case, worst case is that it doesn't run.

    The part that's iffy is the re-writing to the database (i.e.the part in the blogpost that starts with if(NEW_TAGS).That almost certainly breaks the database at a minimum in the sense that it will lead to unpredictable behavior when syncing, so I'd very much discourage anyone to do this with a database they want to keep working with.
  • sdspieg
    @ntrlshrp- thanks much.We're looking into this.
  • ntrlshrp
    Thanks, @adamsmith, for your kudos and for reminding me of the read-only recommendation here: 188BET靠谱吗//www.brodersterzo.com/support/dev/client_coding/direct_sqlite_database_access.I've put that front and center on the post to warn readers.
  • sdspieg

    I can make this work in RStudio until

    188BET靠谱吗> textDF <- readtext(paste0("~/Zotero/storage", "/*/.zotero-ft-cache"),
    + docvarsfrom = "filepaths")

    Which is where I get:

    Error in list_files(file, ignore_missing, TRUE, verbosity) :
    File '' does not exist

    188BET靠谱吗And so for 'dbname =', I entered the path to my zotero.sqlite.Which I guess it accepted.But what if I only want to use one group collection?How can I specify that?

Sign Inor Registerto comment.