ETL processes for medical and scientific papers

paperetl is an ETL library for processing medical and scientific papers.

paperetl supports the following sources:

• Full PDF articles
• PubMed XML
• ArXiv XML
• Text Encoding Initiative (TEI) XML
• CSV with article metadta

paperetl supports the following datastores for parsed articles.

• SQLite
• JSON files
• YAML files

Additional optional datastores are available.

• Elasticsearch

The easiest way to install is via pip and PyPI

pip install paperetl

Python 3.10+ is supported. Using a Python virtual environment is recommended.

paperetl can also be installed directly from GitHub to access the latest, unreleased features.

pip install git+https://github.com/neuml/paperetl

PDF parsing relies on an existing GROBID instance to be up and running. It is assumed that this is running locally on the ETL server. This is only necessary for PDF files.

• GROBID install instructions
• GROBID start service

Note: In some cases, the GROBID engine pool can be exhausted, resulting in a 503 error. This can be fixed by increasing concurrency and/or poolMaxWait in the GROBID configuration file.

A Dockerfile with commands to install paperetl, all dependencies and scripts are available in this repository.

wget https://raw.githubusercontent.com/neuml/paperetl/master/docker/Dockerfile
docker build -t paperetl -f Dockerfile .
docker run --name paperetl --rm -it paperetl

This will bring up a paperetl command shell. Standard Docker commands can be used to copy files over or commands can be run directly in the shell to retrieve input content.

Notebook	Description
Introducing paperetl	Overview of the functionality provided by paperetl

The following example shows how to use paperetl to load a set of medical/scientific articles into a SQLite database.

1. Download the desired medical/scientific articles in a local directory. For this example, it is assumed the articles are in a directory named paperetl/data

2. Build the database

   python -m paperetl.file paperetl/data paperetl/models
   

Once complete, there will be an articles.sqlite file in paperetl/models

Elasticsearch is a supported datastore. It's an optional install feature via the Elasticsearch extra.

pip install paperetl[elasticsearch]

This example assumes Elasticsearch is running locally, change the URL to a remote server as appropriate.

python -m paperetl.file paperetl/data http://localhost:9200

Once complete, there will be an articles index in Elasticsearch with the metadata and full text stored.

paperetl can also be used to convert articles into JSON or YAML files. This is useful if the data is to be fed into another system or for manual inspection/debugging of a single file.

JSON:

python -m paperetl.file paperetl/data json://paperetl/json

YAML:

python -m paperetl.file paperetl/data yaml://paperetl/yaml

Converted files will be stored in paperetl/(json|yaml)