context_term_gst_tool

Context Term Detection Tool from Abstract/Full-Length Articles

Overview

The context term detection tool detects important context terms such as protein, site, disease, species, tissue, and cell line from both abstracts and full-length articles. It processes text from different sections (Title, Abstract, Methods, Results, Discussion, Conclusion) based on PubTator annotations and dictionaries.

• Proteins, diseases, species, and cell lines are detected using PubTator3.
• Tissues are detected using a comprehensive tissue dictionary named Brenda.
• Sites are detected using an in-house sitedetector tool.
• For dictionary-based detection, the tool uses OGER.

The tool first collects biocxml files for a list of PMIDs using the PubTator3 API, from which texts are parsed for dictionary-based context term detection, site detection, and final output creation.

Output Files

Abstracts

Excel Sheet (Outputs_Abstract/1_new_dis_spec_tis_cel_output.xlsx)

The output file is generated by running a pmid.txt file through the pipeline to detect context terms from abstracts.

Columns Description

• pmid: Article PMID
• entity: Detected term
• entity_type: Type of the entity (protein, site, disease, species, cell line, or tissue)
• start_offset, end_offset: Position of the entity term in the sentence
• source: Dictionary from which the entity was detected (e.g., Brenda for tissue)
• id: Unique identifier (e.g., MESH ID, Cellosaurus ID, gene ID, etc.)
• doid_for_diseases: Mapped Disease Ontology ID (DOID) for diseases
• uniprot_ac: UniProt accession ID for genes detected by PubTator
• sent_index: Sentence number where the entity was found
• sentence: Extracted sentence containing the entity

HTML File (Outputs_Abstract/highlighted_entities.html)

• Detected terms are highlighted with multiple colors.
• Legends are available at the beginning to indicate color codes.
• Abstract titles appear as PMID: <pmid>.

Full-Length Articles

Excel Sheet (Outputs_FL/14_fl_dis_spec_tis_cel_output.xlsx)

The output file is generated by running a pmid.txt file through the pipeline to detect context terms from full-length articles.

Additional Files

• missing_full_length_pmids.txt: List of PMIDs without open access to full-length articles.
• pmids_pmcids.xlsx: Mapping of PMIDs to PMCIDs (ignore if available in the output sheet).

Columns Description

• pmid, pmcid: Article IDs
• entity, entity_type, start_offset, end_offset
• source: Dictionary from which the entity was detected
• id, doid_for_diseases, uniprot_ac
• sent_index: Sentence number
• section: Section where the entity was found
• sentence: Extracted sentence containing the entity

HTML File (Outputs_FL/highlighted_entities_FL.html)

• Terms are highlighted in each section.
• Sections are labeled as PMCID_SECTIONNAME (e.g., PMC2391110_ABSTRACT).
• Sentence index matches the Excel sheet.

Glycan Structure Terms (GSTs) Detection

The GST tool detects glycan structure terms from abstracts and full-length articles based on an ML-based BERT model. The model is currently trained on 400 abstracts but may require further improvements.

Abstracts

Excel Sheet (Outputs/final_gst.xlsx)

• pmid: Article PMID
• sent_index: Sentence number
• sent_text: Sentence containing detected GSTs
• BERT_Output: GST terms detected by the ML model (separated by | if multiple)

HTML File (Outputs/Highlighted_ML_Predictions.html)

• GST terms are highlighted in blue.
• Abstract titles appear as PMID: <pmid>.

Full-Length Articles

Excel Sheet (Outputs/final_gst.xlsx)

• pmcid: PMCID
• section: Section where the GST was found
• paragraph, sent_index: Paragraph and sentence index
• sent_text: Extracted sentence
• BERT_Output: Detected GST terms

HTML File (Outputs/Highlighted_ML_Predictions_Full.html)

• GST terms are highlighted in blue.
• Sections appear as PMCID_SECTION_PARAGRAPH.
• Sentence index matches the Excel sheet.

Docker Instructions

Software Requirements

• Docker (v18.09.6 or newer)

Running the Docker Container

1. Make the script executable:

   chmod +x run_docker.sh

2. Run the script:

   ./run_docker.sh

3. Choose an option when prompted:
   • abstract (Context Term Detection - Abstracts)
   • full-length (Context Term Detection - Full-length articles)
   • both (Both abstract and full-length processing)
   • gst_abstract (GST Detection - Abstracts)
   • gst_full (GST Detection - Full-length articles)
4. If asked for a new PMID file, enter yes and provide the file path (e.g., /home/user/context_term_docker/pmid.txt).
5. The process may take 30-45 minutes for 5 PMIDs (Context Term Detection) or 15 minutes for 5 PMIDs (GST Detection).

Docker Container Files

Context Term Detection

• abstract_xml / bioc_xml_fl: Extracted XML files from PubTator3
• abstract_text / output_text_section_FL: Text files for PMIDs
• Outputs_Abstract / Outputs_FL: Output files (Excel, HTML)
• Scripts_Abstract / Scripts_FL: Processing scripts
• Dictionary: Brenda dictionary for tissue detection
• mesh_to_doid: Mapping files for disease ID conversion
• OGER: Dictionary-based entity detection tool
• obo_to_tsv_tissue.py: Updates OGER dictionary with new Brenda terms

GST Detection

• Outputs: Output files (Excel, HTML)
• Scripts: Processing scripts
• Saved_Model: ML model weights, tokenizer, and dictionary

Common Files

• requirements.txt: Lists required dependencies for all tools

Context Term Detection Tool from Abstract/Full-Length Articles

Overview

The context term detection tool detects important context terms such as protein, site, disease, species, tissue, and cell line from both abstracts and full-length articles. It processes text from different sections (Title, Abstract, Methods, Results, Discussion, Conclusion) based on PubTator annotations and dictionaries.

• Proteins, diseases, species, and cell lines are detected using PubTator3.
• Tissues are detected using a comprehensive tissue dictionary named Brenda.
• Sites are detected using an in-house sitedetector tool.
• For dictionary-based detection, the tool uses OGER.

The tool first collects biocxml files for a list of PMIDs using the PubTator3 API, from which texts are parsed for dictionary-based context term detection, site detection, and final output creation.

Output Files

Abstracts

Excel Sheet (Outputs_Abstract/1_new_dis_spec_tis_cel_output.xlsx)

The output file is generated by running a pmid.txt file through the pipeline to detect context terms from abstracts.

Columns Description

• pmid: Article PMID
• entity: Detected term
• entity_type: Type of the entity (protein, site, disease, species, cell line, or tissue)
• start_offset, end_offset: Position of the entity term in the sentence
• source: Dictionary from which the entity was detected (e.g., Brenda for tissue)
• id: Unique identifier (e.g., MESH ID, Cellosaurus ID, gene ID, etc.)
• doid_for_diseases: Mapped Disease Ontology ID (DOID) for diseases
• uniprot_ac: UniProt accession ID for genes detected by PubTator
• sent_index: Sentence number where the entity was found
• sentence: Extracted sentence containing the entity

HTML File (Outputs_Abstract/highlighted_entities.html)

• Detected terms are highlighted with multiple colors.
• Legends are available at the beginning to indicate color codes.
• Abstract titles appear as PMID: <pmid>.

Full-Length Articles

Excel Sheet (Outputs_FL/14_fl_dis_spec_tis_cel_output.xlsx)

The output file is generated by running a pmid.txt file through the pipeline to detect context terms from full-length articles.

Additional Files

• missing_full_length_pmids.txt: List of PMIDs without open access to full-length articles.
• pmids_pmcids.xlsx: Mapping of PMIDs to PMCIDs (ignore if available in the output sheet).

Columns Description

• pmid, pmcid: Article IDs
• entity, entity_type, start_offset, end_offset
• source: Dictionary from which the entity was detected
• id, doid_for_diseases, uniprot_ac
• sent_index: Sentence number
• section: Section where the entity was found
• sentence: Extracted sentence containing the entity

HTML File (Outputs_FL/highlighted_entities_FL.html)

• Terms are highlighted in each section.
• Sections are labeled as PMCID_SECTIONNAME (e.g., PMC2391110_ABSTRACT).
• Sentence index matches the Excel sheet.

Glycan Structure Terms (GSTs) Detection

Overview

The GST tool detects glycan structure terms from abstracts and full-length articles based on an ML-based BERT model. The model is currently trained on 400 abstracts but may require further improvements.

Abstracts

Excel Sheet (Outputs/final_gst.xlsx)

• pmid: Article PMID
• sent_index: Sentence number
• sent_text: Sentence containing detected GSTs
• BERT_Output: GST terms detected by the ML model (separated by | if multiple)

HTML File (Outputs/Highlighted_ML_Predictions.html)

• GST terms are highlighted in blue.
• Abstract titles appear as PMID: <pmid>.

Full-Length Articles

Excel Sheet (Outputs/final_gst.xlsx)

• pmcid: PMCID
• section: Section where the GST was found
• paragraph, sent_index: Paragraph and sentence index
• sent_text: Extracted sentence
• BERT_Output: Detected GST terms

HTML File (Outputs/Highlighted_ML_Predictions_Full.html)

• GST terms are highlighted in blue.
• Sections appear as PMCID_SECTION_PARAGRAPH.
• Sentence index matches the Excel sheet.

Docker Instructions

Run the docker file to execute any of the pipeline discussed above.

Software Requirements

• Docker (v18.09.6 or newer)

Running the Docker Container

1. Make the script executable:

   chmod +x run_docker.sh

2. Run the script:

   ./run_docker.sh

3. Choose an option when prompted:
   • abstract (Context Term Detection - Abstracts)
   • full-length (Context Term Detection - Full-length articles)
   • both (Both abstract and full-length processing)
   • gst_abstract (GST Detection - Abstracts)
   • gst_full (GST Detection - Full-length articles)
4. If asked for a new PMID file, enter yes and provide the file path (e.g., /home/user/context_term_docker/pmid.txt).
5. The process may take 30-45 minutes for 5 PMIDs (Context Term Detection) or 15 minutes for 5 PMIDs (GST Detection).

Docker Container Files

Context Term Detection

• abstract_xml / bioc_xml_fl: Extracted XML files from PubTator3
• abstract_text / output_text_section_FL: Text files for PMIDs
• Outputs_Abstract / Outputs_FL: Output files (Excel, HTML)
• Scripts_Abstract / Scripts_FL: Processing scripts
• Dictionary: Brenda dictionary for tissue detection
• mesh_to_doid: Mapping files for disease ID conversion
• OGER: Dictionary-based entity detection tool
• obo_to_tsv_tissue.py: Updates OGER dictionary with new Brenda terms

GST Detection

• Outputs: Output files (Excel, HTML)
• Scripts: Processing scripts
• Saved_Model: ML model weights, tokenizer, and dictionary

Common Files

• requirements.txt: Lists required dependencies for all tools

Scripts

• Run each script individually by following the execution process inside run_container.sh bash file.
• Scripts are numbered sequentially

Folders

• context_term_abstract_version: Files for the abstract version of the context term detection tool
• context_term_full_length_version: Files for the full-length version of the context term detection tool
• gst_abstract_version: Files for the abstract version of the GST detection tool
• gst_full_version: Files for the full-length version of the GST detection tool

Model Weights

• Model weights are available in saved_model.tar.gz zip file
• Model weights can be found here
• We recommend running the docker to obtain the actual weights of the GST tool

N.B: More details can be found in the documentation.docx file.