notebooks/Abbreviations_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Acronym Extraction"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(C) 2022-2024 by [Damir Cavar](https://damir.cavar.me/)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Version:** 0.3, September 2024\n",
    "\n",
    "**Download:** This and various other Jupyter notebooks are available from my [GitHub repo](https://github.com/dcavar/python-tutorial-for-ipython).\n",
    "\n",
    "**License:** [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/) ([CA BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an example of the use of the *[abbreviations](https://github.com/philgooch/abbreviation-extraction)* module to extract acronyms from documents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the module using:\n",
    "\n",
    "    pip install abbreviations\n",
    "\n",
    "For the *FileChooser* widget in this Jupyter notebook you might need to install also the *[ipyfilechooser](https://github.com/crahan/ipyfilechooser)*:\n",
    "\n",
    "    pip install ipyfilechooser\n",
    "\n",
    "The code below assumes that the text is encoded as UTF-8. If this is not the case for you, adapt the encoding specification in the *get_abbreviation* function below or convert your text to use the UTF-8 character encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U abbreviations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U ipyfilechooser"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following code to activate the *FileChooser* and select a folder with the target text files in it. The target text files can be in subfolders of arbitrary depth within this folder. A good example file is ```bio_1.txt``` in the ```data``` subfolder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46a8735bb95a411d9ef2513275e918cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FileChooser(path='C:\\Users\\damir\\Dropbox\\Develop\\python-tutorial-notebooks\\notebooks', filename='', title='', …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ipyfilechooser import FileChooser\n",
    "fc = FileChooser()\n",
    "display(fc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following code cell we will import the necessary modules *[abbreviations](https://github.com/philgooch/abbreviation-extraction)* and *os* used in the functions below to process subfolders, find target text files, and extract all abbreviations from them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from abbreviations import schwartz_hearst\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function reads the content from a text file in the *folder_path* and *directory* subdirectory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_abbreviations(file_name = \"\"):\n",
    "\tif not os.path.exists(file_name):\n",
    "\t\treturn\n",
    "\tprint(\"Processing file:\", file_name)\n",
    "\ttry:\n",
    "\t\tifp = open(file_name, mode='r', encoding='utf-8')\n",
    "\t\ttext = ifp.read()\n",
    "\t\tifp.close()\n",
    "\texcept IOError:\n",
    "\t\treturn\n",
    "\tif not text:\n",
    "\t\treturn\n",
    "\tmost_common_defs = schwartz_hearst.extract_abbreviation_definition_pairs(doc_text=text, most_common_definition=True)\n",
    "\tfirst_defs       = schwartz_hearst.extract_abbreviation_definition_pairs(doc_text=text, first_definition=True)\n",
    "\treturn most_common_defs, first_defs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the selected text file and print the resulting abbreviations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing file: C:\\Users\\damir\\Dropbox\\Develop\\python-tutorial-notebooks\\notebooks\\data\\bio_1.txt\n",
      "({'ER': 'endoplasmic reticulum'}, {'ER': 'endoplasmic reticulum'})\n"
     ]
    }
   ],
   "source": [
    "abbreviations = get_abbreviations(os.path.join(fc.selected_path, fc.selected_filename))\n",
    "print(abbreviations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "uri_prefix = \"https://www.indiana.edu/nlplab/bioterminology#\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rdflib.namespace import RDF, RDFS, SKOS, OWL, DC, DCTERMS, XSD, TIME, NamespaceManager\n",
    "from rdflib import Graph, URIRef, Literal, Namespace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = {}\n",
    "for x in abbreviations:\n",
    "    entry = tuple(x.items())[0]\n",
    "    dictionary[uri_prefix + \"\".join([ z.title() for z in entry[1].split() ])] = entry\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "g = Graph()\n",
    "vaem_acronym = URIRef(\"https://www.linkedmodel.org/schema/vaem#acronym\")\n",
    "for key in dictionary:\n",
    "    g.add( (URIRef(key), RDFS.label, Literal(dictionary[key][1])) )\n",
    "    g.add( (URIRef(key), vaem_acronym, Literal(dictionary[key][0])) )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "import pytextrank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pytextrank.base.BaseTextRankFactory at 0x22b2e203010>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp = spacy.load(\"en_core_web_trf\") # en_core_web_sm\")\n",
    "nlp.add_pipe(\"textrank\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(fc.selected_path, fc.selected_filename), mode='r', encoding='utf-8') as ifp:\n",
    "    text = ifp.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for phrase in doc._.phrases:\n",
    "    print(phrase.text)\n",
    "    print(phrase.rank, phrase.count)\n",
    "    print(phrase.chunks)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n"
     ]
    }
   ],
   "source": [
    "dictionary = {}\n",
    "for x in doc._.phrases:\n",
    "    entry = x.text # tuple(x.items())[0]\n",
    "    key = URIRef(uri_prefix + \"\".join([ z.title() for z in entry.split() ]))\n",
    "    # = entry\n",
    "    # for key in dictionary:\n",
    "    g.add( (URIRef(key), RDFS.label, Literal(entry)) )\n",
    "    # g.add( (URIRef(key), vaem_acronym, Literal(dictionary[key][0])) )\n",
    "print(dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "nelabels = {\"ORG\": \"Organization\",\n",
    "            \"PERSON\": \"Person\"}\n",
    "for k in nelabels:\n",
    "    g.add( ( URIRef(uri_prefix + nelabels[k]), RDF.type, OWL.Class) )\n",
    "for ent in doc.ents:\n",
    "    # print(ent.text, ent.label_)\n",
    "    if ent.label_ in (\"ORG\", \"PERSON\"):\n",
    "        key = URIRef(uri_prefix + \"\".join([ z.title() for z in ent.text.split() ]))\n",
    "        type = nelabels[ent.label_]\n",
    "        g.add( (key, RDF.type, URIRef(uri_prefix + type)) )\n",
    "        g.add( (key, RDFS.label, Literal(ent.text)) )\n",
    "        # print(ent.text, ent.label_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Graph identifier=N8793d07f04814241b59925a0d5c3aeae (<class 'rdflib.graph.Graph'>)>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g.serialize(destination=\"data/test_graph.ttl\", format=\"turtle\", encoding=\"utf-8\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(C) 2022-2024 by [Damir Cavar](https://damir.cavar.me/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "1e28a5307a9b5c2fbeb0b263581f1cf3bfba9739188743f6a231f74c7de58892"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}