{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import os\n", "import collections\n", "import pickle\n", "import gzip\n", "\n", "from tf.app import use" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using TF app oldbabylonian in /Users/dirk/github/annotation/app-oldbabylonian/code\n", "Using Nino-cunei/oldbabylonian/tf - 1.0.4 in /Users/dirk/github\n" ] }, { "data": { "text/html": [ "Documentation: OLDBABYLONIAN Character table Feature docs oldbabylonian API Text-Fabric API 7.5.1 Search Reference
Loaded features:\n", "

Old Babylonian Letters 1900-1600: Cuneiform tablets : ARK after afterr afteru atf atfpost atfpre author col collated collection comment damage det docnote docnumber excavation excised face flags fraction genre grapheme graphemer graphemeu lang langalt ln lnc lnno material missing museumcode museumname object operator operatorr operatoru otype period pnumber primecol primeln pubdate question reading readingr readingu remarkable remarks repeat srcLn srcLnNum srcfile subgenre supplied sym symr symu trans transcriber translation@ll type uncertain volume oslots

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
API members:\n", "C Computed, Call AllComputeds, Cs ComputedString
\n", "E Edge, Eall AllEdges, Es EdgeString
\n", "ensureLoaded, TF, ignored, loadLog
\n", "L Locality
\n", "cache, error, indent, info, reset
\n", "N Nodes, sortKey, sortKeyTuple, otypeRank, sortNodes
\n", "F Feature, Fall AllFeatures, Fs FeatureString
\n", "S Search
\n", "T Text
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "A = use('oldbabylonian', hoist=globals(), lgc=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Parallels\n", "\n", "We make edges between similar lines.\n", "\n", "When are lines similar?\n", "\n", "If a certain distance metric is above a certain threshold.\n", "\n", "We choose this metric:\n", "\n", "* we reduce a line to the set of readings and graphemes in it, excluding unknown signs and ellipses.\n", "* the similarity between two lines is the length of the intersection divided by the length of the union of their sets times 100." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Preparation\n", "\n", "We pre-compute all sets for all lines." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "READABLE_TYPES = {'reading', 'grapheme', 'numeral', 'complex'}\n", "\n", "def makeSet(l):\n", " if F.lnc.v(l): # comment line\n", " return None\n", " lineSet = set()\n", " for s in L.d(l, otype='sign'):\n", " if F.type.v(s) in READABLE_TYPES:\n", " r = F.readingr.v(s)\n", " if r:\n", " lineSet.add(r)\n", " g = F.graphemer.v(s)\n", " if g:\n", " lineSet.add(g)\n", " return lineSet" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "25923 lines\n" ] } ], "source": [ "lines = {}\n", "\n", "for l in F.otype.s('line'):\n", " lineSet = makeSet(l)\n", " if lineSet:\n", " lines[l] = lineSet\n", " \n", "nLines = len(lines)\n", "print(f'{nLines} lines')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Measure" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def sim(lSet, mSet):\n", " return int(round(100 * len(lSet & mSet) / len(lSet | mSet)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Compute all similarities\n", "\n", "We are going to perform several hundreds of millions of comparisons, each of which is more than an elemetary operation.\n", "\n", "Let's measure time." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "THRESHOLD = 90\n", "\n", "def computeSim(limit=None):\n", " similarity = {}\n", "\n", " lineNodes = sorted(lines.keys())\n", " nLines = len(lineNodes)\n", "\n", " nComparisons = nLines * (nLines - 1) // 2\n", "\n", " print(f'{nComparisons} comparisons to make')\n", " chunkSize = nComparisons // 100\n", "\n", " co = 0\n", " b = 0\n", " si = 0\n", " p = 0\n", "\n", " indent(reset=True)\n", "\n", " stop = False\n", " for i in range(nLines):\n", " nodeI = lineNodes[i]\n", " lineI = lines[nodeI]\n", " for j in range(i + 1, nLines):\n", " nodeJ = lineNodes[j]\n", " lineJ = lines[nodeJ]\n", " s = sim(lineI, lineJ)\n", " co += 1\n", " b += 1\n", " if b == chunkSize:\n", " p += 1\n", " info(f'{p:>3}% - {co:>12} comparisons and {si:>10} similarities')\n", " b = 0\n", " if limit is not None and p >= limit:\n", " stop = True\n", " break\n", "\n", " if s < THRESHOLD:\n", " continue\n", " similarity[(nodeI, nodeJ)] = sim(lineI, lineJ)\n", " si += 1\n", " if stop:\n", " break\n", "\n", " info(f'{p:>3}% - {co:>12} comparisons and {si:>10} similarities')\n", " return similarity" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are going to run it to 1% first and do some checks then." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "335988003 comparisons to make\n", " 4.42s 1% - 3359880 comparisons and 5695 similarities\n", " 4.42s 1% - 3359880 comparisons and 5695 similarities\n" ] } ], "source": [ "similarity = computeSim(limit=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We check the sanity of the results." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "90\n", "100\n" ] } ], "source": [ "print(min(similarity.values()))\n", "print(max(similarity.values()))" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "eq = [x for x in similarity.items() if x[1] >= 100]\n", "neq = [x for x in similarity.items() if x[1] <= 90]" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "5689\n", "5\n" ] } ], "source": [ "print(len(eq))\n", "print(len(neq))" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "((230788, 235394), 100)\n", "((230797, 230811), 90)\n" ] } ], "source": [ "print(eq[0])\n", "print(neq[0])" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "P509373 obverse:1 [a-na] _{d}suen_-i-[din-nam]" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "P510729 obverse:1 a-na {d}suen-i-din-[nam]" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "A.plain(eq[0][0][0])\n", "A.plain(eq[0][0][1])" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "P509373 obverse:10 _a-sza3 a-gar3_ na-ag-[ma-lum] _uru_ x x x{ki}" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "P509373 reverse:7' _a-[sza3 a-gar3_ na-ag]-ma-lum _uru gan2_ x x{ki}" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "A.plain(neq[0][0][0])\n", "A.plain(neq[0][0][1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looks good.\n", "\n", "Now the whole computation.\n", "\n", "But if we have done this before, and nothing has changed, we load previous results from disk.\n", "\n", "If we do not find previous results, we compute them and save the results to disk." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "PARA_DIR = f'{A.tempDir}/parallels'\n", "\n", "def writeResults(data, location, name):\n", " if not os.path.exists(location):\n", " os.makedirs(location, exist_ok=True)\n", " path = f'{location}/{name}'\n", " with gzip.open(path, 'wb') as f:\n", " pickle.dump(data, f)\n", " print(f'Data written to {path}')\n", " \n", "def readResults(location, name):\n", " path = f'{location}/{name}'\n", " if not os.path.exists(path):\n", " print(f'File not found: {path}')\n", " return None\n", " with gzip.open(path, 'rb') as f:\n", " data = pickle.load(f)\n", " print(f'Data read from {path}')\n", " return data" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "File not found: /Users/dirk/github/Nino-cunei/oldbabylonian/_temp/parallels/sim-1.0.4.zip\n", "335988003 comparisons to make\n", " 4.52s 1% - 3359880 comparisons and 5695 similarities\n", " 8.68s 2% - 6719760 comparisons and 10604 similarities\n", " 13s 3% - 10079640 comparisons and 16028 similarities\n", " 17s 4% - 13439520 comparisons and 24005 similarities\n", " 21s 5% - 16799400 comparisons and 30694 similarities\n", " 26s 6% - 20159280 comparisons and 37992 similarities\n", " 30s 7% - 23519160 comparisons and 44479 similarities\n", " 34s 8% - 26879040 comparisons and 49298 similarities\n", " 38s 9% - 30238920 comparisons and 56259 similarities\n", " 43s 10% - 33598800 comparisons and 59520 similarities\n", " 47s 11% - 36958680 comparisons and 64009 similarities\n", " 52s 12% - 40318560 comparisons and 69139 similarities\n", " 56s 13% - 43678440 comparisons and 71511 similarities\n", " 1m 00s 14% - 47038320 comparisons and 75137 similarities\n", " 1m 05s 15% - 50398200 comparisons and 82081 similarities\n", " 1m 09s 16% - 53758080 comparisons and 88712 similarities\n", " 1m 13s 17% - 57117960 comparisons and 92990 similarities\n", " 1m 18s 18% - 60477840 comparisons and 98118 similarities\n", " 1m 22s 19% - 63837720 comparisons and 103792 similarities\n", " 1m 26s 20% - 67197600 comparisons and 110670 similarities\n", " 1m 31s 21% - 70557480 comparisons and 116855 similarities\n", " 1m 35s 22% - 73917360 comparisons and 123838 similarities\n", " 1m 39s 23% - 77277240 comparisons and 127108 similarities\n", " 1m 43s 24% - 80637120 comparisons and 131270 similarities\n", " 1m 48s 25% - 83997000 comparisons and 135406 similarities\n", " 1m 52s 26% - 87356880 comparisons and 140367 similarities\n", " 1m 56s 27% - 90716760 comparisons and 146136 similarities\n", " 2m 00s 28% - 94076640 comparisons and 149919 similarities\n", " 2m 04s 29% - 97436520 comparisons and 155037 similarities\n", " 2m 08s 30% - 100796400 comparisons and 160769 similarities\n", " 2m 13s 31% - 104156280 comparisons and 164146 similarities\n", " 2m 17s 32% - 107516160 comparisons and 171161 similarities\n", " 2m 23s 33% - 110876040 comparisons and 177745 similarities\n", " 2m 27s 34% - 114235920 comparisons and 187290 similarities\n", " 2m 31s 35% - 117595800 comparisons and 192678 similarities\n", " 2m 36s 36% - 120955680 comparisons and 199539 similarities\n", " 2m 40s 37% - 124315560 comparisons and 204863 similarities\n", " 2m 44s 38% - 127675440 comparisons and 208504 similarities\n", " 2m 48s 39% - 131035320 comparisons and 214004 similarities\n", " 2m 53s 40% - 134395200 comparisons and 218780 similarities\n", " 2m 57s 41% - 137755080 comparisons and 224134 similarities\n", " 3m 01s 42% - 141114960 comparisons and 230051 similarities\n", " 3m 05s 43% - 144474840 comparisons and 234167 similarities\n", " 3m 10s 44% - 147834720 comparisons and 238211 similarities\n", " 3m 14s 45% - 151194600 comparisons and 243529 similarities\n", " 3m 18s 46% - 154554480 comparisons and 252826 similarities\n", " 3m 22s 47% - 157914360 comparisons and 258940 similarities\n", " 3m 27s 48% - 161274240 comparisons and 265872 similarities\n", " 3m 31s 49% - 164634120 comparisons and 275280 similarities\n", " 3m 35s 50% - 167994000 comparisons and 282491 similarities\n", " 3m 39s 51% - 171353880 comparisons and 291007 similarities\n", " 3m 43s 52% - 174713760 comparisons and 298643 similarities\n", " 3m 47s 53% - 178073640 comparisons and 305316 similarities\n", " 3m 51s 54% - 181433520 comparisons and 311836 similarities\n", " 3m 56s 55% - 184793400 comparisons and 318089 similarities\n", " 4m 00s 56% - 188153280 comparisons and 327814 similarities\n", " 4m 05s 57% - 191513160 comparisons and 334815 similarities\n", " 4m 09s 58% - 194873040 comparisons and 340222 similarities\n", " 4m 13s 59% - 198232920 comparisons and 346843 similarities\n", " 4m 18s 60% - 201592800 comparisons and 351896 similarities\n", " 4m 22s 61% - 204952680 comparisons and 355541 similarities\n", " 4m 27s 62% - 208312560 comparisons and 360413 similarities\n", " 4m 32s 63% - 211672440 comparisons and 364007 similarities\n", " 4m 37s 64% - 215032320 comparisons and 367280 similarities\n", " 4m 41s 65% - 218392200 comparisons and 373743 similarities\n", " 4m 46s 66% - 221752080 comparisons and 381456 similarities\n", " 4m 51s 67% - 225111960 comparisons and 387048 similarities\n", " 4m 55s 68% - 228471840 comparisons and 393861 similarities\n", " 5m 00s 69% - 231831720 comparisons and 400752 similarities\n", " 5m 04s 70% - 235191600 comparisons and 406637 similarities\n", " 5m 08s 71% - 238551480 comparisons and 412543 similarities\n", " 5m 13s 72% - 241911360 comparisons and 416528 similarities\n", " 5m 18s 73% - 245271240 comparisons and 420429 similarities\n", " 5m 22s 74% - 248631120 comparisons and 423760 similarities\n", " 5m 27s 75% - 251991000 comparisons and 427780 similarities\n", " 5m 31s 76% - 255350880 comparisons and 433772 similarities\n", " 5m 35s 77% - 258710760 comparisons and 440648 similarities\n", " 5m 39s 78% - 262070640 comparisons and 445911 similarities\n", " 5m 44s 79% - 265430520 comparisons and 449549 similarities\n", " 5m 48s 80% - 268790400 comparisons and 452116 similarities\n", " 5m 53s 81% - 272150280 comparisons and 455801 similarities\n", " 5m 57s 82% - 275510160 comparisons and 459217 similarities\n", " 6m 01s 83% - 278870040 comparisons and 466400 similarities\n", " 6m 06s 84% - 282229920 comparisons and 474485 similarities\n", " 6m 10s 85% - 285589800 comparisons and 480417 similarities\n", " 6m 14s 86% - 288949680 comparisons and 488060 similarities\n", " 6m 19s 87% - 292309560 comparisons and 494339 similarities\n", " 6m 23s 88% - 295669440 comparisons and 500244 similarities\n", " 6m 28s 89% - 299029320 comparisons and 505300 similarities\n", " 6m 32s 90% - 302389200 comparisons and 511420 similarities\n", " 6m 36s 91% - 305749080 comparisons and 518972 similarities\n", " 6m 41s 92% - 309108960 comparisons and 525355 similarities\n", " 6m 45s 93% - 312468840 comparisons and 531069 similarities\n", " 6m 49s 94% - 315828720 comparisons and 536378 similarities\n", " 6m 54s 95% - 319188600 comparisons and 543423 similarities\n", " 6m 58s 96% - 322548480 comparisons and 549768 similarities\n", " 7m 03s 97% - 325908360 comparisons and 557939 similarities\n", " 7m 07s 98% - 329268240 comparisons and 564490 similarities\n", " 7m 11s 99% - 332628120 comparisons and 569775 similarities\n", " 7m 16s 100% - 335988000 comparisons and 576523 similarities\n", " 7m 16s 100% - 335988003 comparisons and 576523 similarities\n", "Data written to /Users/dirk/github/Nino-cunei/oldbabylonian/_temp/parallels/sim-1.0.4.zip\n" ] } ], "source": [ "similarity = readResults(PARA_DIR, f'sim-{A.version}.zip')\n", "if not similarity:\n", " similarity = computeSim()\n", " writeResults(similarity, PARA_DIR, f'sim-{A.version}.zip')" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "576523" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(similarity)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, over half a million pairs of similar lines.\n", "\n", "Let's find out which lines have the most correspondences." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "8344 out of 25923 lines have at least one similar line\n" ] } ], "source": [ "parallels = {}\n", "\n", "for (l, m) in similarity:\n", " parallels.setdefault(l, set()).add(m)\n", " parallels.setdefault(m, set()).add(l)\n", " \n", "print(f'{len(parallels)} out of {nLines} lines have at least one similar line')" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "rankedParallels = sorted(\n", " parallels.items(),\n", " key=lambda x: (-len(x[1]), x[0]),\n", ")" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1005 siblings of 230789 = qi2-bi2-[ma]\n", "1005 siblings of 230825 = qi2-bi2-ma\n", "1005 siblings of 230843 = [qi2]-bi2#-ma\n", "1005 siblings of 230907 = qi2-bi2-ma\n", "1005 siblings of 230915 = qi2-bi2-ma\n", "1005 siblings of 231003 = qi2-bi2-ma\n", "1005 siblings of 231021 = qi2-bi2-ma\n", "1005 siblings of 231054 = [qi2]-bi2-ma\n", "1005 siblings of 231067 = qi2-bi2-ma\n", "1005 siblings of 231087 = [qi2]-bi2#-ma\n" ] } ], "source": [ "for (l, paras) in rankedParallels[0:10]:\n", " print(f'{len(paras):>4} siblings of {l} = {T.text(l)}')" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1005 siblings of qi2-bi2-ma\n", "1005 siblings of qi2-bi2-ma\n", "1005 siblings of qi2-bi2-ma\n", "1005 siblings of qi2-bi2-ma\n", "1005 siblings of qi2-bi2-ma\n", "1005 siblings of qi2-[bi2]-ma\n", " 128 siblings of {d}utu u3 {d}marduk li-ba-al-li-t,u2-ka\n", " 128 siblings of {d}utu u3 {d}marduk li-ba-al-li-t,u2-ka\n", " 128 siblings of {d}utu u3 {d}marduk li-ba-al#-li#-t,u2#-[ka]\n", " 128 siblings of [{d}utu u3 {d}]marduk# li#-[ba]-al#-[li]-t,u2#-ka#\n" ] } ], "source": [ "for (l, paras) in rankedParallels[1000:1010]:\n", " print(f'{len(paras):>4} siblings of {T.text(l)}')" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 128 siblings of _{d}utu_ u3 _{d}marduk_ li-ba-al-li-t,u2-ka\n", " 128 siblings of _{d}utu_ u3 _{d}marduk_ li-ba-al#-li#-t,u2#-ka#\n", " 128 siblings of _{d}utu_ u3 _{d}marduk_ li-ba-al-li-t,u2-ka#\n", " 127 siblings of {d}utu u3 {d}marduk li-ba-al-li-t,u2-ka!(KI)\n", " 127 siblings of {d}utu u3 {d}marduk tu-ba-al-li-t,u2-ka\n", " 125 siblings of um-ma ha-am-mu-ra-pi2-ma\n", " 125 siblings of um-ma ha-am-mu-ra-pi2-ma\n", " 125 siblings of um-ma ha-am-mu-ra-pi2-ma\n", " 125 siblings of um-ma ha-am-mu#-[ra]-pi2-ma\n", " 125 siblings of um-ma ha-am-mu#-[ra-pi2-ma]\n" ] } ], "source": [ "for (l, paras) in rankedParallels[1130:1140]:\n", " print(f'{len(paras):>4} siblings of {T.text(l)}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And how many lines have just one correspondence?\n", "\n", "We look at the tail of rankedParallels." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "There are 1994 exclusively parallel pairs of lines\n" ] } ], "source": [ "pairs = [(x, list(paras)[0]) for (x, paras) in rankedParallels if len(paras) == 1]\n", "print(f'There are {len(pairs)} exclusively parallel pairs of lines')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Why not make an overview of exactly how wide-spread parallel lines are?\n", "\n", "We count how many lines have how many parallels." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1990 lines have n sisters where n > 100\n", " 939 lines have n sisters where 20 < n <= 50\n", " 775 lines have n sisters where 10 < n <= 20\n", "2646 lines have n sisters where 2 < n <= 10\n", "1994 lines have n sisters where n <= 2\n" ] } ], "source": [ "parallelCount = collections.Counter()\n", "\n", "buckets = (2, 10, 20, 50, 100)\n", "\n", "bucketRep = {}\n", "prevBucket = None\n", "for bucket in buckets:\n", " if prevBucket is None:\n", " bucketRep[bucket] = f' n <= {bucket:>3}'\n", " elif bucket == buckets[-1]:\n", " bucketRep[bucket] = f' n > {bucket:>3}'\n", " else:\n", " bucketRep[bucket] = f'{prevBucket:>3} < n <= {bucket:>3}'\n", " prevBucket = bucket\n", "\n", "for (l, paras) in rankedParallels:\n", " clusterSize = len(paras) + 1\n", " if clusterSize > buckets[-1]:\n", " theBucket = buckets[-1]\n", " else:\n", " for bucket in buckets:\n", " if clusterSize <= bucket:\n", " theBucket = bucket\n", " break\n", " parallelCount[theBucket] += 1\n", " \n", "for (bucket, amount) in sorted(\n", " parallelCount.items(),\n", " key=lambda x: (-x[0], x[1]),\n", "):\n", " print(f'{amount:>4} lines have n sisters where {bucketRep[bucket]}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Add parallels to the TF dataset\n", "\n", "We can add this information to the Oldbabylonian dataset as an *edge feature*.\n", "\n", "An edge feature links two nodes and may annotate that link with a value.\n", "\n", "For parallels, we link each line to each of its parallel lines and we annotate that link with the similarity between\n", "the two lines. The similarity is a percentage, and we round it to integer values.\n", "\n", "If *n1* is similar to *n2*, then *n2* is similar to *n1*.\n", "In order to save space, we only add such links once.\n", "\n", "We can then use\n", "[`E.sim.b(node)`](https://annotation.github.io/text-fabric/Api/Features/#edge-features)\n", "to find all nodes that are parallel to node.\n" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "metaData = {\n", " '': {\n", " 'name': 'AbB Old Babylonian Cuneiform',\n", " 'editor': 'Cale Johnson et. al.',\n", " 'institute': 'CDL',\n", " 'converters': 'Cale Johnson, Dirk Roorda', \n", " },\n", " 'sim': {\n", " 'valueType': 'int',\n", " 'edgeValues': True,\n", " 'description': 'similarity between lines, as a percentage of the common material wrt the combined material',\n", " },\n", "}" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "simData = {}\n", "for ((f, t), d) in similarity.items():\n", " simData.setdefault(f, {})[t] = d" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "ghBase = os.path.expanduser('~/github')\n", "subdir = 'parallels'\n", "path = f'{A.org}/{A.repo}/{subdir}/tf'\n", "location = f'{ghBase}/{path}'\n", "module = A.version" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " | 0.53s T sim to /Users/dirk/github/Nino-cunei/oldbabylonian/parallels/tf/1.0.4\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "TF.save(edgeFeatures=dict(sim=simData), metaData=metaData, location=location, module=module)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Turn the parallels feature into a module\n", "\n", "The new `sim` feature is a big data feature. You do not want to load it all the time.\n", "\n", "Here we show how to turn it into a module, so that users can easily load it in a Jupyter notebook or in the TF browser." ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n", "Create release data for Nino-cunei/oldbabylonian/parallels/tf\n", "zip files end up in /Users/dirk/Downloads/Nino-cunei-release/oldbabylonian\n", "zipping Nino-cunei/oldbabylonian 1.0.1 with 1 features ==> parallels-tf-1.0.1.zip\n", "zipping Nino-cunei/oldbabylonian 1.0.4 with 1 features ==> parallels-tf-1.0.4.zip\n" ] } ], "source": [ "%%bash\n", "text-fabric-zip 'Nino-cunei/oldbabylonian/parallels/tf'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I have added this file to a new release of the Oldbabylonian Github repo." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Use the parallels module\n", "\n", "We load the Oldbabylonian corpus again, but now with the parallels module." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TF app is up-to-date.\n", "Using annotation/app-oldbabylonian commit 82fffd1ec080e9aada4018099c1ad7686317fb2c (=latest)\n", " in /Users/dirk/text-fabric-data/__apps__/oldbabylonian.\n", "No new data release available online.\n", "Using Nino-cunei/oldbabylonian/tf - 1.0.1 rv1.1 (=latest) in /Users/dirk/text-fabric-data.\n", "No new data release available online.\n", "Using Nino-cunei/oldbabylonian/parallels/tf - 1.0.1 rv1.1 (=latest) in /Users/dirk/text-fabric-data.\n" ] }, { "data": { "text/html": [ "Documentation: OLDBABYLONIAN Character table Feature docs oldbabylonian API Text-Fabric API 7.5.0 Search Reference
Loaded features:\n", "

Old Babylonian Letters 1900-1600: Cuneiform tablets : after afterr afteru atf atfpost atfpre col collated collection comment damage det docnote docnumber excised face flags fraction grapheme graphemer graphemeu lang langalt ln lnc lnno missing object operator operatorr operatoru otype pnumber primecol primeln question reading readingr readingu remarkable remarks repeat srcLn srcLnNum srcfile supplied sym symr symu trans translation@ll type uncertain volume oslots

Nino-cunei/oldbabylonian/parallels/tf: sim

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
API members:\n", "C Computed, Call AllComputeds, Cs ComputedString
\n", "E Edge, Eall AllEdges, Es EdgeString
\n", "ensureLoaded, TF, ignored, loadLog
\n", "L Locality
\n", "cache, error, indent, info, reset
\n", "N Nodes, sortKey, sortKeyTuple, otypeRank, sortNodes
\n", "F Feature, Fall AllFeatures, Fs FeatureString
\n", "S Search
\n", "T Text
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "A = use('oldbabylonian', hoist=globals(), check=True, mod='Nino-cunei/oldbabylonian/parallels/tf')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lo and behold: you see the parallels module listed with one feature: `sim`. It is in *italics*, which indicates\n", "it is an edge feature.\n", "\n", "We just do a quick check here and in another notebook we study parallels a bit more, using the feature `sim`.\n", "\n", "We count how many similar pairs their are, and how many 100% similar pairs there are." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 1.55s 574579 results\n" ] } ], "source": [ "query = '''\n", "line\n", "-sim> line\n", "'''\n", "results = A.search(query)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 1.58s 573397 results\n" ] } ], "source": [ "query = '''\n", "line\n", "-sim=100> line\n", "'''\n", "results = A.search(query)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Remarkably, most of the pairs are 100 percent similar. Let's show just a few:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
nplineline
1 P510729 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 a-na {d}suen-i-din-[nam] 235393
2 P510730 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 a-na {d}suen-i-din-nam 235420
3 P510731 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 a-na {d}suen-i-din-nam 235433
4 P510732 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 a-na {d}suen#-i-din-nam# 235463
5 P497779 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 a-na {d}suen#-[i]-din-nam 235477
6 P510733 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 [a-na] {d}[suen-i-din-nam] 235502
7 P510734 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 [a-na {d}suen-i-din-nam] 235528
8 P510736 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 a-na {d}suen-i-din-nam 235584
9 P510737 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 a-na {d}suen-i-din-nam# 235614
10 P370926 obverse:1[a-na] _{d}suen_-i-[din-nam] 230787 a-na {d}suen-i-din-nam 235628
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "A.table(results, start=1, end=10, withNodes=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is also a lower level way to work with edge features.\n", "\n", "We can list all edges going out from a reference node.\n", "What we see is tuple of pairs: the target node and the similarity between the reference node and that target node." ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((235919, 100),\n", " (235983, 100),\n", " (235998, 100),\n", " (236046, 100),\n", " (236063, 100),\n", " (236077, 100),\n", " (236115, 100),\n", " (236123, 100),\n", " (236195, 100),\n", " (236222, 100),\n", " (236277, 100),\n", " (236320, 100),\n", " (243205, 100),\n", " (244633, 100),\n", " (245089, 100),\n", " (246839, 100),\n", " (246870, 100),\n", " (246892, 100),\n", " (246911, 100),\n", " (246924, 100),\n", " (246948, 100),\n", " (246961, 100),\n", " (247114, 100),\n", " (247136, 100),\n", " (247235, 100),\n", " (247263, 100),\n", " (247272, 100),\n", " (247317, 100),\n", " (247345, 100),\n", " (247355, 100),\n", " (247371, 100),\n", " (247386, 100),\n", " (247415, 100),\n", " (247436, 100),\n", " (247459, 100),\n", " (247542, 100),\n", " (247696, 100),\n", " (247708, 100),\n", " (247733, 100),\n", " (247749, 100),\n", " (247794, 100),\n", " (247806, 100),\n", " (247829, 100),\n", " (247867, 100),\n", " (247885, 100),\n", " (248014, 100),\n", " (248041, 100),\n", " (248118, 100),\n", " (248172, 100),\n", " (248381, 100),\n", " (248407, 100),\n", " (254854, 100),\n", " (256079, 100))" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "refNode = 235900\n", "\n", "E.sim.f(refNode)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Likewise, we can observe the nodes that target the reference node:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((230787, 100),\n", " (235393, 100),\n", " (235420, 100),\n", " (235433, 100),\n", " (235463, 100),\n", " (235477, 100),\n", " (235502, 100),\n", " (235528, 100),\n", " (235584, 100),\n", " (235614, 100),\n", " (235628, 100),\n", " (235656, 100),\n", " (235683, 100),\n", " (235729, 100),\n", " (235755, 100),\n", " (235777, 100),\n", " (235810, 100),\n", " (235821, 100),\n", " (235837, 100),\n", " (235850, 100),\n", " (235864, 100),\n", " (235878, 100))" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "E.sim.t(refNode)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Both sets of nodes are similar to the reference node and it is inconvenient to use both `.f()` and `.t()` to get the similar lines.\n", "\n", "But there is another way:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((230787, 100),\n", " (235393, 100),\n", " (235420, 100),\n", " (235433, 100),\n", " (235463, 100),\n", " (235477, 100),\n", " (235502, 100),\n", " (235528, 100),\n", " (235584, 100),\n", " (235614, 100),\n", " (235628, 100),\n", " (235656, 100),\n", " (235683, 100),\n", " (235729, 100),\n", " (235755, 100),\n", " (235777, 100),\n", " (235810, 100),\n", " (235821, 100),\n", " (235837, 100),\n", " (235850, 100),\n", " (235864, 100),\n", " (235878, 100),\n", " (235919, 100),\n", " (235983, 100),\n", " (235998, 100),\n", " (236046, 100),\n", " (236063, 100),\n", " (236077, 100),\n", " (236115, 100),\n", " (236123, 100),\n", " (236195, 100),\n", " (236222, 100),\n", " (236277, 100),\n", " (236320, 100),\n", " (243205, 100),\n", " (244633, 100),\n", " (245089, 100),\n", " (246839, 100),\n", " (246870, 100),\n", " (246892, 100),\n", " (246911, 100),\n", " (246924, 100),\n", " (246948, 100),\n", " (246961, 100),\n", " (247114, 100),\n", " (247136, 100),\n", " (247235, 100),\n", " (247263, 100),\n", " (247272, 100),\n", " (247317, 100),\n", " (247345, 100),\n", " (247355, 100),\n", " (247371, 100),\n", " (247386, 100),\n", " (247415, 100),\n", " (247436, 100),\n", " (247459, 100),\n", " (247542, 100),\n", " (247696, 100),\n", " (247708, 100),\n", " (247733, 100),\n", " (247749, 100),\n", " (247794, 100),\n", " (247806, 100),\n", " (247829, 100),\n", " (247867, 100),\n", " (247885, 100),\n", " (248014, 100),\n", " (248041, 100),\n", " (248118, 100),\n", " (248172, 100),\n", " (248381, 100),\n", " (248407, 100),\n", " (254854, 100),\n", " (256079, 100))" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "E.sim.b(refNode)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's make sure that `.b()` gives the combination of `.f()` and `.t()`." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "the intersection of f and t is set()\n", "t | f = b ? True\n" ] } ], "source": [ "f = {x[0] for x in E.sim.f(refNode)}\n", "b = {x[0] for x in E.sim.b(refNode)}\n", "t = {x[0] for x in E.sim.t(refNode)}\n", "\n", "# are f and t disjoint ?\n", "\n", "print(f'the intersection of f and t is {f & t}')\n", "\n", "# is b the union of f and t ?\n", "\n", "print(f't | f = b ? {f | t == b}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.2" } }, "nbformat": 4, "nbformat_minor": 2 }