<p><span style="text-align: left; color: #ff6600; font-family: Verdana,Times,serif; font-size: xx-large;">Evaluation</span></p>
<p style="text-align: justify;">For the evaluation of the results of task A and B different strategies and metrics are applied in order to allow for more fine-grained scores.</p>
<p><span style="font-family: Verdana;"><strong>TASK A.</strong><br /> </span></p>
<p><span style="font-family: Verdana;">Systems will be evaluated using standard evaluation metrics, including accuracy, precision, recall and F1-score. The submissions will be ranked by F1-score.<br /> The metrics will be computed as follows:</span></p>
<ul>
<li><em>Accuracy</em> = (number of correctly predicted instances) / (total number of instances)</li>
<li><em>Precision</em> = (number of correctly predicted instances) / (number of predicted labels)</li>
<li><em>Recall</em> = (number of correctly predicted labels) / (number of labels in the gold standard)</li>
<li><em>F1-score</em> = (2 * Precision * Recall) / (Precision * Recall)</li>
</ul>
<h4 style="text-align: justify;"><strong>TASK B.</strong></h4>
<p style="text-align: justify;">Systems will be evaluated on the basis of two criteria: partial match and exact match.</p>
<ul>
<li><strong>Partial match</strong>: <span>each dimension to be predicted (Hate Speech HS, Target TR and Aggressiveness AG) will be evaluated independently of the others using standard evaluation metrics, including&nbsp;</span><em>accuracy</em><span>,&nbsp;</span><em>precision</em><span>,&nbsp;</span><em>recall</em><span>and&nbsp;</span><em>F1-score</em><span>&nbsp;as defined above. The report for each participant will include all the measures and a summary of the performance in terms of macro-average&nbsp;</span><em>F1-score</em><span>, computed as follows</span>:<br /> <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAA2CAYAAADpjc+LAAAMKGlDQ1BJQ0MgUHJvZmlsZQAASImVlwdUk8kWgOcvSUhIaIFQpITeROlVeg0gIFWwEZJAQokhIYjYkUUF1oKKBSu6KmJbCyCLDQsWFsHeHxZUlHWxYAPlTRJAV89777x7zvz/lzt37tx7M/+cGQBUY9giUTaqBkCOME8cGxrAnJicwiQ9AghQBirAHBiyORKRf0xMJIAy/P6nvL8BraFctZP5+rn/v4o6lyfhAIDEQE7jSjg5kA8DgLtyROI8AAg9UG86I08EmQijBJpiGCBkMxlnKNhdxmkKjpTbxMcGQk4FQInKZoszAFCRxcXM52RAPyrlkO2FXIEQchNkHw6fzYU8AHl0Ts50yKpWkK3SvvOT8Q+faSM+2eyMEVbkIhelIIFElM2e+X+W439LTrZ0eA5T2Kh8cVisLGdZ3bKmR8iYCvm8MC0qGrIG5GsCrtxexk/50rCEIfuPHEkgrBlgAIBSueygCMj6kE2E2VGRQ3qfdEEICzKsPRovyGPFK8aiXPH02CH/aAFPEhw3zGyxfC6ZTak0K8F/yOcmPo817LOxkB+fpIgTbc8XJEZBVoF8T5IVFzFk86KQHxg1bCOWxspihv85BtLFIbEKG8wsRzKcF+bJF7Cihjgyjx8fphiLTeWw5bHpQM7kSSZGDsfJ5QUFK/LCinjChKH4sQpRXkDskP12UXbMkD3WxMsOlelNILdJ8uOGx/bmwcWmyBcHoryYeEVsuGYmOzxGEQNuAyJBIAgCTCCFLQ1MB5lA0NZT3wN/KXpCABuIQQbgAbshzfCIJHmPED7jQCH4CxIPSEbGBch7eSAf6r+MaBVPO5Au782Xj8gCTyHngAiQDX9L5aOEI7MlgidQI/hpdg6MNRs2Wd9POqbqsI4YTAwihhFDiNa4Hu6De+GR8OkHmyPujnsMx/XNnvCU0EF4RLhO6CTcniYoEv8QOROMB50wxpCh7NK+zw63gF5d8ADcG/qHvnEGrgfscGc4kz/uC+d2gdrvY5WOZPytlkO+yPZklKxN9iNb/RiBio2Ky4gXWaW+r4UirrSRagWO9PyYR+B39ePCd8SPlthi7BDWgp3CLmBNWD1gYiewBqwVOybjkbXxRL42hmeLlceTBf0IfpqPPTSnrGoS+1r7bvuBoT6QxyvIk30sgdNFM8WCDH4e0x/u1jwmS8gZM5rpaO8Ad1HZ3q/YWt4y5Hs6wrj4TZd7EgCPUqjM+KZjwz3o6FMA6O+/6UzfwGW/HIBj7RypOF+hw2UPAqAAVfil6AJDuHdZwYwcgSvwAn4gGISDaBAPksFUWGc+XKdiMAPMBgtACSgDy8FqsB5sBtvALrAXHAT1oAmcAufAJdAOroO7cK10gZegF7wH/QiCkBAaQkd0ESPEHLFFHBF3xAcJRiKRWCQZSUUyECEiRWYjC5EypAJZj2xFapDfkaPIKeQC0oHcRh4i3cgb5DOKoVRUEzVALdCxqDvqj0ag8egUNAPNRQvRYnQpuhatRvegdegp9BJ6He1EX6J9GMCUMQZmjNlh7lggFo2lYOmYGJuLlWKVWDW2D2uE//RVrBPrwT7hRJyOM3E7uF7D8AScg+fic/FyfD2+C6/Dz+BX8Yd4L/6VQCPoE2wJngQWYSIhgzCDUEKoJOwgHCGchd9OF+E9kUhkEC2JbvDbSyZmEmcRy4kbifuJJ4kdxMfEPhKJpEuyJXmToklsUh6phLSOtId0gnSF1EX6qKSsZKTkqBSilKIkVCpSqlTarXRc6YrSM6V+shrZnOxJjiZzyTPJy8jbyY3ky+Qucj9FnWJJ8abEUzIpCyhrKfsoZyn3KG+VlZVNlD2UJygLlOcrr1U+oHxe+aHyJ6oG1YYaSJ1MlVKXUndST1JvU9/SaDQLmh8thZZHW0qroZ2mPaB9VKGrjFFhqXBV5qlUqdSpXFF5pUpWNVf1V52qWqhaqXpI9bJqjxpZzUItUI2tNletSu2o2k21PnW6uoN6tHqOern6bvUL6s81SBoWGsEaXI1ijW0apzUe0zG6KT2QzqEvpG+nn6V3aRI1LTVZmpmaZZp7Nds0e7U0tJy1ErUKtKq0jml1MjCGBYPFyGYsYxxk3GB81jbQ9tfmaS/R3qd9RfuDzigdPx2eTqnOfp3rOp91mbrBulm6K3Trde/r4Xo2ehP0Zuht0jur1zNKc5TXKM6o0lEHR93RR/Vt9GP1Z+lv02/V7zMwNAg1EBmsMzht0GPIMPQzzDRcZXjcsNuIbuRjJDBaZXTC6AVTi+nPzGauZZ5h9hrrG4cZS423GrcZ95tYmiSYFJnsN7lvSjF1N003XWXabNprZmQ23my2Wa3ZHXOyubs533yNeYv5BwtLiySLRRb1Fs8tdSxZloWWtZb3rGhWvla5VtVW16yJ1u7WWdYbrdttUBsXG75Nlc1lW9TW1VZgu9G2YzRhtMdo4ejq0TftqHb+dvl2tXYPxzDGRI4pGlM/5tVYs7EpY1eMbRn71d7FPtt+u/1dBw2HcIcih0aHN442jhzHKsdrTjSnEKd5Tg1Or51tnXnOm5xvudBdxrsscml2+eLq5ip23efa7Wbmluq2we2mu6Z7jHu5+3kPgkeAxzyPJo9Pnq6eeZ4HPf/2svPK8trt9Xyc5TjeuO3jHnubeLO9t3p3+jB9Un22+HT6Gvuyfat9H/mZ+nH9dvg987f2z/Tf4/8qwD5AHHAk4EOgZ+CcwJNBWFBoUGlQW7BGcELw+uAHISYhGSG1Ib2hLqGzQk+GEcIiwlaE3WQZsDisGlZvuFv4nPAzEdSIuIj1EY8ibSLFkY3j0fHh41eOvxdlHiWMqo8G0azoldH3YyxjcmP+mECcEDOhasLTWIfY2bEtcfS4aXG7497HB8Qvi7+bYJUgTWhOVE2cnFiT+CEpKKkiqXPi2IlzJl5K1ksWJDekkFISU3ak9E0KnrR6Utdkl8klk29MsZxSMOXCVL2p2VOPTVOdxp52KJWQmpS6O3WAHc2uZvelsdI2pPVyAjlrOC+5ftxV3G6eN6+C9yzdO70i/XmGd8bKjG6+L7+S3yMIFKwXvM4My9yc+SErOmtn1mB2Uvb+HKWc1JyjQg1hlvDMdMPpBdM7RLaiElFnrmfu6txecYR4hwSRTJE05GnCQ3ar1Er6i/Rhvk9+Vf7HGYkzDhWoFwgLWmfazFwy81lhSOFvs/BZnFnNs41nL5j9cI7/nK1zkblpc5vnmc4rntc1P3T+rgWUBVkL/iyyL6ooercwaWFjsUHx/OLHv4T+UluiUiIuubnIa9HmxfhiweK2JU5L1i35WsotvVhmX1ZZNlDOKb/4q8Ova38dXJq+tG2Z67JNy4nLhctvrPBdsatCvaKw4vHK8SvrVjFXla56t3ra6guVzpWb11DWSNd0ro1c27DObN3ydQPr+euvVwVU7d+gv2HJhg8buRuvbPLbtG+zweayzZ+3CLbc2hq6ta7aorpyG3Fb/ran2xO3t/zm/lvNDr0dZTu+7BTu7NwVu+tMjVtNzW793ctq0VppbfeeyXva9wbtbdhnt2/rfsb+sgPggPTAi99Tf79xMOJg8yH3Q/sOmx/ecIR+pLQOqZtZ11vPr+9sSG7oOBp+tLnRq/HIH2P+2Nlk3FR1TOvYsuOU48XHB08Unug7KTrZcyrj1OPmac13T088fe3MhDNtZyPOnj8Xcu50i3/LifPe55sueF44etH9Yv0l10t1rS6tR/50+fNIm2tb3WW3yw3tHu2NHeM6jl/xvXLqatDVc9dY1y5dj7recSPhxq2bk2923uLeen47+/brO/l3+u/Ov0e4V3pf7X7lA/0H1f+y/tf+TtfOYw+DHrY+int09zHn8csnkicDXcVPaU8rnxk9q3nu+LypO6S7/cWkF10vRS/7e0r+Uv9rwyurV4f/9vu7tXdib9dr8evBN+Vvdd/ufOf8rrkvpu/B+5z3/R9KP+p+3PXJ/VPL56TPz/pnDJAG1n6x/tL4NeLrvcGcwUERW8yWHwUw2ND0dADe7ASAlgzPDu0AUCYp7mZyQRT3STmB/8SK+5tcXAHY6QdAwnwAIuEZZRNs5pCp8C07gsf7AdTJaaQNiSTdyVHhiwpvLISPg4NvDQAgNQLwRTw42L9xcPDLdhjsbQBO5iruhDKR3UG32Mjo8jjd+eAH+TeFKXA6yHNX8wAAAgtpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iPgogICAgICAgICA8dGlmZjpSZXNvbHV0aW9uVW5pdD4yPC90aWZmOlJlc29sdXRpb25Vbml0PgogICAgICAgICA8dGlmZjpDb21wcmVzc2lvbj4xPC90aWZmOkNvbXByZXNzaW9uPgogICAgICAgICA8dGlmZjpPcmllbnRhdGlvbj4xPC90aWZmOk9yaWVudGF0aW9uPgogICAgICAgICA8dGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0aW9uPjI8L3RpZmY6UGhvdG9tZXRyaWNJbnRlcnByZXRhdGlvbj4KICAgICAgPC9yZGY6RGVzY3JpcHRpb24+CiAgIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+Cg9FKpMAABcfSURBVHgB7Z0FkB1FE8cHdwjuBAiuwQkuRVFAcC8IhRRSSAWHYIHCisLdrXB3dw0OwT1IcHdnv/71R2/N7c2TfXfZy+V1V+3bfTv+39npme6e3rEyoeDkCDgCjoAj4AiURGDskvE9uiPgCDgCjoAjoAg4A/GO4Ag4Ao6AI9ASAs5AWoLNEzkCjoAj4Ag4A/E+4Ag4Ao6AI9ASAs5AWoLNEzkCjoAj4Ag4A/E+4Ag4Ao6AI9ASAs5AWoLNEzkCjoAj4Ag4A/E+4Ag4Ao6AI9ASAs5AWoLNEzkCjoAj4Ag4A/E+MMYj8Pvvv1fWxirLqqJRv/76axXFhN9++62ScqospKq+UFU5KezGTd30e45AVQiMHDkyHHPMMWHGGWfUIv/888/w888/6/VYY40V+vTpE/76668wePDgMM0005Sq1h9//BE233zzsMYaa4TddttN05511lnhrbfeyvP6/vvvw4orrhg22GCD8Mgjj4Qbb7wxD/vss8/CHnvsEeaff/68XOp76KGHhksuuSRMNtlk4eCDDw4TTjih1n+99dYLSyyxRDjjjDPCqquumqcZVRe0b8iQIYoRWEHffvttXtzEE0+s15tuumno379/fr/ZC7C67rrrwgMPPBDGHrvzXBMvSFtssUXYb7/9wpJLLlk3W/I4/fTTwy233KLxwO7II48Mq6yySjjhhBPC6quvHq6//no9wHNUU5X97pVXXglXX321YghDpn3WRp7XRBNNpMdcc80Vttxyy7zp4P/xxx8He470VTCfc845w4ABA7SvwTyq7HN55ewCX1hOjkBPIfD3339nH3zwQbbAAgvgky3bdddds2HDhmVPPfVUdtNNN2Xbbrut3v/www9LVVGYULbUUktlhxxySPbvv//maX/55Zfstttu0zwp74Ybbshk9qvhwqiyBx98MA+76qqr8jAiyMusYfICZy+88EL2ySefZFtvvbXeIy705ZdfZtNNN52WoTdG8c93332X7bjjjloH2itMULG75557ssMOO0zv33nnnaVqAV4HHHBANs8882Qy4NVM+8QTT2j+t956a804n376qT4HsBbGm3300UeZDITZiy++mD9zypGJQyYDZsMyaxZUMqDKficTJMVp6NChGc9i+eWX1//0o4cffji74IIL9P8OO+zQoa8Kc8hkMqJh9CmeKe8E14Ynz6rqPhdDDUdzcgR6FAEGcGMgsjroUBfCGg1kHRLIH16qjTfeOFt33XUzBooi3XffffoCEv7PP/90CH7jjTc0jJebQS2mU045RcOeffbZ/LbVXWaZ+T0bWIcPH57fG5UXMF0GFGNicVlrrrlmVsQ0Dk9dM5AzSH311VepYL3H4AbDotyTTjopGe+dd97RcOLAqIsEk19kkUWUyRPGs5AVScZAWnwuxbTd8d+eHfUrYtRd/Y4+RBsfffTRvMowZ8o8//zz83unnXZaJqu9/L9dXHHFFRqXSYKR9VGekaxg9HbVfc7q4gzEkPBzjyHw8ssv60vCoF2c8TJQXXbZZR1mZo0qescdd2h+b7/9djKqiFw0/NJLL+0Ufs4552gYL3lMMCUGNl78u+66Kw7K1lprreybb77J7xkDE9FYxqqmGaKu5F2rzrXy+OGHHzJwq5X2mmuuyRiom6URI0ZoXiJOqpuEZ0KZHJtttlmnAZ962aTgxBNPrJnXqaeeqrNqi/DMM89onqykmiUG3rKTDPKuot/BiC666KK8KTwLw+XVV1/N77OKe//99/P/XNCPBg0apHjEqzx7RjEDaaXPdSisxT/OQFoEzpN1HwI2aCNyMWJGVWtma3FSZ2Z8iAhqDd72AosMPnvvvfc6ZMFLuP766+sLe//993cI48/FF1+sYaRldm2EmIa0MSHiYnBFHNcMGQOJVzfNpLNyaLMxK2alrMBSq69GeSJmYWBCLFKLWJnQtttvv10HwxTjt9VaKizOl3J+/PHH/JYNhLVWj3nE6AIGEg+mUVDdyyr7nVWkHtOyOHbmOdIuDvqY0eWXX674F7G1vtBsn7P8unLurBmTnuHkCFSFgHTeIDN6LW7ccccNMqiHm2++OciAqMrvsvUQfUoQ5qPKc/Irksjgw+uvvx5++umnQNyXXnopiDxeDxEzBHn5grywYcEFFywmDRtttFGQ2aOmpX4oYiEMAEyJbYlmmGEGvRT5tio+7X53n6kzJIO6KlxFph5EbBUGDhwYxhlnnFLFoZBFab7MMsuEqaaaqmZalN8YEsjKK4CxMOXc8IFEopcJIpLR9Cj5URLXommnnVaNESwcHFHIi55KjR3sfnefq+53Vv8nn3xSL9dZZ526uBBJRFVBGKzGR4H+2muvhQMPPDCI3k3v8axibKvqc1r4fz+d37A41K8dgVGMwNdff62DNsXICkQPrkUkEeadd14uS5HM1OrGf/rpp/NwLH9SJLPfwMBWpMknn1ytiOaee259sVdeeeXw3HPPhSmnnLIYVe+J7FuZDYNVkcEUEzQKL8bnv8zWldlyLSsYtc7hGgsn6laWGKR4Hv369auZVMQualEkKzC1KqIcrKvEyCHHjDxEHKP1gJkV6aGHHlKGM8EEE2gQ5a6wwgphpplm0v/zzTefnsuaEJdhmFX3OxoUMy0sAxuRTQ5gIkxcoFlmmSWIIl6t12LmQRj9sEyfI01XyVcgXUXQ03cJAVnS62DMoC0iF33JRMmtK4jxxhuvdN7MfqGUWSkvMKsbiFUP/+PjiCOO0DAGvVqDEaaWIirQeAyS++67bxCFr/5P/bDCwdy2SO+++26YddZZw9prr60H5saQKKbze4RbWcX0/P/iiy8CKw5WTJiD0hYYKCsi0pYlVhIMVrXaD8PCrBnTapHDa92sfrTHCGYCTTHFFHrYfTtPPfXUyogph4PVYDwY0g8gzKhThCkwzNywO/bYY7XeDK52DyZofSGVR5X9zsoXPZlOlmDwqf5p8TiDgYgI9RZMW0Sbet23b19dmcd4aUD0U6vPRVG67dIZSLdB6Rm1gsBjjz2myeJBSxSwOhA0yk8UlDr4xPFsv4KY2Ma39dpeYAbc4r4IxDf2wjIbjokBzsRV3F9sscUCTA4ijcjw9Tr1Y4NhKkz0NcoEYARGiy++eH6PcAabWsRAASFyYkCFYALseUmJ7zRC9BOXy21LA3OAGRXp7rvvDs8//7zum2HvznHHHZdHSa38wEzMpvM4dsEs2TBm1SnGCclVHPt/6hH157Bnw8rQ7vGM601Aqux31gZEUDBo+l5qhWvxONMmRLEwRSYDCy20kK4uuGfPPY4fX9frc3G8brmWjuLkCPQIAlioYOIoHTmTl6LpOmCZZQrtosIQu3ryw36+SBaWUrCbEhvFZKzUJQ9MdA8//PAO2aHsNQsZGRg6hPHH2tasMthMXmWA7pRXrRspc9BaceP7sqlPLbdkA198O5MZvypsU3U2ay/SxmT1Jo2Z3hqWMojnZqZxGlOU85xS1lYoxQlr1qDAlOjUsRmyZ1NVv7M6YaBAmc0Yh7Dfg7jx3hAzTMCKMEXWrtTzS8XvjnulViBijqayXOS1zR4sJZkVOTkCRQSYZSFKkEE7l98X46T+IxtnFks6mzVbPJOjm3zd7nNG9g6J2WmndCYiQAQy6aSTajz7efPNN0NqRUM4M13bRW/xOSPWYiaIqKEZ/Ya8zJocMVEzxIrJjA+WW265ZpJoHOpEfVlF2W5oS4xoit3+k0wySac6n3nmmRp/pZVWsuh6BiswYHZtorrZZptNRXHMtmXzW4f4/EHnwQqAdPEuf4vIyguy1aTdb3SuJ0qM01bd7yib1RS6CwjPB43IVrjoSqz/YBgBifl5UjxXts81qkMz4aUYyDbbbBMQA2BBYMQSC7ny448/rodwznD00Udr5yDO0ksvnW/FtzR+dgRAANcVRvVkuhbHzigLUSoWB0DCWe6jR8CaKh6MkYdjtQIV5c+IoGR2p2GLLrpo/sLqDflhAiSbvtQyyO6he5C9EMqMGHiLhFIbay8sZsoOhMW8Uv8xBoD5QqnyU2m4B8Nl0E6lAU+Zvep7jBjRCLHJQQcdFGQ/RyfGi5sNmA76IERfEMz7wgsv1GtcczA+GIOEgTERhbmQrmjtRTzcyWBEkWIummkXf6rsd1ZVMGSSAtOsZ6RAfBic1TGOy4RJzMwVO9mYaVnn51Hd5/KC4otWljG281XySe4wJU+Wk2yYSYkSWinT04xZCODWgv5jx7LLLpuJrqHpRrJcR9yUEnOIklXzlUFN8zMXJFYWZzbCQbiWiO9zLYriXBxDHDbKIRYoxhMfUzU36eG+QiZXncRh5JciE/uIVVcquMM92yhp9WFfSkqM1iFR4Q+ikdS7KUxA22kb10Q5nrebcuLd9TJpzMOsLnE4uIORhcXn3XffPSPvIrHfgXipTZ7FuPbfRFi2K9vup85V9jvKR9wqE+5OGIhvrFT1MsM/xgr3OkbxsxfjAbut57J9rkPiFv8wMyhFJgulgXSoej6KAA65s5Mj0N0I1GMgbKhjY10tWXHZutjAhB8tdg/jpyveSFjMzwaB4o71Yrye/F+LgVAnBiJ2drMpszsIpgBuuOCQvTcd/IsV8zcfXN1VdjH/rv6vst+VqWtP9bnSDMR2O8JA4t2vqcayY/Lzzz9PBfk9R6BLCNR7kclYRCSqEG7kkqNLlUgktpV3PfcdiWSV36rHQFCGsyOfVQIK76ro2muv1WfGymV0pdGx3/VknyulAxGmocovzhA7c2MlJvLN448/Pjfd22qrrcL000///8i9+Bf5Li6ZsXWnjez6lZlUzRZhk8+OakwbhYHWjGcB8sLorlPcjLNBqxahPEbOGZdN3TDrq2fySP7oAGSW0kEvUKucMeE+ZpLgsv/++wdxMlhJk3gGKEgpc++9966kzFFRCDobZPAyMIWdd94511+MirIsT/Z2sMsavQ561d5KVfe7Hu9zZTg9MxO8ZcrD1aMofxbFmS59ESGMCYTIQr53kLfX2s35yiuv7NRElugbbrhhp/j4eAK7IuGhEzk++eG7yK4R/cmAn0fH7C8uGzNUYWSZDFL5/aJJJonNXJO06KM4I5pgFtXbifbTJtyC1yOxDKpsFUy/x8V7b6CddtopqQMp1h3361UQ/rVGV7FV3P7Rrd/1dJ8rJcIyBZcNZiju6GBiYZCh0OF+SjEXP4DedG0DN8t9GAAPa5dddtF2FpmneScVKwv18kk7Y3EfMmAjxAJHHXWU5oOi1eTpscKNMk18gNM/8jIGA8ay0tP09ixicQP13HPPPTUc5bRsrlJlrzGRRgpX6kM7ZFNbqQOX0ylGae3urjODDfsyaDtKWjy4OjWHAH2DPTT0JZ4vCmjrZ83l0L6xvN91fvalfGFh6x1TyuFcFV9iow6YBiJKKkvs2GUfgNlW10tv7hlwvcCynoMv0ImP/jDzzDPnSc8+++wglmlqoicK1jDHHHNoGGacRjLr10t5BOr+QpiT7jLG343Fx/xRZoZBLC3UZJQd0NRXXnYVC5rZKmaQiLpY7mNnj0kfJta0CRHbJptsoq4iSHfvvfdqOYi9qA87mznqEXlgZslRhmJMyqQrGxfzz6FDh+pRNm27x0fkLB/p0qPdsSjbfu93CcQ685Tad2znq2SjKw0U5DIwqaUVM5rutNyoXYv/h2ApQj3KHib+aZQ/4bG5MpY3EOIfZvUoriCxx8/rwJfuYkIEhstoFLksfSFbqVDvlIKXGaG1KV4pxKsZwotlWbnxM7L0rGwweCAd7WcWWo+wOuLrZ5gPNnsQv4z4xtro5/J92DFzzLqrD9QbB5oJa1qExcBpbieofPwtBZbA6EZScvhmKtFqHAbGVo5my7Mv19HelE0/mNhX2WCejQZmXEXYg69lwYZ4gTiIkIhvxFfiLC1YG0OycM72MR7imY04Fi2ym1XT0oY4zzht1dfWFj/7YOh9oOf6QFff+6YZiH1GkYedWmmg+G2k0OxqZatOD2O0DwzR7ljPQF3MtxJhqc+JFutreiLipz7xGZfHYG/7D4rGCymcYV62yiB/TKhjPYm4ouj0tb9i/ar8Tx39cAy8D/RsH+jqO9+0DoTvHhiJZVInT5cyc1bXBBanmTM6Bj5Mg7uTvfbaq5kklcZBpyAKa9UnUDDX6E8w1YRiHUwjtwvyoPIP5ODOwPLQjP77wQ025owQ3kpxhw2hi8B1BkT5Cy+8sF7HP3ghxV2CES40RKmt/nfIq5Hew9Jx5lsJD4nfqFouzeO48TV+kWLfPXFY8Ro8nBwBR6B3I9AUA+FlRxlrlPpYTSP3xJaWM3sS2CNizuDMtXMcpyev8TPEII2fH1kJBNngpAM3dWKfi8z0VaFuvohgCDiQq0fs4RARk0ZZbbXVAor5IuFHzAjGaj6U+B6A0XbbbZcc2OOP7+BsDaV6qwRjhFGVJZgiH2kqy3jKluPxex4BHB7iIwzDD5wvOrUnAk0xELxnmldNBsvUDLgsfOeee27AAsk+z1g2/aiKD7PEcRyf7TQvp7JHI4jeIV8FWNkwGCMsl4rEoM4nVO0LaxYuCnHd0GcMgvt4e7XNZzitE5Nbi67O7fgDM+PbDymKv32Q+pIf7YIx8r2J8ccfP5VFfg/rLTED1jo1Y61mCfGEWya+pfNz70GAfoRnXj4sZcR3QfbZZx+fOBggbXRuioEw0zaTTganogfNsnjhTZUj5XK7bF7dHZ9Bn93jiIyMgTDQwziNeIkgG+TBRjbtBUR7Rqyy+EiSWGzp96FhNnzwBlNaTCnjgVZ0H/oRINJuv/32YfDgwZZNECOBYCsT2aSY/PAOkYlndN5556nHWVsJ4G2WjwzRJnayNyKey8knn9womoe3IQJMrGAg9ElErkw0xPJP+xsra6c2Q0AGw7rE5kGzNBJodAOSyMgbWhzVzfS/QDNZHZ02H5pfGdoq3yzQmrKByCzQ4h3oKLmFseTKYHFDnWH1hE8f0gsz6ICTedLECSW70CFMfW1zIjvWi9ZVsfFC8WM+msF/P1ijxc+JHfF4mkV5Tl0wPRZX/HESv3YESiHAWIC5fuw12bwZN/ORpFKFeeRegUBdKywb8BiAigeDXVdpdGQgskLo0FaZVeX/2cOBpVRM7Nq2Hd4xRrhrLsYlHffjeFxjcYVr7BSZKTGMyqyyUvG4B/MyRmdlwKzAuYod4rXq5ffHDASYSBVdm4wYMUL7c2pP05jRam9FPQTGIlAGmx4hHLaJKwq1bkLEMjoQu7+xaJp99tlVbIcYiHvssq730SOW82JKq3FwIBk7mSy2CzEZlk5Aj+VSSqFeTFPmf1wX10uUQc7jlkGA/jtkyBDV0aEjdWV6GfTGjLhN6UDGjKY21wrMXc3k1T6P2oxH4b59+zZXgMRCH9LIaqvpzBIRy9QlkdxvOQINEUCXh/6DT6/yFcN6E6aGmXmEXotAaXfuvbalXnFHwBHoFgTwqyabYoOZjotbHfWthTGIU3sh4AykvZ63t9YR6DICiHdFH6iiK5gHhANQTNGd2gsBZyDt9by9tY5AtyIwcODAgGdp9ITOQLoV2l6RWY8ykHjvQq9AyyvpCDgCnRDo16+f6g1ZmTi1FwI9wkCQofItkUGDBina4qQwDBgwQD/r2l7we2sdgd6FAC5MijRs2LDQv3//0r7wivn4/96HQI+a8fY+uLzGjkD7IsBHzMQTt3pYkE83qKk7HzPDrcnw4cPzD6O1L0Lt1/IeWYG0H8zeYkeg9yPQp08fdd/Dng/84WHuLhsJw8iRI5159P7H21ILfAXSEmyeyBFoXwTE/Y5+OpmNg77/o337AS13BtLez99b7wg4Ao5Aywi4CKtl6DyhI+AIOALtjYAzkPZ+/t56R8ARcARaRsAZSMvQeUJHwBFwBNobAWcg7f38vfWOgCPgCLSMwP8ATzAmutN10QoAAAAASUVORK5CYIIA" alt="" width="235" height="31" /><br /><br /></li>
<li><strong>Exact match</strong>: all the dimensions to be predicted will be jointly considered computing the <em>Exact Match Ratio</em> (Kazawa, 2005). Given the multi-label dataset consisting of n multi-label samples (x<sub>i</sub>,Y<sub>i</sub>), where x<sub>i</sub> denotes the i-th instance and Y<sub>i</sub> represents the corresponding set of labels to be predicted (HS &isin; {0,1}, TR &isin; {0,1} and AG &isin; {0,1}), the Exact Match Ratio (EMR) will be computed as follows:<br />
<div class="eq-c"><em><img style="vertical-align: bottom;" src="C:\cygwin64\home\manuela\bundlehateval" alt="" /></em></div>
where Z<sub>i</sub> denotes the set of labels predicted for the i-th instance and I is the indicator function.<br /><br /><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAPoAAABWCAYAAADmMouoAAAMKGlDQ1BJQ0MgUHJvZmlsZQAASImVlwdUk8kWgOcvSUhIaIFQpITeROlVeg0gIFWwEZJAQokhIYjYkUUF1oKKBSu6KmJbCyCLDQsWFsHeHxZUlHWxYAPlTRJAV89777x7zvz/lzt37tx7M/+cGQBUY9giUTaqBkCOME8cGxrAnJicwiQ9AghQBirAHBiyORKRf0xMJIAy/P6nvL8BraFctZP5+rn/v4o6lyfhAIDEQE7jSjg5kA8DgLtyROI8AAg9UG86I08EmQijBJpiGCBkMxlnKNhdxmkKjpTbxMcGQk4FQInKZoszAFCRxcXM52RAPyrlkO2FXIEQchNkHw6fzYU8AHl0Ts50yKpWkK3SvvOT8Q+faSM+2eyMEVbkIhelIIFElM2e+X+W439LTrZ0eA5T2Kh8cVisLGdZ3bKmR8iYCvm8MC0qGrIG5GsCrtxexk/50rCEIfuPHEkgrBlgAIBSueygCMj6kE2E2VGRQ3qfdEEICzKsPRovyGPFK8aiXPH02CH/aAFPEhw3zGyxfC6ZTak0K8F/yOcmPo817LOxkB+fpIgTbc8XJEZBVoF8T5IVFzFk86KQHxg1bCOWxspihv85BtLFIbEKG8wsRzKcF+bJF7Cihjgyjx8fphiLTeWw5bHpQM7kSSZGDsfJ5QUFK/LCinjChKH4sQpRXkDskP12UXbMkD3WxMsOlelNILdJ8uOGx/bmwcWmyBcHoryYeEVsuGYmOzxGEQNuAyJBIAgCTCCFLQ1MB5lA0NZT3wN/KXpCABuIQQbgAbshzfCIJHmPED7jQCH4CxIPSEbGBch7eSAf6r+MaBVPO5Au782Xj8gCTyHngAiQDX9L5aOEI7MlgidQI/hpdg6MNRs2Wd9POqbqsI4YTAwihhFDiNa4Hu6De+GR8OkHmyPujnsMx/XNnvCU0EF4RLhO6CTcniYoEv8QOROMB50wxpCh7NK+zw63gF5d8ADcG/qHvnEGrgfscGc4kz/uC+d2gdrvY5WOZPytlkO+yPZklKxN9iNb/RiBio2Ky4gXWaW+r4UirrSRagWO9PyYR+B39ePCd8SPlthi7BDWgp3CLmBNWD1gYiewBqwVOybjkbXxRL42hmeLlceTBf0IfpqPPTSnrGoS+1r7bvuBoT6QxyvIk30sgdNFM8WCDH4e0x/u1jwmS8gZM5rpaO8Ad1HZ3q/YWt4y5Hs6wrj4TZd7EgCPUqjM+KZjwz3o6FMA6O+/6UzfwGW/HIBj7RypOF+hw2UPAqAAVfil6AJDuHdZwYwcgSvwAn4gGISDaBAPksFUWGc+XKdiMAPMBgtACSgDy8FqsB5sBtvALrAXHAT1oAmcAufAJdAOroO7cK10gZegF7wH/QiCkBAaQkd0ESPEHLFFHBF3xAcJRiKRWCQZSUUyECEiRWYjC5EypAJZj2xFapDfkaPIKeQC0oHcRh4i3cgb5DOKoVRUEzVALdCxqDvqj0ag8egUNAPNRQvRYnQpuhatRvegdegp9BJ6He1EX6J9GMCUMQZmjNlh7lggFo2lYOmYGJuLlWKVWDW2D2uE//RVrBPrwT7hRJyOM3E7uF7D8AScg+fic/FyfD2+C6/Dz+BX8Yd4L/6VQCPoE2wJngQWYSIhgzCDUEKoJOwgHCGchd9OF+E9kUhkEC2JbvDbSyZmEmcRy4kbifuJJ4kdxMfEPhKJpEuyJXmToklsUh6phLSOtId0gnSF1EX6qKSsZKTkqBSilKIkVCpSqlTarXRc6YrSM6V+shrZnOxJjiZzyTPJy8jbyY3ky+Qucj9FnWJJ8abEUzIpCyhrKfsoZyn3KG+VlZVNlD2UJygLlOcrr1U+oHxe+aHyJ6oG1YYaSJ1MlVKXUndST1JvU9/SaDQLmh8thZZHW0qroZ2mPaB9VKGrjFFhqXBV5qlUqdSpXFF5pUpWNVf1V52qWqhaqXpI9bJqjxpZzUItUI2tNletSu2o2k21PnW6uoN6tHqOern6bvUL6s81SBoWGsEaXI1ijW0apzUe0zG6KT2QzqEvpG+nn6V3aRI1LTVZmpmaZZp7Nds0e7U0tJy1ErUKtKq0jml1MjCGBYPFyGYsYxxk3GB81jbQ9tfmaS/R3qd9RfuDzigdPx2eTqnOfp3rOp91mbrBulm6K3Trde/r4Xo2ehP0Zuht0jur1zNKc5TXKM6o0lEHR93RR/Vt9GP1Z+lv02/V7zMwNAg1EBmsMzht0GPIMPQzzDRcZXjcsNuIbuRjJDBaZXTC6AVTi+nPzGauZZ5h9hrrG4cZS423GrcZ95tYmiSYFJnsN7lvSjF1N003XWXabNprZmQ23my2Wa3ZHXOyubs533yNeYv5BwtLiySLRRb1Fs8tdSxZloWWtZb3rGhWvla5VtVW16yJ1u7WWdYbrdttUBsXG75Nlc1lW9TW1VZgu9G2YzRhtMdo4ejq0TftqHb+dvl2tXYPxzDGRI4pGlM/5tVYs7EpY1eMbRn71d7FPtt+u/1dBw2HcIcih0aHN442jhzHKsdrTjSnEKd5Tg1Or51tnXnOm5xvudBdxrsscml2+eLq5ip23efa7Wbmluq2we2mu6Z7jHu5+3kPgkeAxzyPJo9Pnq6eeZ4HPf/2svPK8trt9Xyc5TjeuO3jHnubeLO9t3p3+jB9Un22+HT6Gvuyfat9H/mZ+nH9dvg987f2z/Tf4/8qwD5AHHAk4EOgZ+CcwJNBWFBoUGlQW7BGcELw+uAHISYhGSG1Ib2hLqGzQk+GEcIiwlaE3WQZsDisGlZvuFv4nPAzEdSIuIj1EY8ibSLFkY3j0fHh41eOvxdlHiWMqo8G0azoldH3YyxjcmP+mECcEDOhasLTWIfY2bEtcfS4aXG7497HB8Qvi7+bYJUgTWhOVE2cnFiT+CEpKKkiqXPi2IlzJl5K1ksWJDekkFISU3ak9E0KnrR6Utdkl8klk29MsZxSMOXCVL2p2VOPTVOdxp52KJWQmpS6O3WAHc2uZvelsdI2pPVyAjlrOC+5ftxV3G6eN6+C9yzdO70i/XmGd8bKjG6+L7+S3yMIFKwXvM4My9yc+SErOmtn1mB2Uvb+HKWc1JyjQg1hlvDMdMPpBdM7RLaiElFnrmfu6txecYR4hwSRTJE05GnCQ3ar1Er6i/Rhvk9+Vf7HGYkzDhWoFwgLWmfazFwy81lhSOFvs/BZnFnNs41nL5j9cI7/nK1zkblpc5vnmc4rntc1P3T+rgWUBVkL/iyyL6ooercwaWFjsUHx/OLHv4T+UluiUiIuubnIa9HmxfhiweK2JU5L1i35WsotvVhmX1ZZNlDOKb/4q8Ova38dXJq+tG2Z67JNy4nLhctvrPBdsatCvaKw4vHK8SvrVjFXla56t3ra6guVzpWb11DWSNd0ro1c27DObN3ydQPr+euvVwVU7d+gv2HJhg8buRuvbPLbtG+zweayzZ+3CLbc2hq6ta7aorpyG3Fb/ran2xO3t/zm/lvNDr0dZTu+7BTu7NwVu+tMjVtNzW793ctq0VppbfeeyXva9wbtbdhnt2/rfsb+sgPggPTAi99Tf79xMOJg8yH3Q/sOmx/ecIR+pLQOqZtZ11vPr+9sSG7oOBp+tLnRq/HIH2P+2Nlk3FR1TOvYsuOU48XHB08Unug7KTrZcyrj1OPmac13T088fe3MhDNtZyPOnj8Xcu50i3/LifPe55sueF44etH9Yv0l10t1rS6tR/50+fNIm2tb3WW3yw3tHu2NHeM6jl/xvXLqatDVc9dY1y5dj7recSPhxq2bk2923uLeen47+/brO/l3+u/Ov0e4V3pf7X7lA/0H1f+y/tf+TtfOYw+DHrY+int09zHn8csnkicDXcVPaU8rnxk9q3nu+LypO6S7/cWkF10vRS/7e0r+Uv9rwyurV4f/9vu7tXdib9dr8evBN+Vvdd/ufOf8rrkvpu/B+5z3/R9KP+p+3PXJ/VPL56TPz/pnDJAG1n6x/tL4NeLrvcGcwUERW8yWHwUw2ND0dADe7ASAlgzPDu0AUCYp7mZyQRT3STmB/8SK+5tcXAHY6QdAwnwAIuEZZRNs5pCp8C07gsf7AdTJaaQNiSTdyVHhiwpvLISPg4NvDQAgNQLwRTw42L9xcPDLdhjsbQBO5iruhDKR3UG32Mjo8jjd+eAH+TeFKXA6yHNX8wAAAgtpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iPgogICAgICAgICA8dGlmZjpSZXNvbHV0aW9uVW5pdD4yPC90aWZmOlJlc29sdXRpb25Vbml0PgogICAgICAgICA8dGlmZjpDb21wcmVzc2lvbj4xPC90aWZmOkNvbXByZXNzaW9uPgogICAgICAgICA8dGlmZjpPcmllbnRhdGlvbj4xPC90aWZmOk9yaWVudGF0aW9uPgogICAgICAgICA8dGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0aW9uPjI8L3RpZmY6UGhvdG9tZXRyaWNJbnRlcnByZXRhdGlvbj4KICAgICAgPC9yZGY6RGVzY3JpcHRpb24+CiAgIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+Cg9FKpMAABR3SURBVHgB7Z117CW1E8B7x+Ee3P44LMgFywEHHBokgQAHBA7XoEECBA8cF1yDQ3AJ7u4c7u4Owd3d9jef/q6bvv129e37PptJ3tvdbjudTjvtdDrtDokEjIJyQDnQ0xwY2tOl08IpB5QDlgMq6NoQlAN9wAEV9D6oZC2ickAFXduAcqAPOKCC3geVrEVUDqigaxtQDvQBB1TQ+6CStYjKARV0bQPKgT7ggAp6H1SyFlE5oIKubUA50AccUEHvg0rWIioHVNC1DSgH+oADKuh9UMlaROWACrq2AeVAH3BABb0PKlmLqBwYpixQDsCB77//3rz77rvmpZdeMl9++aU5+OCDzR133GFuu+02M3r0aLPFFlsoo7qYA0P04Ikurr0aSX/88cfNVlttZd5//31z8cUXm48//tgcf/zxZsopp7S5vPnmm2bGGWesMUdFNZgcUNV9MLndwXktv/zy5tBDD7UUbrvttuarr76yo/yIESPMNNNMY6aYYooOpl5Jy+OACnoeh/rk/X///Weuu+46W9oFF1zQnHzyyebrr782DzzwgNloo43ikb1P2NFzxVRB77kqrVYg5uXPPPOMTczcfNiwYebVV1+1z6uttlo1pJqqYziggt4xVdFeQl588UWrrh9zzDFmvvnms8Q8+uijZtpppzVLLbVUe4nT3JvmgAp60yzsDQQPPfSQLQhqOvD333+b++67z6yzzjrmr7/+MrfeeqsN17/u5IAKenfWW61U//rrr+bOO+80K664ohk+fLjF/emnn5rHHnvMvPfee2bllVc2c8wxR615KrLB5YCuow8uvzsyN4xuL7/8sjnjjDPs3Bwip556ajPrrLPaefv9999vRo4c2ZG0K1HFOKDr6MX41Jexfv75ZzN06FAr9H3JgB4qtAp6D1WmFkU5kMYBnaOncaaDw5944glrGX/yySc7mEolrZM4oHP0TqqNHFpeeeUVM2bMGOumSlTWuhWUA0U4oCN6ES51SJyppprKGsd22223DqFIyegWDuiQ0C01JXQ6R5ZJJpmki6hWUjuBAzqid0ItKA2pHHj66acNNol+hQkTJphnn3226eLXIuh77rmnGTJkSKnf2LFjzb///ttQALyzknj22GMPU+QT7mzCSKY988wzG/D/8ccfZvHFFx8Qz0+3zDLLmAMPPNA88sgjhfJtyKBLHv755x+7WYVGVPV37733Wo+5IkV+5513Unm+3nrrGTbUJIE6P+qoo8yyyy5rfvzxx/j12WefHcQ122yz2d12RPziiy8Mz369cp9sDzHSmm7SaEvS4T9T/qQc+OR88sknZumll7a0F5EDP23DPfvRmwVpOJFsiohWWGGFSJDbnxAXyeaISHrjSHymoxtvvDE64ogjInHCsO933333SCq4IWvwfPvtt9GOO+4Y49lkk00iYURDvOSD9PpxfPKfe+65ozfeeCMZzT6La2ck1uqG+OPGjbNh559/fkwfeLbffvtIOocgnnYGSudn6ZdNKJXI+P3336N55523gQeu3opeqcfvvvuucP6//PJLdPTRR8d5ig999OCDD0bURxIIc22A9uMD72RHXYwHesWhZwAe4u23335xPNIk25uPt9l7cROOaPOOf9SR7OuPrr/++ujKK6+0P3EjbmhfxOVdHtxzzz0Wr2wjrlwGRq1aACbKwQVxQcWlMohXvLBsnEsvvTT4Hjzrr79+jGfUqFERDTMNaECLLLJIHB/mXXHFFWnRbTg0uA4n2WBpvD4+2bqZiasdL5sVdGimE3aNkitCQWOFn1m/hx9+2HYSSb4V4cO1114b50l+IaD+xdho491yyy2hKLaxi09+jEs25AyIR5txggfNrQYGFsfH3377LZjdBx98ELc74h522GHBeKHAyy67zOIX78XQ69yw2gQ9KTyff/55MHMaE43khRdeCL6Xk03iCoQZeQ0KZhGPn+yyslfxzw7idoGMJC5NSGOQaUD8vipjXV6tuNYh6NDFCOH4wFXmgoXIpcES/6OPPioUn0gIsBxoEefHKByCiy66yMbZd999M0evyy+/PMZFfSXBlQ0tbTDg2GOPjWSvwADNwuWdHEBks1BqXJfGv/oDoJwG5L8qdF+boPvCs+6660ao4SFAFT799NMjca8MvY5uvvnmuAJpTAi6nHYSjOvn6RqsHJpgR6ZggomB40RVd/EvueSSAVH7RdD9UQ9+MOUpoo7T6JimyaaXAbxLC5B5djxdoE5DAwFh0IFan9eJvP3223EdJgXsmmuuse9C08M0+poNl7P1IjlzL4iGqaevpVJ+BsayAL/hT5E2nsRdm6CPHz8+ZnxyFEQVZL6bJtyOKKe2wYhNN900xheai9IgmWeinh100EFx3Dx1iMa92GKLxfHfeustl729+j0nTE2WpSFymx7qGtEhXwxlMS8oL/WUZxMhnRwimdqZ8z4Jvh0lbSBwHWxIy0ri8+fEfsfw/PPP2/LkTfmS+Fr5LHv8G3gM76qA3zGnTY3T8NYi6D7TaSyvvfZaQ35nnXVWplrjIrte/8gjj4wwXICLX1LQXYfAO+Zfu+yySxxXLPcOXfDqjwShntHNhcBdtecNZlxToF92jIp1AIZSx2uuIS2n2XycEIM/1HnSccNv3qfZb5I0+IPLXXfdZbURcPALaQzJ9IPxfPvttzfwtlmbz+GHH27xJbWYvLLUIuhuzuYaC8LEPBlru+vNzjvvvDxaItcby4EHkc+gZFpZ2rGFZfSWY4rjBkIFp6n5LnN/bof6iZWfFQOmARtssEFcKYwS0JMHrpfFPlDmhzZSRE3288f66oTB8ZrRL09T8nGk3fsWanAnO+u0dEXC0RD8FZmQ8cyppeSd7NjT8vDTwBdGcdLX1QGm5Vs0/PXXX4/bE3TlaZtF8PoGzTJ1VIug+5lToNCvCFGnnHKKnZ999tlndr7j8PiC7ox+CArzfUZwFy9NJXQMZDT0DUIunX/FmnvTTTdFaZZTh8tdmZZUWaqiYZYVdJdnK66Uw5/SUKY6OhBolbXguI7A+9NPPw0ogptXF+msXWI3ffPrr8hylUvfyiu0+as3ZY1vabT5nUfRDhFcTbvACg4japPw+v8gapdZc801Dc4pH374ocEvm/v555/fRQleRQiNCJiRxmZmmWUWe7KJi8gurR122ME+yshjzzaTpRcz+eSTW8cPFw/ngyz30B9++MGIqm+jS4OyRyU99dRTRtZsbZh0FEYanN2D7XDmXaFBOiLrnJEX173HYQWHjqLnpONgUQSoi6rAQRNXXXWVkcZpUXC+u6iJ5oQTTihVtlD+fBTCwdprr22Pj3bP7vrnn3+628KbdWaYYQaz0korxZt89t9/f4MjVruBethuu+2MCKUlRYyc9qz8OjYh+ThkSa/4gSBCVFPg5tVSoqC1lFFaKiA3D9fro+oD4HU9IiM1o7GbSzoVKGkbkFNLM/MRoY5HFmfwoedlFEmjPxPhIL2EtiK/OsjxbRTkmbaWXSYvZzwEHw4kITjggANsGctoOknDat6yaijfVoSddNJJDfVV1fgWos23MfmabiiuH9a0CyxnitH7A4zGybPFcJWkF8+D5557zkaRuZy9+iMzoz1ulDvvvLMR1c8wqgOcaybqi70nnF8WyDw8fr3GGmvYkZtRdZtttrHhoqoamQfHcfrxhk8vyfQmLjrPYoOJn8vecB4dR1EBYvdIHYEmm2yysqjtJ6Q4AgvgLPp55pmnNI66E3D2nvgAxGilY7NyEQfUeMOHNYpC06q7U4XJcMMNNxygdsm6eIM6LaOwCVUqJ46iTi+88MKWdj4FtMACC1j1Rwxzhh/A1RXQd/bntFL3+SAbMfGHuoy672C55ZZzt2bjjTe2KioBfIYIwfdVpDhi4Aa8VK64XAbeZgdBM6p/HkjPnBeltvdME0499VQjhkh7jhwdN1OpqkAn4VRYBoI555wziMp10qG2EUwggWKMi1+hKk866aTxcztu+HadP6iJ5mllIkSLGChtm2nmCziyshBCHQ7zh/ey91hTV1lllVhNybN24vWGNTtphMIQhJqOOu072vgqn1AfySaHmERUeZm3x3nnqZi+x500qgZjW1IFDFmF44wTN9DuVH9oLPoro6Imsmz5o1tJoK5QFZuBCy64IOaJm5aF8DmDLnmKhhiK0hBWtu01JE48iNYRycas6Oqrr068Kf7oTzVpA0w38bcPAW13rbXWis4999wBr+++++5oyy23tCtBA15KgK+6l3HtbWpE5+seTh1mNEZ9SgO+1skIJh2DmW666Rqi0ePz22effRpG/5lmmimOR4+/9957x8+o2RNk9xWASrjEEkvE70I3jFAO6HX90Z9eFaOhrMfbKDfccIPd5ebiZ10xYrHLCkMi90UBlbaoka0ozrriYYRjSoTaiVZVFaRdNmhRHBudBosuuqh9Rb3Kkmd87HRafL/tYexyRsS0+FnhaB2nnXaaNc5yrr0/bUQD5YOTGP789ujjo5ziaBRrLtAjvgipWiEaoAi0EUH30bACZqRjtAbhXXfd1Wq4DRHkAYOyA78Nu7DUqyCvDL67Kp5s9FQhYGmNUVSICBp33Mid3KnkennSJf2w/WW1vKUL6PKX1UJeRVLZ8cjTyaNtiL91hjn/hTrcR51LK/WXx1On1RG3iMOM3zay2l4R3jCiYzRmo48PjMj++j9OXCHwjW+UM0sLwsefMqYtBcv26EhWOqJvvvkmlFWE8xnpcfYqugQMosrr6PgiO6u4q0j8fdnFIwYSu5UQbyifUSFfatKQnh8V7HcWzsruq3yo9vgUJ9d8ZQ9yg9rvc8kx1+WDtRKLvQ/k6++IovOhAfQTuM6OTrnZNXR4x8YUx3MEgHbhT82SvHXecwguqnkIZBkuYmoFPoebjl5GukzcIVx5Yf50j7xCnpRuC6mjBd7R3ugU+KGKi4bYsEWbuFW8D/3pSsi7MKs8lQTdH8ldAYtcfYGFKLek4qf1e356Rhjn91xuC6Ofxt2HRiGnLbg47kqnk3TccB53Lg55J+NkMbOb3+F85DrlvGXKvHK6DsPx0b/iged35j4uvBqJyzw9tKklaUvx8XIfGkh8/Ml7OiOZekbiPxCJAdb6/ftxkiN6si0SN6s9JulzzyE7hExdI9x40Q6OO+64II8Y4MCBjOR5gPrl4L6SoCeR6PPgcECWE2O3Yrz3EJhzzjkn2mmnnUrtJAtR63yoq4w0IXxVw5zxLjkoVMWXlQ5BxpMO4Unr1NFAGGgQwJCgZ+Ev887XkNOmLm5grOL9p4JepjbaHJcG4EYF1ENGYDeFYc6Xpu7mke00mSyVOYSDvQzsE6gT6LzcASZ1Opqk0ejsCGh+aeCcuco4qKThygqXI8xs/YYcf1wdsdJUpZ5V0LM432HvEALfqIiXGZtyUOVQiasA+wpIX1btdcuKoUZZhQ4/DSo124+hi3lyK8FNQ9OWZ8WRywof08Is+0KzNLppScgOgG2DDh5bBPGqgAp6Fa61KQ3zMtRHKp2tvABzTJ7TGmoWqaiuNB7S06DLANZfBDHpE1EGR1ZchB3amM+2cmRnJCcPOrwQ4BJ84YUXBufMofhVw9z6uHPvdnjc+Xh05NRXVWhqHV0aiMIgcgBfA9yNRcAMGzgAOVaokB9BiEyx3FpPQ9ag8YDDnVkaUiiqXfPnHaeSytKWEUE3Ml0Y4BMRTFwhkI9ViNXayHTF5olXXd0gnYl1z8UHA56GQJxXQsG1hzlX7tVXX70BNxvDcMvGZbspqNpDaLrB58C4iUdgsWQDOKswc/WyvT3za2k4Tf1CqxyDz5XqOconrmz5MUCi0aSdHFw9h2Ip3ZQMbY1lYuipMg/Pyq3pTS1N9TKauDAH2OorfgV2446cLmLT4R2GvzebUNjcwbfZigC4tt566yJRM+MstNBCHevdl0n4xJfwDGAbNPsd5pprrolvBvciVn3DhyrE7mHwiAP4XHWdoKp7ndxsIS5UOBqmLDvF7rtunzcbiw455BAjHoiFKMDlV05HjT94UChRIhKuoUsuuWQitLsenWDj6svuSZmrt6UAbGxi6sDUbPTo0bV0wsmCqKAnOdKhz24rsD+Hw/8awG+anXwzzzxzYepXXXXVwnF7NSK+92JMtH7s7dx3wE5J6lAs6mb66advCbuHoNe3BLMiHRQOsFmIkajottpBIUoz6TgOqKB3XJUoQcqB+jlQ74y/fvoUo3JAOVADB1TQa2Bit6DAurvXXnvZ/c5VaeYTxnynHUu1QvdwQFX37qmrpinFKj9ixAh7SAMWfP+AhTzkLN2NGTMmPh8QB4+RI0fmJdP3HcIBHdE7pCIGg4zhw4cbOWDBnHjiiaWEHNrwVEO4OYlHofs4oMtr3VdnlSlGWFHdqwDqOlBGC6iSj6ZpDQd0RG8NXzsKK3NzOXrLyOGH1ouO00oV+osDOqL3QX1zhDJHA2+22WbWhZavxLAZRnyqrUqexgK+niKnnbTNYyyNLg0vzwEV9PI867oUONNw+i4gWz+t4OJSmzyF1Ebw/mSPuhV0L0hvu5QDqrp3acWVJZtNE4Db7rj55pvbDwjIrrfUK8cct8v/u2z5NH42B3REz+ZPz7zlSzgIrb8kpm6zPVO9uQXRET2XRd0fIXTAAh+cYCNH1o+5PL70Ct3PAR3Ru78Oc0sgxy/bLZAY39jayjIb36/jU9RZqjnWeh31c9nbFRFU0Luimpoj0j9gYfz48UY+gGA4NMLf8lokBzY6ysknNqq7FkmncdrPAVXd218HLafAP2CB9fSsUTyNGD5/Pfvss9uz4ogzatQoM3bsWHsqSloaDe8cDqive+fURUspYa7NQRXtPGChpQVU5JkcUEHPZI++VA70BgdUde+NetRSKAcyOaCCnskefakc6A0OqKD3Rj1qKZQDmRxQQc9kj75UDvQGB1TQe6MetRTKgUwOqKBnskdfKgd6gwMq6L1Rj1oK5UAmB1TQM9mjL5UDvcEBFfTeqEcthXIgkwP/Ay5P/96LI879AAAAAElFTkSuQmCCAA==" alt="" width="152" height="51" /><br /> The submissions will be ranked by <em>EMR</em>. This choice is motivated by the willingness of capturing the most difficult task of capturing the entire phenomena, and therefore to identify the most dangerous behaviours against the targets.</li>
</ul>
<p>&nbsp;</p>
<h4><strong>Scoring program</strong></h4>
<p>The evaluation script is available in this GitHub repository:&nbsp;<a href="https://github.com/msang/hateval/tree/master/SemEval2019-Task5/evaluation" target="_blank">https://github.com/msang/hateval/tree/master/SemEval2019-Task5/evaluation</a><a href="https://github.com/msang/hateval/tree/master/evaluation" target="_blank"><br /></a></p>
<h4>NOTE</h4>
<p>During the 'Practice' phase, the prediction files submitted by participants to the task page will be evaluated for the task A, and for demonstration purposes only; if participants wish to test the script on prediction files for task B as well, they could use the version available in the GitHub repository.</p>
<p>For the 'Development' and 'Evaluation' phases, the script will provide a complete evaluation for each language and task for any submitted file, provided that the latter meet the submission requirements (see Data).</p>