{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model evaluation\n",
    "\n",
    "In this tutorial, we will describe the Model Evaluation mode, one of the Simulation modes available in *larvaworld*.\n",
    "\n",
    "This mode is used to evaluate a number of larva models for similarity with a preexisting reference dataset, most often one retained via monitoring real experiments.\n",
    "\n",
    "Let's import the relevant classes :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext param.ipython\n",
    "import panel as pn\n",
    "\n",
    "import larvaworld as lw\n",
    "from larvaworld.lib import reg\n",
    "from larvaworld.lib.sim.model_evaluation import EvalRun, EvalConf, DataEvaluation\n",
    "# from larvaworld.lib.reg.generators import ExpConf\n",
    "\n",
    "# Setting the verbosity level to 0 to get more information\n",
    "lw.VERBOSE = 1\n",
    "\n",
    "# Tutorial safety switches (avoid GUI/heavy compute by default)\n",
    "RUN_EVAL_DEMO = False\n",
    "RUN_PLOTS_DEMO = False\n",
    "\n",
    "DEMO_REF_ID = \"exploration.30controls\"\n",
    "DEMO_MODEL_IDS = [\"explorer\", \"navigator\"]\n",
    "DEMO_N = 5\n",
    "DEMO_DURATION_MIN = 0.5  # >= 0.33 min recommended for 20s metrics\n",
    "DEMO_SCREEN_KWS = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A look at the respective configuration class makes it easy to get an idea of the involved arguments:\n",
    "\n",
    "- Reference dataset, designated via ID or directory\n",
    "- Larva models retrieved via ID and the respective larvagroup IDs and size\n",
    "- Evaluation metrics and setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the attributes of the EvalConf class\n",
    "%params EvalConf\n",
    "\n",
    "# Show the attributes of the EvalConf class as a nested dictionary\n",
    "# EvalConf.param"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preconfigured larva-model configurations can be inspected and selected by a unique ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ids = reg.conf.Model.confIDs\n",
    "print(ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The existing reference datasets can be inspected via their IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "refIDs = reg.conf.Ref.confIDs\n",
    "print(refIDs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model-evaluation launcher accepts also a number of runtype arguments :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the attributes of the EvalRun class\n",
    "%params EvalRun\n",
    "\n",
    "# Show the attributes of the EvalRun class as a nested dictionary\n",
    "EvalRun.param"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A model-evaluation simulation can be launched easily : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kws = {\n",
    "    \"refID\": DEMO_REF_ID,\n",
    "    \"modelIDs\": DEMO_MODEL_IDS,\n",
    "    \"experiment\": \"dish\",\n",
    "    \"N\": DEMO_N,\n",
    "    \"duration\": DEMO_DURATION_MIN,\n",
    "    \"screen_kws\": DEMO_SCREEN_KWS,\n",
    "}\n",
    "\n",
    "r = EvalRun(**kws)\n",
    "if RUN_EVAL_DEMO:\n",
    "    r.simulate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further plotting is possible :\n",
    "- The simulated and reference datasets\n",
    "- The competing larva-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if RUN_PLOTS_DEMO:\n",
    "    r.plot_results(show=False)\n",
    "    r.plot_models(show=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "larvaworld_autoversioning",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}