python_interface.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e47b776-89dd-4f71-9ef6-b954e5368efd",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Python interface: practical guide"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "011c54df-7f8d-4bef-9ebf-3285dcccd1db",
   "metadata": {
    "tags": []
   },
   "source": [
    "***Part 1: calling C++ from Python*** \n",
    "\n",
    "In this first part use the *spinfoam.py* module in order to call C++ source code implemented in the `./src` folder.\n",
    "We do this by using the spinfoam shared library.\n",
    "\n",
    "This allows us to take full advantage of the efficiency of the C++ code in order to deal with the most hard (computationally speaking) sections of the algorithm.\n",
    "\n",
    "***Part 2: pure Python***   \n",
    "\n",
    "In this second part we manipulate the csv files, computed and stored in the first part, with Python code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db89d8de-f9d3-4fb2-846c-04f2736f4774",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Step 0: computing the required fastwigxj hash tables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "736f26e0-2ded-41a8-ad16-7d29be008697",
   "metadata": {},
   "source": [
    "First, we must compute all the necessary hash tables of 3j, 6j and 9j Wigner symbols.\n",
    "These tables are, in fact, a key ingredient for *step 1* and *step 2*.\n",
    "\n",
    "For the calculation, we use the _wigxjpf_ and _fastwigxj_ libraries [Johansson et al., 2015], stored under `./ext/`.\n",
    "\n",
    "In the following, with \"type x\" I mean \"type x in linux shell terminal\"\n",
    "\n",
    "1. Change directory to `./ext/wigxjpf/`, type **make**.\n",
    "2. Change directory to `./ext/fastwigxj/`, type **make**. \n",
    "   In in the followng, the latter is supposed to be the current directory.\n",
    "   \n",
    "3. Type \"bin/hash_js --max-E-3j=50 /dev/null ./table_50.3j\".\n",
    "4. Type \"bin/hash_js --max-E-6j=40 /dev/null ./table_40.6j\".\n",
    "5. Type \"bin/gen_9j --flat-lim=16 | bin/combine_js | bin/unique_js ./comb_16.9j\".\n",
    "6. Type \"bin/hash_js ./comb_16.9j ./hashed_16.9j\".\n",
    "7. Move the hash tables \"hashed_16.9j\", \"table_40.6j\" and \"table_50.3j\" generated in steps 3-6 in a directory of your choice.\n",
    "   \n",
    "   For example, create a folder `./data_folder/fastwigxj_tables/` and move all the hash tables in it.\n",
    "\n",
    "   This is the default folder which I refer to in the following, but you can also choose another folder and provide a different \n",
    "   path.\n",
    "   \n",
    "**Important**: when using Jupyter notebook, if the hash table is not found in the provided path, the kernel dies with no message. \n",
    "\n",
    "When using python script from linux terminal, the C++ returns an error message which is correctly printed. \n",
    "\n",
    "Therefore, we must be sure that the fastwigxj hash tables above are actually stored before moving on.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea564d13-01d1-4a42-a177-96146ebb52af",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Step 1: creating the **spinfoam** shared library"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8de91852-d629-40c6-adf5-ea96c0ba52b3",
   "metadata": {},
   "source": [
    "We first import the *spinfoam.py* module "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3d982fc5-a9a2-4501-8d45-181955fad376",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spinfoam as sf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30041d2d-0b5b-472d-ac90-1e9adc05d561",
   "metadata": {},
   "source": [
    "We can always clean all the previous binary and object files, as well as the shared library along with the corresponding folders, with the following cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "367bbbac-1170-4c2d-a690-e5660dde5be5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cleaning wigxjpf...\n",
      "rm -f *.test\n",
      "rm -rf obj/\n",
      "rm -rf gen/\n",
      "rm -rf lib/\n",
      "rm -rf bin/\n",
      "rm -rf mod/\n",
      "cleaning fastwigxj...\n",
      "rm -rf gen/\n",
      "rm -rf obj/\n",
      "rm -rf lib/\n",
      "rm -rf bin/\n",
      "rm -rf mod/\n",
      "rm -f test.3j.mark test.6j.mark test.9j.mark test_float128.6j.mark\n",
      "\n",
      "rm -rf obj\n",
      "rm -rf lib\n",
      "rm -rf bin\n"
     ]
    }
   ],
   "source": [
    "sf.spinfoam_clean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e4c026d-94ca-4853-b1f5-4902d69fdec3",
   "metadata": {},
   "source": [
    "We can compile and generate all the required files with the following cell, provided that *all dependencies are satisifed*.\n",
    "\n",
    "For details, see the README file.\n",
    "\n",
    "Notice that you can also compile in debug mode with **spinfoam_compile_debug()**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c7c4265f-449d-41bd-850a-b07c215ec63e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "compiling wigxjpf...\n",
      "  AUTO   gen/wigxjpf_auto_config.h.TEST_LONG_DOUBLE\n",
      "  AUTO   gen/wigxjpf_auto_config.h.TEST_FLOAT128\n",
      "  AUTO   gen/wigxjpf_auto_config.h.TEST_THREAD\n",
      "  AUTO   gen/wigxjpf_auto_config.h.TEST_UINT128\n",
      "  AUTO   gen/wigxjpf_auto_config.h\n",
      "   CC    obj/prime_factor.o\n",
      "   CC    obj/calc.o\n",
      "   CC    obj/trivial_zero.o\n",
      "   CC    obj/c_wrap.o\n",
      "   CC    obj/fortran_wrap.o\n",
      "   CC    obj/wigxjpf_error.o\n",
      "   CC    obj/calc_float128.o\n",
      "   CC    obj/c_wrap_float128.o\n",
      "   AR    lib/libwigxjpf_quadmath.a\n",
      "   AR    lib/libwigxjpf.a\n",
      "   CC    obj/wigxjpf.o\n",
      "  LINK   bin/wigxjpf\n",
      "compiling fastwigxj...\n",
      "  AUTO   gen/fastwigxj_auto_config.h.TEST_LONG_DOUBLE\n",
      "  AUTO   gen/fastwigxj_auto_config.h.TEST_FLOAT128\n",
      "  AUTO   gen/fastwigxj_auto_config.h.TEST_THREAD\n",
      "  AUTO   gen/fastwigxj_auto_config.h.TEST_THREAD_MUTEX\n",
      "  AUTO   gen/fastwigxj_auto_config.h.TEST_LSFENCE\n",
      "  AUTO   gen/fastwigxj_auto_config.h.TEST_SSE4_1\n",
      "  AUTO   gen/fastwigxj_auto_config.h.TEST_SSE4_2\n",
      "  AUTO   gen/fastwigxj_auto_config.h.TEST_AVX2\n",
      "  AUTO   gen/fastwigxj_auto_config.h\n",
      "  DEPS   obj/fortran_wrap.d\n",
      "  DEPS   obj/canonicalise.d\n",
      "  DEPS   obj/fastwigxj.d\n",
      "  DEPS   obj/gen_header.d\n",
      "  DEPS   obj/gen_buf.d\n",
      "  DEPS   obj/fastwigxj_cc.d\n",
      "  DEPS   obj/clookup.d\n",
      "  DEPS   obj/lookuptest.d\n",
      "  DEPS   obj/info_js.d\n",
      "  DEPS   obj/hash_js.d\n",
      "  DEPS   obj/unique_js.d\n",
      "  DEPS   obj/combine_js.d\n",
      "  DEPS   obj/gen_3j.d\n",
      "  DEPS   obj/gen_9j.d\n",
      "   CC    obj/fastwigxj.o\n",
      "   CC    obj/canonicalise.o\n",
      "   CC    obj/fortran_wrap.o\n",
      "   AR    lib/libfastwigxj.a\n",
      "   CXX   obj/gen_9j.o\n",
      "   CXX   obj/gen_buf.o\n",
      "   CXX   obj/gen_header.o\n",
      "  LINK   bin/gen_9j\n",
      "   CXX   obj/gen_3j.o\n",
      "  LINK   bin/gen_3j\n",
      "   CXX   obj/combine_js.o\n",
      "  LINK   bin/combine_js\n",
      "   CXX   obj/unique_js.o\n",
      "  LINK   bin/unique_js\n",
      "   CXX   obj/hash_js.o\n",
      "  LINK   bin/hash_js\n",
      "   CXX   obj/info_js.o\n",
      "   CXX   obj/fastwigxj_cc.o\n",
      "  LINK   bin/info_js\n",
      "   CXX   obj/lookuptest.o\n",
      "  LINK   bin/lookuptest\n",
      "   CC    obj/clookup.o\n",
      "  LINK   bin/clookup\n",
      "\n",
      "   CXX    obj/chain_class.o\n",
      "   CXX    obj/mcmc.o\n",
      "   CXX    obj/hash_21j_symbols.o\n",
      "   CXX    obj/python_mirror.o\n",
      "   CXX    lib/libspinfoam.so\n",
      "   CXX    bin/Metropolis_Hastings_parallel_run\n",
      "   CXX    bin/Hashing_21j\n"
     ]
    }
   ],
   "source": [
    "sf.spinfoam_compile()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8334c26a-3748-4cbc-a1bc-8ac7ccaa653c",
   "metadata": {},
   "source": [
    "### Step 2: Hashing tables of 21j Wigner symbols "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81574aea-08e5-490b-8b1b-eeea84ab94cd",
   "metadata": {},
   "source": [
    "We now compute the 21j Wigner symbols hash tables.\n",
    "These  are a key ingredient for the Metropolis-Hastings algorithm, as they will be will be retrieved in *step 3*.\n",
    "\n",
    "We use [parallel hashmap](https://github.com/greg7mdp/parallel-hashmap) in order to efficiently store and dump to disk the Wigner symbols. \n",
    "\n",
    "All the necessary _fastwigxj_ hash tables of 3j, 6j and 9j Wigner symbols must have been previously computed and stored in `./data_folder/fastwigxj_tables/` (or another folder of your choice, provided that the correct path is provided to the function below). See *step 0* for details\n",
    "\n",
    "The C++ code searches for all files in the provided folder with extension \".3j\", \".6j\" or \".9j\", and the largest tables found are set for loading.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7f6ac00e-a8f7-4ca2-9d40-6872b9d50f2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is the folder with the pre-computed hash tables of 3j, 6j and 9j Wigner symbols.\n",
    "# Jupyter kernel dies if tables are not found (C function returns \"EXIT FAILURE\")\n",
    "fastwigxj_tables_path = \"./data_folder/fastwigxj_tables\"\n",
    "\n",
    "# this is the folder in which we wanto to store the computed 21j symbols hash tables\n",
    "hash_tables_path = \"./data_folder/21j_hash_tables\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "70b4276d-88ae-4e55-b02f-4643f1fa27fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hashing all 21j symbols with j <= 0.5...\n",
      "Maximum amount is roughly 5184 symbols, starting... \n",
      "Completed! The hash table has been stored\n",
      "Hashing all 21j symbols with j <= 1...\n",
      "Maximum amount is roughly 103680 symbols, starting... \n",
      "Completed! The hash table has been stored\n",
      "Hashing all 21j symbols with j <= 1.5...\n",
      "Maximum amount is roughly 960000 symbols, starting... \n",
      "Completed! The hash table has been stored\n",
      "Hashing all 21j symbols with j <= 2...\n",
      "Maximum amount is roughly 5670000 symbols, starting... \n",
      "Completed! The hash table has been stored\n"
     ]
    }
   ],
   "source": [
    "# 'spin' is the value of all boundary spins of the spinfoam\n",
    "\n",
    "for spin in [0.5, 1, 1.5, 2]:\n",
    "    sf.Hashing_21j_symbols(hash_tables_path, fastwigxj_tables_path, spin)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f1d1df2-c9c7-4843-80a6-c33b8d5e148d",
   "metadata": {},
   "source": [
    "### Step 3: Metropolis-Hastings "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b7619c1-ee6f-4eb8-9c32-5d830e8ecc0b",
   "metadata": {},
   "source": [
    "We can now compute and assemble the Markov chains by running the MH algorithm. The computed draws are stored in the given folder.\n",
    "\n",
    "**We run in parallel an independent Markov chain for each provided thread**.\n",
    "\n",
    "*It is advisable for the performance to use a number of Markov chains equal to or less than the physical number of cores present on the system.*\n",
    "\n",
    "Each thread retrieves from the 21j hash table (computed in step 2) the required symbol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4bb83c97-c40d-44ce-b6df-2cd8735432d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "optimal number of threads: 12\n"
     ]
    }
   ],
   "source": [
    "# folder in which we want to store the data\n",
    "data_folder = \"./data_folder/data\"\n",
    "\n",
    "# this is the folder with the pre-computed hash tables of 21j Wigner symbols (see Step 2).\n",
    "# Jupyter kernel dies if table is not found (C function returns \"EXIT FAILURE\")\n",
    "# hash_tables_path = \"./data_folder/21j_hash_tables\"\n",
    "\n",
    "# MH parameters\n",
    "spin = 0.5\n",
    "length = 1000000\n",
    "sigma = 0.40\n",
    "burnin = 12\n",
    "verbosity = 0\n",
    "\n",
    "# set optimal number of threads\n",
    "import multiprocessing\n",
    "\n",
    "number_of_threads = int(multiprocessing.cpu_count())\n",
    "print(f\"optimal number of threads: {number_of_threads}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e64ed585-75a5-4c42-8525-362fd1028e1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting 12 independent Markov chains...\n",
      "Completed! All draws have been stored\n"
     ]
    }
   ],
   "source": [
    "sf.Metropolis_Hastings_parallel_run(\n",
    "    data_folder,\n",
    "    hash_tables_path,\n",
    "    spin,\n",
    "    length,\n",
    "    sigma,\n",
    "    burnin,\n",
    "    verbosity,\n",
    "    number_of_threads,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "413210ee-92e2-4420-a851-5bbcdbd19521",
   "metadata": {},
   "source": [
    "### Step 4: Manipulate the CSV file with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a31da4-e42e-4f91-8050-aecc949e58ba",
   "metadata": {},
   "source": [
    "We can now manipulate the csv files, computed in *step 3*, in order to extract the informations we are interested in.\n",
    "\n",
    "See `notebooks` folder"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "06e89d38a6ef794860305859554581b1e0816139527ef4062fee213b9bc6f60b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}