Merge pull request #78 from instamatic-dev/tweaks

Maintenance and documentation

.bumpversion.cfg

@@ -1,16 +1,16 @@
 [bumpversion]
-current_version = 1.10.0
+current_version = 2.0.0
 
 [comment]
 comment = The contents of this file cannot be merged with that of setup.cfg until https://github.com/c4urself/bump2version/issues/185 is resolved
 
-[bumpversion:file:instamatic/__init__.py]
+[bumpversion:file:src/instamatic/__init__.py]
 search = __version__ = '{current_version}'
 replace = __version__ = '{new_version}'
 
-[bumpversion:file:setup.cfg]
-search = version = {current_version}
-replace = version = {new_version}
+[bumpversion:file:pyproject.toml]
+search = version = "{current_version}"
+replace = version = "{new_version}"
 
 [bumpversion:file:CITATION.cff]
 search = version: "{current_version}"

.github/workflows/publish.yml

@@ -12,16 +12,16 @@ jobs:
     - uses: actions/checkout@v3
     - uses: actions/setup-python@v4
       with:
-        python-version: 3.9
+        python-version: '3.11'
 
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install setuptools wheel twine
-        python setup.py sdist bdist_wheel
+        pip install -U build setuptools wheel
+        python -m build --no-isolation
 
     - name: Publish package
-      uses: pypa/gh-action-pypi-publish@master
+      uses: pypa/gh-action-pypi-publish@release/v1
       with:
         user: __token__
         password: ${{ secrets.PYPI_TOKEN }}

.github/workflows/test.yml

@@ -37,7 +37,7 @@ jobs:
       id: cache-python-env
       with:
         path: ${{ env.pythonLocation }}
-        key: ${{ env.pythonLocation }}-${{ hashFiles('setup.cfg') }}
+        key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
 
     - name: Install
       if: steps.cache-python-env.outputs.cache-hit != 'true'

CITATION.cff

@@ -27,5 +27,5 @@ doi: "10.5281/zenodo.1090388"
 license: "BSD-3-Clause"
 message: "If you use this software, please cite it using these metadata."
 title: Instamatic
-version: "1.10.0"
+version: "2.0.0"
 ...

MANIFEST.in

@@ -13,17 +13,17 @@ include setup_win.bat
 include tests/*.py
 include tests/config/*.yaml
 include tests/config/*/*.yaml
-include instamatic/camera/*.dll
-include instamatic/camera/*.h
-include instamatic/camera/*.lockfile
-include instamatic/camera/*.md
-include instamatic/camera/tpx/*.bpc
-include instamatic/camera/tpx/*.dacs
-include instamatic/camera/tpx/*.txt
-include instamatic/config/*.yaml
-include instamatic/config/alignments/*.yaml
-include instamatic/config/calibration/*.yaml
-include instamatic/config/camera/*.yaml
-include instamatic/config/microscope/*.yaml
-include instamatic/config/scripts/*.md
-include instamatic/neural_network/*.p
+include src/instamatic/camera/*.dll
+include src/instamatic/camera/*.h
+include src/instamatic/camera/*.lockfile
+include src/instamatic/camera/*.md
+include src/instamatic/camera/tpx/*.bpc
+include src/instamatic/camera/tpx/*.dacs
+include src/instamatic/camera/tpx/*.txt
+include src/instamatic/config/*.yaml
+include src/instamatic/config/alignments/*.yaml
+include src/instamatic/config/calibration/*.yaml
+include src/instamatic/config/camera/*.yaml
+include src/instamatic/config/microscope/*.yaml
+include src/instamatic/config/scripts/*.md
+include src/instamatic/neural_network/*.p

docs/config.md

@@ -26,7 +26,7 @@ instamatic.autoconfig.exe
 ```
 To help generate some of the input files (in particular templates for the microscope/calibration files).
 
-Examples of configuration files can be found [here](https://github.com/instamatic-dev/instamatic/tree/main/instamatic/config).
+Examples of configuration files can be found [here](https://github.com/instamatic-dev/instamatic/tree/main/src/instamatic/config).
 
 ## settings.yaml
 

docs/examples/montage_processing.ipynb

@@ -319,7 +319,7 @@
     "    for i, image in enumerate(buffer):\n",
     "        x, y, z, _, _ = stagepos[i]\n",
     "        shape = image.shape\n",
-    "        # binsize = ctrl.cam.getBinning()\n",
+    "        # binsize = ctrl.cam.get_binning()\n",
     "\n",
     "        d = {}\n",
     "        d['StageXYZ'] = x / 1000, y / 1000, z / 1000\n",

docs/merlin.md

@@ -4,7 +4,7 @@ The [MerlinEM detector](https://quantumdetectors.com/products/merlinem/) by Quan
 
 Instamatic communicates with the Merlin control software via TCP/IP connection. This means that there is a fast connection with little overhead when collecting data.
 
-In continous read/write mode, instamatic can achieve gapless data acquisition `MerlinCamera.getMovie()`. When continuously collecting single images using `MerlinCamera.getImage()`, there is a ~3 ms overhead per frame.
+In continous read/write mode, instamatic can achieve gapless data acquisition `MerlinCamera.get_movie()`. When continuously collecting single images using `MerlinCamera.get_image()`, there is a ~3 ms overhead per frame.
 
 ## Setup
 
@@ -16,7 +16,7 @@ camera: merlin
 
 You can set up the parameters for the Merlin camera through `camera/merlin.yaml` in your config directory.
 
-For an example config file, see: [`camera/merlin.yaml`](https://github.com/instamatic-dev/instamatic/blob/main/instamatic/config/camera/merlin.yaml). [Click here](/config.md#camerayaml) to go to the page with all common camera configuration parameters.
+For an example config file, see: [`camera/merlin.yaml`](https://github.com/instamatic-dev/instamatic/blob/main/src/instamatic/config/camera/merlin.yaml). [Click here](/config.md#camerayaml) to go to the page with all common camera configuration parameters.
 
 In addition, the Merlin camera has the following parameters to configure:
 
@@ -34,18 +34,24 @@ host: '10.0.0.123'
 
 **detector_config**
 : The parameters under `detector_config` are directly sent to the Merlin software when instamatic starts.
-These can be used to put the Merlin in the desired state for data acquisition. Check the Merlin documention for more information.
+These can be used to put the Merlin in the desired state for data acquisition. Any command with a `SET` type can be adjusted. Check the Merlin EM documention for more information.
+
+: The default sets continuous read-write mode and 12-bit mode to minimize the gap between frames. It also disables automatic file writing (`FILEENABLE`) and image processing by Merlin, and enables headless `RUNHEADLESS` mode for faster throughput. If you don't want this, you can delete these lines or modify them.
 
-: For example:
+```yaml
+detector_config:
+  CONTINUOUSRW: 1
+  COUNTERDEPTH: 12
+  FILEENABLE: 0
+  RUNHEADLESS: 1
+```
+
+: You could for example also set the thresholds and bias in the instamatic config by adding these lines:
 ```yaml
 detector_config:
   THRESHOLD0: 120
   THRESHOLD1: 40
   HVBIAS: 511
-  COUNTERDEPTH: 12
-  FILEENABLE: 0
-  CONTINUOUSRW: 1
-  RUNHEADLESS: 0
 ```
 
 ## Notes on using instamatic with Merlin

docs/programs.md

@@ -604,7 +604,7 @@ instamatic.xdsserver [-h]
 
 ## instamatic.temserver_fei
 
-Utility script to enable rotation control from a dmscript. See [https://github.com/instamatic-dev/instamatic/tree/master/dmscript] for usage.
+Utility script to enable rotation control from a dmscript. See [https://github.com/instamatic-dev/InsteaDMatic] for usage.
 
 **Usage:**  
 ```bash

docs/tem_api.md

@@ -31,7 +31,7 @@ ctrl = TEMController.initialize(tem_name='jeol', cam_name=None)
 
 Once a `ctrl` (control) object has been initialized, it becomes possible to play around with the lenses and stage interactively. Type `ctrl.` and hit `tab` to see the autocomplete options. Or write use `?` to request the doc string for a function (e.g. `TEMController.initialize?`).
 
-Based on this you can write your own python scripts to control the microscope and/or camera. See in `instamatic/instamatic/experiments/cred/experiment.py` for an idea how this is used. All the microscope control interface can be found in `instamatic/TEMController/`
+Based on this you can write your own python scripts to control the microscope and/or camera. See in `src/instamatic/experiments/cred/experiment.py` for an idea how this is used. All the microscope control interface can be found in `src/instamatic/TEMController/`
 
 The `ctrl` object allows full control over the electron microscope. For example, to read out the position of the sample stage:
 ```python

docs/tvips.md

@@ -71,7 +71,7 @@ Stop live view
 : Stop the live view in EMMENU.
 
 Search
-: Switch the microscope to search mode. Internally, these are defined as [`instamatic` scripts](https://github.com/instamatic-dev/instamatic/tree/main/instamatic/config/scripts) and can therefore be easily modified. The parameters of the search mode are defined in `$instamatic\scripts\search_mode.py`, for example to put the screen down, unblank the beam, and switch to MAG1 mode:
+: Switch the microscope to search mode. Internally, these are defined as [`instamatic` scripts](https://github.com/instamatic-dev/instamatic/tree/main/src/instamatic/config/scripts) and can therefore be easily modified. The parameters of the search mode are defined in `$instamatic\scripts\search_mode.py`, for example to put the screen down, unblank the beam, and switch to MAG1 mode:
 
 ```python
 ctrl.screen.down()

mkdocs.yml

@@ -86,5 +86,5 @@ plugins:
     include_source: True
 
 watch:
-  - instamatic/
+  - src/instamatic/
   - docs/

pyproject.toml

@@ -1,7 +1,125 @@
 [build-system]
-requires = ["setuptools>=40.6.0", "wheel"]
+requires = ["setuptools>=61.2"]
 build-backend = "setuptools.build_meta"
 
+# https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
+[project]
+name = "instamatic"
+version = "2.0.0"
+description = "Python program for automated electron diffraction data collection"
+readme = "README.md"
+requires-python = ">=3.7"
+authors = [
+	{name = "Stef Smeets", email = "s.smeets@esciencecenter.nl"},
+]
+keywords = [
+    "electron-crystallography",
+    "electron-microscopy",
+    "electron-diffraction",
+    "serial-crystallography",
+    "3D-electron-diffraction",
+    "micro-ed",
+    "data-collection",
+    "automation",
+]
+license = {text = "BSD License"}
+classifiers = [
+    "Programming Language :: Python :: 3.7",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Development Status :: 5 - Production/Stable",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: BSD License",
+    "Operating System :: Microsoft :: Windows",
+    "Topic :: Scientific/Engineering :: Human Machine Interfaces",
+    "Topic :: Scientific/Engineering :: Chemistry",
+    "Topic :: Software Development :: Libraries",
+]
+dependencies = [
+    "comtypes >= 1.1.7",
+    "h5py >= 2.10.0",
+    "ipython >= 7.11.1",
+    "lmfit >= 1.0.0",
+    "matplotlib >= 3.1.2",
+    "mrcfile >= 1.1.2",
+    "numpy >= 1.17.3",
+    "pandas >= 1.0.0",
+    "pillow >= 7.0.0",
+    "pywinauto >= 0.6.8",
+    "pyyaml >= 5.3",
+    "scikit-image >= 0.17.1",
+    "scipy >= 1.3.2",
+    "tifffile >= 2019.7.26.2",
+    "tqdm >= 4.41.1",
+    "virtualbox >= 2.0.0",
+    "pyserialem >= 0.3.2",
+]
+
+[project.urls]
+homepage = "https://github.com/instamatic-dev/instamatic"
+issues = "http://github.com/instamatic-dev/instamatic/issues"
+documentation = "https://instamatic.readthedocs.io"
+changelog = "https://github.com/instamatic-dev/instamatic/releases"
+
+[project.optional-dependencies]
+develop = [
+    "bump2version",
+    "check-manifest",
+    "pre-commit",
+    "pytest >= 5.4.1",
+    "coverage",
+]
+serval = [
+    "serval-toolkit"
+]
+docs = [
+    "markdown-include",
+    "mkdocs",
+    "mkdocs-jupyter",
+    "mkdocs-gen-files",
+    "mkdocs-material",
+    "mkdocstrings[python]",
+]
+publishing = [
+    "twine",
+    "wheel",
+    "build",
+]
+
+[project.scripts]
+"instamatic" = "instamatic.main:main"
+"instamatic.controller" = "instamatic.TEMController.TEMController:main_entry"
+# experiments
+"instamatic.serialed" = "instamatic.experiments.serialed.experiment:main"
+"instamatic.camera" = "instamatic.camera.camera:main_entry"
+# calibrate
+"instamatic.calibrate_stage_lowmag" = "instamatic.calibrate.calibrate_stage_lowmag:main_entry"
+"instamatic.calibrate_stage_mag1" = "instamatic.calibrate.calibrate_stage_mag1:main_entry"
+"instamatic.calibrate_beamshift" = "instamatic.calibrate.calibrate_beamshift:main_entry"
+"instamatic.calibrate_directbeam" = "instamatic.calibrate.calibrate_directbeam:main_entry"
+"instamatic.calibrate_stagematrix" = "instamatic.calibrate.calibrate_stagematrix:main_entry"
+"instamatic.flatfield" = "instamatic.processing.flatfield:main_entry"
+"instamatic.stretch_correction" = "instamatic.processing.stretch_correction:main_entry"
+# tools
+"instamatic.browser" = "scripts.browser:main"
+"instamatic.viewer" = "scripts.viewer:main"
+"instamatic.defocus_helper" = "instamatic.gui.defocus_button:main"
+"instamatic.find_crystals" = "instamatic.processing.find_crystals:main_entry"
+"instamatic.find_crystals_ilastik" = "instamatic.processing.find_crystals_ilastik:main_entry"
+"instamatic.learn" = "scripts.learn:main_entry"
+# server
+"instamatic.temserver" = "instamatic.server.tem_server:main"
+"instamatic.camserver" = "instamatic.server.cam_server:main"
+"instamatic.dialsserver" = "instamatic.server.dials_server:main"
+"instamatic.VMserver" = "instamatic.server.vm_ubuntu_server:main"
+"instamatic.xdsserver" = "instamatic.server.xds_server:main"
+"instamatic.temserver_fei" = "instamatic.server.TEMServer_FEI:main"
+"instamatic.goniotoolserver" = "instamatic.server.goniotool_server:main"
+# setup
+"instamatic.autoconfig" = "instamatic.config.autoconfig:main"
+
 [tool.ruff]
 # Enable Pyflakes `E` and `F` codes by default.
 select = [

readme.md

@@ -26,7 +26,7 @@ Cameras supported:
 
 Instamatic has been developed on a JEOL-2100 with a Timepix camera, and a JEOL-1400 and JEOL-3200 with TVIPS cameras (XF416/F416).
 
-[1]: Support for Gatan cameras is somewhat underdeveloped. As an alternative, a DigitalMicrograph script for collecting cRED data on a OneView camera (or any other Gatan camera) can be found at [dmscript](https://github.com/instamatic-dev/InsteaDMatic).
+[1]: Support for Gatan cameras is somewhat underdeveloped. As an alternative, a DigitalMicrograph script for collecting cRED data on a OneView camera (or any other Gatan camera) can be found [here](https://github.com/instamatic-dev/InsteaDMatic).
 
 
 ## Installation
@@ -50,7 +50,7 @@ The package requires Windows 7 or higher. It has been mainly developed and teste
 
 ## Package dependencies
 
-Check [setup.cfg](setup.cfg) for the full dependency list and versions.
+Check [pypoject.toml](pypoject.toml) for the full dependency list and versions.
 
 ## Documentation
 
@@ -76,7 +76,8 @@ Alternatively, some of the methods implemented in `Instamatic` are described in:
 
 * **`demos/`** - Jupyter demo notebooks
 * **`docs/`** - Documentation
-* **`instamatic/`**
+* **`src/`** - Source code for instamatic
+* **`src/instamatic/`**
   * **`TEMController/`** - Microscope interaction code
   * **`calibrate/`** - Tools for calibration
   * **`camera/`** - Camera interaction code
@@ -103,5 +104,4 @@ Alternatively, some of the methods implemented in `Instamatic` are described in:
   * **`navigation.py`** - Optimize navigation paths
   * **`tools.py`** - Collection of functions used throughout the code
 * **`scripts/`** - Helpful scripts
-* **`setup.cfg`** - Dependency/build system declaration
-* **`setup.py`** - Old-style build script
+* **`pyproject.toml`** - Dependency/build system declaration

scripts/process_dm.py

@@ -7,7 +7,7 @@
 from instamatic.processing.ImgConversionDM import ImgConversionDM as ImgConversion
 
 # Script to process cRED data collecting using the DigitalMicrograph script `insteaDMatic`
-# https://github.com/instamatic-dev/instamatic/tree/master/dmscript
+# https://github.com/instamatic-dev/InsteaDMatic
 #
 # To use:
 #     Run `python process_dm.py cred_log.txt`