Merge branch 'main' into tickets/SITCOM-1444

AuxTel/Non-Standard-Operations/EUI-Access/EUI-Access.rst

@@ -73,9 +73,9 @@ Connecting to the EUI Remote desktop
 #. Login with Username/PW of **ATMCS/ATSpectrograph/ATDome EUI access** on 1Password **AuxTel** Vault.
 
 
-#. The following tabs can now be accesible from the web-browser that should always be open:
+#. The following tabs can now be accessible from the web-browser:
 
-    * *Main Box Dome Control: http://139.229.170.45:8000/MainBoxControl.html*   
+    * *Main Box Dome Control:   http://139.229.170.45:8000/MainBoxControl.html*   
 
     .. figure:: /AuxTel/Non-Standard-Operations/_static/EUI-ATDome.png    
 

AuxTel/Non-Standard-Operations/index.rst

@@ -41,3 +41,11 @@ AuxTel Non-standard Operations
     :glob:
 
     Emergency-Shutdown/Emergency-Shutdown.rst
+
+
+.. toctree::
+    :maxdepth: 3
+    :titlesonly:
+    :glob:
+
+    Recovery-after-Shutdown/Recovery-after-Shutdown.rst

AuxTel/Troubleshooting/ATCS/AuxTel-M1-Cover-Fails-to-Open.rst

@@ -82,16 +82,33 @@ The recovery of the M1 cover control must be done from the *ATMCS EUI*.
 
 #. **Engage E-stop in main AT Control Cabinet and ATDome Main Box** located on the first floor. 
 
-#. **Release E-stop starting with the ATDome Main Box followed by the AT Control Cabinet**. 
+#. **Release E-stop starting with the ATDome Main Box, followed by the AT Control Cabinet**. 
 
    Refer to :ref:`release E-stop procedure <E-Stop-Procedure-How-Engage-Disengage-E-stop-System>` for more details. 
 
-#. **Open LSST Auxiliary Telescope MCS & Pneumatics EUI aka ATMCS** in the web-browser. 
-   
+#. **Open LSST Auxiliary Telescope MCS & Pneumatics EUI ATMCS** in the web-browser.    
+
    Address is *http://139.229.170.47:8000/atmcs.html*
-
-    .. figure:: /AuxTel/Non-standard-Operations/_static/EUI-ATMCS.png  
+
+#. Select **Request Control of VI** in the menu that opens with the mouse's right button to gain control of the GUI. 
+
+#. **Change to** *M1 Covers* **tab**.  
+
+    .. figure:: /AuxTel/Troubleshooting/ATCS/_static/ATMCS_M1Covers.png
 
-      ATMCS EUI
+      ATMCS EUI *M1 Covers* tab   
 
+#. Under *M1 Covers Commands* click :guilabel:`start` followed by :guilabel:`enable` will give you **control of the M1 Covers** opening and closing mechanism. 
+
+#. Press :guilabel:`open` to confirm **M1 covers are opening correctly**. 
+
+   The LED lights marked *C1OP, C2OP, C3OP, and C4OP* (corresponding to the numbers of the four cover petals) will illuminate as each petal opens.
+   Petal #3 is the final petal to open and close. 
+
+#. To **allow CSC remote control**, close the M1 covers first by clicking :guilabel:`close` button. 
+   Indicators *C1CL* to *C4CL* will now turn on. 
+
+   Continue with :guilabel:`disable` and :guilabel:`standby` in the *M1 Covers Commands* section. 
+
+   Finally **Release Control of VI** in the mouse's right-click menu. 
 

AuxTel/Troubleshooting/General-Troubleshooting/AuxTel-Elevation-out-of-range.rst

@@ -0,0 +1,130 @@
+.. Review the README in this procedure's directory on instructions to contribute.
+.. Static objects, such as figures, should be stored in the _static directory. Review the _static/README in this procedure's directory on instructions to contribute.
+.. Do not remove the comments that describe each section. They are included to provide guidance to contributors.
+.. Do not remove other content provided in the templates, such as a section. Instead, comment out the content and include comments to explain the situation. For example:
+	- If a section within the template is not needed, comment out the section title and label reference. Include a comment explaining why this is not required.
+    - If a file cannot include a title (surrounded by ampersands (#)), comment out the title from the template and include a comment explaining why this is implemented (in addition to applying the ``title`` directive).
+
+.. Include one Primary Author and list of Contributors (comma separated) between the asterisks (*):
+.. |author| replace:: *I. Sotuela*
+.. If there are no contributors, write "none" between the asterisks. Do not remove the substitution.
+.. |contributors| replace:: *Carlos Morales, Erik Dennihy*
+
+.. This is the label that can be used as for cross referencing this procedure.
+.. Recommended format is "Directory Name"-"Title Name"  -- Spaces should be replaced by hyphens.
+.. _AuxTel-AuxTel-Troubleshooting-General-Troubleshooting-AuxTel-AuxTel-Elevation-out-of-range:
+.. Each section should include a label for cross-referencing to a given area.
+.. Recommended format for all labels is "Title Name"-"Section Name" -- Spaces should be replaced by hyphens.
+.. To reference a label that isn't associated with a reST object such as a title or figure, you must include the link with an explicit title using the syntax :ref:`link text <label-name>`.
+.. An error will alert you of identical labels during the build process.
+
+######################################
+AuxTel Elevation Out of range
+######################################
+
+.. _AuxTel-Elevation-out-of-range-Overview:
+
+Overview
+========
+
+.. In one or two sentences, explain when this AT correct_pointing failure procedure needs to be used. Describe the symptoms that the user sees to use this procedure.
+
+This procedure should be used when the AuxTel :file:`auxtel/correct_pointing.py` script fails with the error 
+
+.. code-block:: text
+
+    Rejected: elevation out-of-range.
+
+.. _AuxTel-Elevation-out-of-range-Error-Diagnosis:
+
+Error diagnosis
+===============
+
+.. This section should provide a simple overview of known or suspected causes for the error.
+.. It is preferred to include them as a bulleted or enumerated list.
+.. Post screenshots of the error state or relevant tracebacks.
+
+- The :file:`auxtel/correct_pointing.py` script fails with the following error traceback:
+
+    .. code-block:: text
+       
+       lsst.ts.salobj.base.AckError: msg='Command failed', 
+       ackcmd=(ackcmd private_seqNum=942292490, ack=<SalRetCode.CMD_FAILED: -302>, 
+       error=6611, result='Rejected : elevation out of range')
+
+- This occurs when the target search in the :file:`auxtel/correct_pointing.py` script is not successful in the specified location in the sky within the `mag_limit`, `mag_range`, and `radius`.
+
+.. _AuxTel-Elevation-out-of-range-Procedure-Steps:
+
+Procedure Steps
+===============
+
+.. _AuxTel-Elevation-out-of-range-Step-1:
+
+To resolve this error while running the :file:`auxtel/correct_pointing.py` script, follow these steps:
+
+#. Try **widening the target search**. 
+
+   Run :file:`auxtel/correct_pointing.py` with a configuration that represents a larger search area. 
+   The script defaults are:
+
+        .. code-block:: text
+            :caption: :file:`auxtel/correct_pointing.py` default configuration
+            
+            az: 90.0
+            el: 60.0
+            mag_limit: 6.0
+            mag_range: 4.0
+            radius: 5
+
+    
+   You can adjust the above parameters to:
+
+      * Decrease the `mag_limit` to target brighter sources.
+      * Increase the search magnitude range (`mag_range`).
+      * Increase the search `radius` in degrees.
+      * Or a combination of the above.
+
+#. If the script works, the pointing correction will center the target in the detector. 
+     If the problem persists, proceed to steps 2.a - 2.b:
+
+     a. Find a target using the standard script :file:`auxtel/track_target.py` with the following configuration:
+
+         .. code-block:: text
+            :caption: :file:`auxtel/track_target.py`
+
+            find_target:
+               az: 90.0
+               el: 60:0
+               mag_limit: 8.0
+
+
+     b. Run the external script :file:`auxtel/correct_pointing.py` on the same area of the sky using the configuration. 
+        Modify `az` and `el` accordingly:
+
+         .. code-block:: text
+            :caption: :file:`run_command.py`
+
+            az: 90.0
+            el: 60.0
+
+#. Fill out the script failure information in the ticket `OBS-186 <https://rubinobs.atlassian.net/browse/OBS-186>`_ for record-keeping.
+
+#. Verify that the AuxTel is functioning correctly and the targets are centered in the detector.
+
+.. _AuxTel-Elevation-out-of-range-Post-Condition:
+
+Post-Condition
+==============
+
+- AuxTel is properly pointed, with targets centered in the detector.
+- System is ready for further observations.
+
+.. _AuxTel-Elevation-out-of-range-Procedure-Contingency:
+
+Contingency
+===========
+
+If the procedure was not successful, report the issue in the `#summit-auxtel <https://lsstc.slack.com/archives/C01K4M6R4AH>`__ channel.
+
+If the pointing is still not accurate, open a ticket to keep a record of the loss of pointing accuracy.

AuxTel/Troubleshooting/General-Troubleshooting/AuxTel-Images-out-of-focus.rst

@@ -0,0 +1,119 @@
+.. This is a template for Image out of focus when some part of the observatory enters an abnormal state. This comment may be deleted when the template is copied to the destination.
+
+.. Review the README in this procedure's directory on instructions to contribute.
+.. Static objects, such as figures, should be stored in the _static directory. Review the _static/README in this procedure's directory on instructions to contribute.
+.. Do not remove the comments that describe each section. They are included to provide guidance to contributors.
+.. Do not remove other content provided in the templates, such as a section. Instead, comment out the content and include comments to explain the situation. For example:
+	- If a section within the template is not needed, comment out the section title and label reference. Include a comment explaining why this is not required.
+    - If a file cannot include a title (surrounded by ampersands (#)), comment out the title from the template and include a comment explaining why this is implemented (in addition to applying the ``title`` directive).
+
+.. Include one Primary Author and list of Contributors (comma separated) between the asterisks (*):
+.. |author| replace:: *I. Sotuela*
+.. If there are no contributors, write "none" between the asterisks. Do not remove the substitution.
+.. |contributors| replace:: *Carlos Morales, Karla Aubel, Erik Dennihy*
+
+.. This is the label that can be used as for cross referencing this procedure.
+.. Recommended format is "Directory Name"-"Title Name"  -- Spaces should be replaced by hyphens.
+.. _AuxTel-AuxTel-Troubleshooting-General-Troubleshooting-AuxTel-Image-out-of-focus-Procedure:
+.. Each section should includes a label for cross referencing to a given area.
+.. Recommended format for all labels is "Title Name"-"Section Name" -- Spaces should be replaced by hyphens.
+.. To reference a label that isn't associated with an reST object such as a title or figure, you must include the link an explicit title using the syntax :ref:`link text <label-name>`.
+.. An error will alert you of identical labels during the build process.
+
+#########################
+AuxTel Image out of focus
+#########################
+
+.. _Image-out-of-focus-Procedure-Overview:
+
+Overview
+========
+
+This procedure should be used when images appear out of focus due to a *WEP failure* or other related issues. 
+Users will notice images ranging from slightly out of focus to large donut appearances when *ATHexapod* offsets are not correctly applied.
+
+.. _AuxTel-Image-out-of-focus-Procedure-Error-Diagnosis:
+
+Error diagnosis
+===============
+
+- The focus script :file:`auxtel/latiss_wep_align.py` might fail, leaving the system out of focus.
+- These out of focus images might vary from slightly out of focus image when the LATISS WEP align script took the system out of focus,
+  to complete out-of-focus images that can appear as large donuts when *ATHexapod* offsets are left in intra or extra focus positions due to a script failure.
+
+
+.. figure:: ./_static/min-focus.png
+    :name: in-focus
+    :width: 300
+
+    Slightly out-of-focus image.
+
+.. figure:: ./_static/out-focus.png
+    :name: out-focus
+    :width: 300
+
+    Extreme out-of-focus image.
+
+.. _AuxTel-Image-out-of-focus-Procedure-Procedure-Steps:
+
+Procedure Steps
+===============
+
+
+#. Pause LOVE *ATQueue*.
+
+.. Add link below to ScriptQueue document once ready. 
+
+#. Clear all *ATAOS* offsets for the x, y, and z axes. Load and run the standard script :file:`auxtel/offset_ataos.py` **moving the script up in the queue** with the following configuration:
+
+     .. code-block:: python
+        :caption: :file:`auxtel/offset_ataos.py`
+
+        reset_offsets: all
+
+#. Run the :file:`auxtel/latiss_wep_align.py` external script with the configuration below to find a new target in the area of the sky of your choice. Replace `az` and `el`.
+
+     .. code-block:: python
+        :caption: :file:`auxtel/latiss_wep_align.py`
+
+        find_target:
+           az: 135
+           el: 40
+           mag_limit: 8.0
+
+
+#. Keep playing the rest of the scripts in the queue, ensuring the system is now in focus.
+
+#. Complete the procedure by verifying the system is in focus.
+
+.. _AuxTel-Image-out-of-focus-Procedure-Post-Condition:
+
+Post-Condition
+==============
+
+- System is back in focus.
+
+.. _AuxTel-Image-out-of-focus-Procedure-Contingency:
+
+Contingency
+===========
+
+If the procedure was not successful, report the issue immediately in the `#summit-auxtel <https://lsstc.slack.com/archives/C01K4M6R4AH>`__ channel.
+
+#. If step 2 doesn't work and you see a large number in the *ATAOS offsets values* at the *AuxTel Monitor* LOVE dashboard, it is because offsets had been wrongly accumulated. 
+
+   Restart *ATAOS*: Cycle *ATAOS* CSC to ``STANDBY`` and back to ``ENABLED`` state. 
+
+#. If *ATAOS* was restarted, corrections need to be re-enabled. 
+
+   Use :file:`run_command.py` with the following configuration to re-enable *ATAOS* corrections. 
+
+     .. code-block:: text
+        :caption: :file:`run_command.py` to enable AOS corrections
+
+        component: ATAOS
+        cmd: enableCorrection
+        parameters:
+           hexapod: true
+           m1: true
+           atspectrograph: true

AuxTel/Troubleshooting/General-Troubleshooting/AuxTel-Lost-Pointing-Accuracy.rst

@@ -0,0 +1,74 @@
+.. Review the README in this procedure's directory on instructions to contribute.
+.. Static objects, such as figures, should be stored in the _static directory. Review the _static/README in this procedure's directory on instructions to contribute.
+.. Do not remove the comments that describe each section. They are included to provide guidance to contributors.
+.. Do not remove other content provided in the templates, such as a section. Instead, comment out the content and include comments to explain the situation. For example:
+	- If a section within the template is not needed, comment out the section title and label reference. Include a comment explaining why this is not required.
+    - If a file cannot include a title (surrounded by ampersands (#)), comment out the title from the template and include a comment explaining why this is implemented (in addition to applying the ``title`` directive).
+
+.. Include one Primary Author and list of Contributors (comma separated) between the asterisks (*):
+.. |author| replace:: *I. Sotuela*
+.. If there are no contributors, write "none" between the asterisks. Do not remove the substitution.
+.. |contributors| replace:: *Carlos Morales, Erik Dennihy*
+
+.. This is the label that can be used as for cross referencing this procedure.
+.. Recommended format is "Directory Name"-"Title Name"  -- Spaces should be replaced by hyphens.
+.. _AuxTel-AuxTel-Troubleshooting-General-Troubleshooting-AuxTel-Lost-Pointing-Accuracy-Procedure:
+.. Each section should include a label for cross-referencing to a given area.
+.. Recommended format for all labels is "Title Name"-"Section Name" -- Spaces should be replaced by hyphens.
+.. To reference a label that isn't associated with a reST object such as a title or figure, you must include the link with an explicit title using the syntax :ref:`link text <label-name>`.
+.. An error will alert you of identical labels during the build process.
+
+#############################
+AuxTel Lost Pointing Accuracy
+#############################
+
+.. _AuxTel-Lost-Pointing-Accuracy-Procedure-Overview:
+
+Overview
+========
+
+This procedure should be used when the AuxTel loses pointing accuracy, causing targets to not appear centered in the detector during the first acquisition.
+
+.. _AuxTel-Lost-Pointing-Accuracy-Procedure-Error-Diagnosis:
+
+Error diagnosis
+===============
+
+.. This section should provide a simple overview of known or suspected causes for the error.
+.. It is preferred to include them as a bulleted or enumerated list.
+.. Post screenshots of the error state or relevant tracebacks.
+
+- Targets are not centered in the detector, appearing at the edge of the field or not in the field at all.
+- The issue might occur if the :file:`auxtel/correct_pointing.py` script has not centered the right star.
+- This can arise when two or more similar magnitude stars are present in the field, potentially confusing the script.
+- The issue should be mitigated by using the bright star catalog, but sometimes SIMBAD returns a field with two or more bright stars that may cause mispointing.
+- See `OBS-88 <https://rubinobs.atlassian.net/browse/OBS-88>`_ for some off-centered images examples.
+
+.. _AuxTel-Lost-Pointing-Accuracy-Procedure-Steps:
+
+Procedure Steps
+===============
+
+#. Pause the LOVE *ATQueue*.
+
+#. **If** the pointing is completely lost and no bright star is seen in the detector, the absorbed zero-point pointing offsets need to be reset.  
+
+   In order to reset the absorbed pointing offsets, recycle *ATPtg* CSC by sending it to ``STANDBY``, and then back to ``ENABLED``. 
+
+#. Add a :file:`auxtel/correct_pointing.py` script to the queue and move it to the top of the waiting scripts. 
+
+   Leave the configuration empty (defaults are `az`: 90, `el`: 60) or you can choose an area (`az`, `el`) of the sky by introducing new values.  
+   You could also launch a wider target search by modifying the `mag_limit`, `mag_range`, or the search `radius`.
+
+    .. code-block:: text
+        :caption: :file:`auxtel/correct_pointing.py`
+
+        az: 180
+        el: 35
+        mag_limit: 6.0
+        mag_range: 4.0
+        radius: 5.0
+
+#. Play the *ATQueue* and while the script is running, confirm in the incoming *RubinTV* images that the right target is being centered.
+
+#. If the pointing is lost again, repeat steps 1-3 with a different configuration, different location of the sky, or widen the search to ensure the target is appropriate for the :file:`auxtel/correct_pointing.py` script.