Add AuxTel Troubleshooting summary table

lsst-ts · Nov 5, 2024 · 893fb94 · 893fb94
1 parent 8fceaeb
commit 893fb94
Show file tree

Hide file tree

Showing 3 changed files with 83 additions and 46 deletions.
diff --git a/AuxTel/Standard-Operations/Daytime-Operations/Daytime-Checkout.rst b/AuxTel/Standard-Operations/Daytime-Operations/Daytime-Checkout.rst
@@ -103,18 +103,4 @@ consider running this script as it will further exercise the system.
 Other options include: :file:`prepare_for/onsky.py`, :file:`prepare_for/vent.py` or :file:`shutdown_all.py`
 
 
-This procedure was last modified on |today|... |author| replace:: *E. Dennihy*
-.. If there are no contributors, write "none" between the asterisks. Do not remove the substitution.
-.. |contributors| replace:: *none*
-
-.. _AuxTel-DayTime-Operations-Daytime-Checkout:
-
-.. Links 
-
-.. _`summit's RubinTV`: https://summit-lsp.lsst.codes/rubintv/summit/auxtel 
-.. _`USDF RubinTV`: https://usdf-rsp-dev.slac.stanford.edu/rubintv 
-
-Quick Troubleshooting
-=====================
-
 This procedure was last modified on |today|.
diff --git a/AuxTel/Troubleshooting/index.rst b/AuxTel/Troubleshooting/index.rst
@@ -7,21 +7,90 @@ AuxTel Known Faults
 A comprehensive list of common faults will be displayed here along their troubleshooting procedures. 
 
 .. list-table:: Known Faults
-   :widths: 10 50 20 40 
+   :widths: 10 35 35 20 
    :header-rows: 1 
 
-   * - CSC
-     - Issue/Diagnosis
-     - Happens mostly when
+   * - OCS
+     - Issue
+     - Common in / Error traceback
      - Troubleshooting procedure
-   * - Row 1, column 1
-     -
-     - Row 1, column 3
-     - Row 1, column 4
-   * - Row 2, column 1
-     - Row 2, column 2
-     - Row 2, column 3
-     - 
+   * - ATCS/ATHexapod
+     - ATHexapod fails to enable - No connection to athexapod controller
+     - :ref:`Daytime checkout <AuxTel-DayTime-Operations-Daytime-Checkout>`
+
+        raise OSError(err, f'Connect call failed {address}')OSError: [Errno 113] Connect call failed ('139.229.170.48', 50000)
+     - :ref:`ATHexapod fails to enable with the rest of ATCS <ATHexapod-fails-to-enable-with-the-rest-of-ATCS>`
+   * - 
+     - ATMCS fails to enable - e-stop is engaged
+     - :ref:`Daytime checkout <AuxTel-DayTime-Operations-Daytime-Checkout>`
+
+        Fault event in ATMCS while in enable state (port=. Nasmyth 1 drive fault bit is ON)
+     - :ref:`AuxTel Mount Control System Fails to Enable / E-Stop is Engaged <ATCS-Troubleshooting-AuxTel-Mount-Control-System-Fails-to-Enable>`
+   * - LATISS/ATSpectrograph
+     - ATSpectrograph failed after power down
+     - :ref:`Daytime checkout <AuxTel-DayTime-Operations-Daytime-Checkout>`
+
+        raise base.AckTimeoutError( lsst.ts.salobj.base.AckTimeoutError: msg='Timed out waiting for command acknowledgement',
+        ackcmd=(ackcmd private_seqNum=1137560160, ack=<SalRetCode.CMD_NOACK: -301>, error=0, result='No command acknowledgement seen')
+     - :ref:`ATSpectrograph failed - grating stage position and timed out <LATISS-Troubleshooting-ATspectrograph-failed>`
+   * - LATISS/ATCamera
+     - ATCamera Recovering from Fault State
+     - :ref:`Daytime checkout <AuxTel-DayTime-Operations-Daytime-Checkout>`, :ref:`Calibrations <Daytime-Operations-LATISS-Daily-Calibrations-BIAS-DARK-FLAT-all-filters-empty-Procedure>` , :ref:`Nightime Operations <AuxTel-Nighttime-Operations-index>` 
+     - :ref:`AT camera recovery <LATISS-Troubleshooting-ATcamera-recovery>`
+   * - ATCalSys/ATWhiteLight
+     - ATWhiteLight failed to turn on
+     - :ref:`Calibrations <Daytime-Operations-LATISS-Daily-Calibrations-BIAS-DARK-FLAT-all-filters-empty-Procedure>` 
+
+        RuntimeError: White Light Lamp failed to turn on after 900 s.
+     - TO DO
+   * - ATCS - M1 Cover
+     - AuxTel M1 Cover Fails to Open.
+     - :ref:`Daytime checkout <AuxTel-DayTime-Operations-Daytime-Checkout>`, :ref:`Calibrations <Daytime-Operations-LATISS-Daily-Calibrations-BIAS-DARK-FLAT-all-filters-empty-Procedure>`, :ref:`Prepare for on-sky <AuxTel-Nighttime-Operations-Open-for-On-Sky-Operations>`
+
+        result='ERROR: Command OPENM1COVER rejected while M1 covers controller in StandbyState state.'
+     - :ref:`AuxTel M1 Cover Fails to Open <AuxTel-Troubleshooting-AuxTel-M1-Cover-Fails-To-Open>`. Explanation `video <https://confluence.lsstcorp.org/download/attachments/210241325/M1%20cover%20reset%20using%20EUI.mp4?version=3&modificationDate=1700889964000&api=v2>`__
+   * - Scheduler Driven Observations
+     - Rotator out of range error and ATPtg might go into `FAULT` state. 
+     - :ref:`Nightime Operations <AuxTel-Nighttime-Operations-index>`
+
+        error=6611,  result='Rejected : Rotator out of range. Target in rotator limit (-170 to 170 degrees) but out of slew limit margin (1 degs)')
+     - :ref:`AuxTel Rotator Out Of Range <AuxTel-Rotator-Out-Of-Range>`  
+   * - 
+     - Lost pointing: Targets do not appear centered in the detector in the first acquisition
+     - :ref:`Nightime Operations <AuxTel-Nighttime-Operations-index>`
+
+        RuntimeError("Centroid finding algorithm was unsuccessful.")
+     - :ref:`AuxTel Lost Pointing Accuracy <AuxTel-AuxTel-Troubleshooting-General-Troubleshooting-AuxTel-Lost-Pointing-Accuracy-Procedure>`
+   * - 
+     - AT out of focus after a 'WEP' failure
+     - :ref:`Nightime Operations <AuxTel-Nighttime-Operations-index>`
+
+     - :ref:`AuxTel Image out of focus <AuxTel-AuxTel-Troubleshooting-General-Troubleshooting-AuxTel-Image-out-of-focus-Procedure>`
+   * - 
+     - Correct Pointing fails
+     - :ref:`Nightime Operations <AuxTel-Nighttime-Operations-index>`
+
+        Rejected: elevation out-of-range.
+     - :ref:`AuxTel Elevation Out of range <AuxTel-AuxTel-Troubleshooting-General-Troubleshooting-AuxTel-AuxTel-Elevation-out-of-range>`
+   * - ATCS/ATMCS
+     - ATMount fails to move and times out
+     - :ref:`Nightime Operations <AuxTel-Nighttime-Operations-index>`
+
+        RuntimeError: Telescope timed out getting in position.
+     - :ref:`AuxTel Mount Fails to Move and Times Out <ATCS-Troubleshooting-AuxTel-Mount-Fails-to-Move-and-Times-Out>`
+   * - 
+     - ATMount fails to move and Azimuth Max Velocity error exceeded
+     - :ref:`Nightime Operations <AuxTel-Nighttime-Operations-index>`
+
+        Fault event in ATMCS while in tracking enable state.  Azimuth drive #2 fault bit is ON.  Azimuth drive #1 fault bit is ON.  Azimuth max velocity error  exceeded.
+     - :ref:`AuxTel Mount Fails to Move <ATCS-Troubleshooting-AuxTel-Mount-Fails-to-Move>`
+   * - ATCS/ATDome
+     - ATDome shutter fails to close
+     - :ref:`Shutdown <AuxTel-Nighttime-Operations-Shutdown>`
+     - :ref:`AuxTel Emergency Shutdown <AuxTel-Non-Standard-Operations-AuxTel-Emergency-Shutdown>`
+
+
+
 
 Per component
 =============
@@ -33,26 +102,23 @@ Per component
 
     ATCS/index.rst
 
-
 .. toctree::
     :maxdepth: 3
     :titlesonly:
     :glob:
 
     LATISS/index.rst
 
-
 .. toctree::
     :maxdepth: 3
     :titlesonly:
     :glob:
 
     ATCalSys/index.rst
 
-
 .. toctree::
     :maxdepth: 3
     :titlesonly:
     :glob:
 
-    General-Troubleshooting/index.rst
+    General-Troubleshooting/index.rst
diff --git a/...Non-Standard-Operations/MTCS/TMA/TMA-Restart-MTMount-Operation-Manager-(OM).rst b/...Non-Standard-Operations/MTCS/TMA/TMA-Restart-MTMount-Operation-Manager-(OM).rst
@@ -33,21 +33,6 @@ This document outlines the procedure to restart the MTMount Operation Manager (O
 Restarting the OM may be necessary for troubleshooting or maintenance purposes. 
 The OM is responsible for managing various operations related to the telescope mount. 
 
-.. _TMA-Restart-MTMount-Operation-Manager-Diagnosis:
-
-Error diagnosis
-===============
-
-When trying to change the MTMount CSC from ``STANDBY`` to ``DISABLED`` state:
-
-- MTMount goes to ``FAULT``.
-- It could not retrieve telemetry from the CCW.
-- Raises the following error message: 
-
-.. code-block:: text
-  :caption: MTMount Error Message when transitioning from ``STANDBY`` to ``DISABLED``
-
-  RuntimeError: Telemetry client timed out waiting for telemetry; giving up.
 
 .. _TMA-Restart-MTMount-Operation-Manager-Steps:
 
@@ -132,7 +117,7 @@ Wait for up to 5 minutes before proceeding to the next step and check that the O
 
 6. **Open the EUI Application**: Open the EUI application to resume operations. If you don't know how to open it, refer to the "Opening TMA EUI" section in `this document <https://confluence.lsstcorp.org/display/OOD/How+create+a+VNC+connection+to+TMA+EUI>`__.
 
-
+.. _TMA-Restart-MTMount-Operation-Manager-Post-Condition:
 
 Post-Condition
 ==============