calibration.tex

%auto-ignore
\section{\label{sec:dom_calibration}Calibration and Stability}

Regular calibration of each individual DOM allows translation of the
recorded signals into a measurement of incident Cherenkov light at a
particular time, the basis of particle event reconstructions.  The
primary DOM calibration routine is DOMCal (section~\ref{sec:domcal}), a
calibration procedure run yearly on in-ice DOMs and monthly on IceTop
DOMs that provides the constants used to convert DOM waveforms into
physical units (section~\ref{sec:waveformcal}).  Global time calibration across the array of DOMs is 
provided by the RAPCal procedure (section~\ref{sect:dom:rapcal}) that runs during
data-taking.  The relative and absolute optical efficiency of the DOMs is
determined both by laboratory and \emph{in situ} measurements
(section~\ref{sec:domeff}). The stability of these calibrations is also
relevant, as not every calibration parameter can be tracked continuously during
data-taking (sections~\ref{sec:baselines} to \ref{sec:optical_stability}).
Understanding the statistical properties and time-dependence of the background ``dark
noise'' is also important for low-energy neutrino analyses, supernova
searches, and detector simulation (section~\ref{sect:darknoise}).

The determination of the optical properties of the ice is crucial to
the understanding of IceCube data but is beyond the scope of this work.
Further details can be found in refs.~\cite{Aartsen:2013rt,IC3:spice_lea}.

\subsection{\label{sec:domcal}DOMCal}

The calibration of the PMT waveforms recorded by the DOMs, i.e. translation
of digitized samples into voltage and time, as well as accounting for the
gain of the PMT itself, is achieved via DOM-by-DOM calibration constants
determined by the DOMs themselves. The primary reference inputs to
the DOM calibration software (DOMCal) are a precision electronic
pulser circuit providing known charges; a circuit providing a reference DC
bias voltage level; the 20 MHz oscillator on the Main
Board, used as the timing reference; and single photoelectrons, either from
background ``dark noise'' photons or a low-luminosity LED on the Main
Board.  The Main Board oscillator is manufactured by Vectron
International (model C2560A-0009 TCXO) and has a certified stability
of roughly $1 \times 10^{-11}$ for a sample interval of 5 seconds~\cite{ICECUBE:DAQ}.
The Main Board LED is used to illuminate the PMT on the same DOM, as opposed
to the LED flashers described in 
section~\ref{sec:flasher} which stimulate other DOMs. This LED produces
zero to a few tens of photoelectrons at the PMT~\cite{ICECUBE:DAQ}. Analysis and fitting of the
calibration results is done by the DOMCal software, with the results being
transmitted from each DOM to the surface as an XML file of fit parameters.

Because the operating conditions for
the in-ice DOMs are so stable, DOMCal is only run once a year on the full
detector. IceTop DOMs are calibrated once per month in order to track
temperature-dependent changes.  Calibration is
typically performed on half the detector at a time, the other half
remaining in data-taking mode in case of a transient event (such as a
supernova) during calibration. The total calibration run length is a
little under three hours. Because the calibration procedure produces
light, these runs are excluded from normal analyses.

First, the discriminator circuits used to launch the DOM are calibrated
using the electronic pulser circuit, which is capacitively coupled into the
analog front-end before the Delay Board~\cite{ICECUBE:DAQ}.  This
calibration is later refined using actual 
PMT waveforms, once the PMT gain is known.  Next, the ATWD voltage levels
are calibrated by sweeping the input DC bias voltage and recording the
corresponding waveforms at each DC level.  Because of the slight variations in the ATWD circuits, this
calibration provides an individual linear relationship between ATWD counts
and voltage for each time sample and gain channel.

The average ATWD offset at zero input voltage, or baseline, is needed for
charge integration and can in principle be determined using the previous
ATWD calibration. However, in
practice, these baselines are extremely sensitive to the operating
condition of the DOMs, and since data-taking conditions cannot be exactly
replicated while running DOMCal, the baselines used during data-taking are
determined instead by using averaged forced triggers taken during a normal
run (section~\ref{sec:baselines}).  DOMCal can still use baselines that it measures
for numerical charge integration during the calibration procedure.

The highest-gain channels of each ATWD are calibrated using the electronic
pulser circuit, and then the gains of the other ATWD channels and the fADC
are determined by varying the pulser output and comparing the charges
simultaneously recorded in multiple gain channels.  This relative
calibration is later refined using PMT waveforms stimulated by the Main
Board LED.

The tunable ATWD sampling speed is calibrated by digitizing the Main Board oscillator
waveform and recording the number of clock cycles as a function of ATWD
speed setting; the relationship between configuration setting and sampling
speed is fit with a quadratic polynomial.  The fADC sampling speed is
slaved to the 20 MHz oscillator, which is used as a timing reference, and
so is not separately calibrated.  The relative timing of 
the ATWD and fADC waveforms is determined using the electronic pulser circuit;
non-linear fitting of the digitized waveforms to known pulse templates is
required in order to determine the fADC time offset to the required accuracy.
The transit time of the PMT and Delay Board as a function of PMT high
voltage is determined by calculating the delay between the digitized
current pulse through the Main Board LED and the observed light pulse in
the ATWDs.

The PMT gain as a function of high voltage is calibrated using background
``dark noise'' photons --- the charge $e$ prior to amplification is quantized
and known.  At each voltage level, a histogram of many observed charges is
recorded, where the charge is determined by numerical integration of the waveform.
Each histogram is fit with the sum of an exponential and a Gaussian
function (figure~\ref{fig:domcal_hvfit}).  The peak of the Gaussian component is used
to 
determine the amplification of the PMT at each voltage, and a linear fit
of $\log_{10}(\mathrm{gain})$ versus $\log_{10}(\mathrm{voltage})$ allows
the high voltage of each PMT to be tuned to the desired operating point ($10^7$
for in-ice DOMs; figure~\ref{fig:domcal_hv_settings}).  Small 
corrections ($3-5\%$) to the gain of each DOM are determined using charge
distributions recorded by the Processing and Filtering system
(section~\ref{sect:online:filter}) during normal data-taking.  These
corrections are used to eliminate a systematic difference in the charge as
determined by DOMCal's numerical integration and the waveform pulse
unfolding used in data processing. 

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.6\textwidth]{graphics/dom/domcal/hvfit.pdf}
 \caption{Sample SPE charge spectrum at a gain of $10^{7.5}$ as recorded by DOMCal and fit
   \textit{in situ} with the sum of an exponential and a Gaussian function.}
 \label{fig:domcal_hvfit}
\end{figure}

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.6\textwidth]{graphics/dom/domcal/inice_hv_2016.pdf}
 \caption{Distribution of PMT high voltages at $10^7$ gain for in-ice DOMs.}
 \label{fig:domcal_hv_settings}
\end{figure}

\subsection{\label{sec:waveformcal}Waveform Calibration}

The deposited charge is the basis for all energy measurements in IceCube. The
calibration constants needed are 1) the pedestal, the value the digitizer
reads when the input voltage is zero, and 2) the gain, the input voltage change
required to increase the readout value by one count.  An ATWD waveform
consists of readouts from 128 effectively independent 
digitizers, while an fADC waveform consists of successive outputs of a 
pipelined digitizer (256 samples). Two calibration constants per digitizer are needed to turn each of these
raw readout values (an integral number of ADC counts) into a
voltage, from which the deposited charge is calculated.

The baseline is the mean of the pedestals of each sample in a waveform, while the pedestal pattern is
the deviation of each sample's pedestal from the common baseline.  The fADC
has no additional pedestal pattern, but for the ATWD, it is important to distinguish
between the pedestals of individual samples 
and the common baseline of the entire waveform.  In order to remove the
sample-dependent offset, the DOM subtracts the pedestal pattern from
the ATWD waveform before sending it to the surface.  Thereafter, both ATWD
and fADC waveforms can be calibrated by first subtracting the common
baseline from each sample, then multiplying by the gain. Correct charge
measurement and energy reconstruction depends on correct measurement
of the baseline, as discussed in section~\ref{sec:baselines}.

The pedestal pattern is computed by the DAQ software at the beginning of each 
run by averaging 200 forced-readout waveforms.  Accidental single photoelectrons in
the individual waveforms are averaged out, but a single coincident bright event can
result in an offset in the pedestal average.  In order to avoid such
light contamination, a second average is computed, and the
autocorrelation coefficient of the difference between the two pedestal
averages is computed at two different lags.  This autocorrelation test detects averages in which a
single waveform contains at least a 15 PE pulse (approximately 0.1 PE in the
average).  If light contamination is detected, the procedure is repeated.
The shift between the baseline of the pairs is also calculated to verify
that the baseline is stable.  This procedure ensures that fewer than 1 DOM
in 1000 runs will contain a contaminated baseline.

The baseline is set to about 10\% of the maximum value of the
digitizer counts in
order to capture signals that go below the baseline. Since 2012, the
baseline value for each DOM is configured (section~\ref{sect:online:daqdomconfig}) in order to ensure 
stability. The baseline value differs for each digitizer channel in
each DOM, ranging from 112 to 161 counts in the fADC and 109 to 172
counts in the ATWD. The baselines for each digitizer channel in each DOM are measured with
beacon hits, forced triggers that are collected during normal data
acquisition at a rate of 1.2~Hz per DOM
in in-ice DOMs and 4.8~Hz in IceTop DOMs. Beacon waveforms
from the fADC and ATWD of a typical DOM are shown in figure~\ref{fig:raw_baselines}.

\begin{figure}[!h]
  \captionsetup[subfigure]{labelformat=empty}
  \centering
  \subfloat[]{\includegraphics[width=0.5\textwidth]{graphics/dom/reliability/fadc_average_beacon_raw.pdf}}
  \subfloat[]{\includegraphics[width=0.5\textwidth]{graphics/dom/reliability/atwd_average_beacon_raw.pdf}}
  \caption{Averaged beacon waveforms from the fADC (left) and ATWD (right) of
    an IceCube DOM. The waveforms are an average of approximately $1000$ beacon
   launches. The baseline, which is the mean value of the
    beacon waveform, is shown as a horizontal red line. Typical raw SPE
    waveforms are inset.}
  \label{fig:raw_baselines}
\end{figure}

The digitizer gain is measured by DOMCal (section~\ref{sec:domcal}), a single
value for the fADC waveform and a sample-dependent value for the ATWD
waveform. The calibrated waveform voltage is then

\begin{equation}
  \mathrm{voltage} = \mathrm{(counts - baseline) \times gain}
\end{equation}

The waveform start time is then corrected for the transit time,
the average time it takes a pulse to propagate through the entire
PMT and Delay Board. The transit time correction $t_{\mathrm{transit}}$ is
dependent on the PMT high voltage $V$:

\begin{equation}
  t_{\mathrm{transit}} = \frac{m}{\sqrt{V}} + b
\end{equation}

\noindent where $m$ and $b$ are determined by DOMCal. The typical value of $m$
is 2000~ns$\cdot \sqrt{\mathrm{V}}$, and the typical value of $b$ is
80~ns, which includes the 75~ns delay line of the Main Board. The
typical transit time is therefore 130~ns.  Waveform start times from the
second ATWD chip and the fADC are further 
corrected for the delay $\Delta t$ with respect to the first ATWD
chip as determined by DOMCal, so the total start time correction is
$t_{\mathrm{transit}} + \Delta t$. 

Finally, the waveforms are corrected for the effects of droop from the
transformer that couples the Main Board to the PMT output. The toroid
coupling effectively acts as a high-pass filter on the PMT output
that makes the tails of the waveforms ``droop'' and even
undershoot the baseline. This effect is temperature-dependent and is larger
at lower temperatures. The droop correction inverts the effect of the high-pass
filter and eliminates the undershoot in the waveform tails. This is
done by calculating the expected reaction voltage from the toroid at
each sample time, and adding the reaction voltage to the calibrated waveform
to compensate. The reaction voltage decays 
according to a temperature-dependent model of the transformer behavior. 
When a readout contains consecutive launches from the same
DOM, the reaction voltage at the end of the last launch is used to
correct for the residual droop in the following launch.  DOMs
use two types of toroid transformers: the ``old toroid'' with a short
time constant that was used in early DOM production, and a ``new
toroid'' with a longer time constant that produces less
distortion. Of 5484~DOMs deployed in IceCube, 1204 are the old
toroid type, and 4280 are the new toroid type. The full correction is
modeled with two time constants, 
where the DOM's transient response $\tilde{\delta}(t)$ to an input
signal $\delta(t)$ is given by

\begin{equation}
\tilde{\delta}(t) = \delta (t) - N((1 - f) e^{-t/\tau_1} +f\ e^{-t/\tau_2})
\label{eq:droop}
\end{equation}

\noindent where the first time constant $\tau_1$ is given by 

\begin{equation}
  \tau_1(T) = A + \frac{B}{1 + e^{(-T/C)}}\ .
  \label{eq:tau1}
\end{equation}

\noindent In eqs.~\ref{eq:droop} and \ref{eq:tau1}, $T$ is the
temperature, and the constants $A$, $B$, $C$, $N$ and $f$ were
determined empirically with a dedicated analysis. For the old toroids, the
second time constant $\tau_2 =0.75\tau_1$. For the new toroids, $\tau_2$ is 500~ns.

\subsection{\label{sect:dom:rapcal}RAPCal}

The Reciprocal Active Pulsing Calibration (RAPCal) is a
continuously-running procedure for translating hit timestamps based on the 
individual free-running DOM clocks to the synchronized surface clocks in the
ICL, in order to establish a standard time base for the array to
$O(\mathrm{ns})$ accuracy.  Subsequently, the ICL-based hit timestamps can
then be translated to UTC.  The scheme used for the time transfer involves
transmission of a pulse to and from each DOM and is shown in
figure~\ref{fig:rapcal_symmetry}.  The base implementation is described in
\cite{ICECUBE:DAQ}; we describe here the details of the 
algorithm and validation of the DOM relative time calibration. 

While data transmission is paused, a bipolar pulse is generated and sent to
each DOM over the power/communications wire pair.  After 
this pulse has been received, a bipolar pulse --- having the same shape
as the one generated on the surface --- is generated in the DOM and sent
back to the surface.  The pulses are timestamped using the local
clock count at each transmission and reception point.  The symmetry of the
pulses sent in each direction and symmetry of the down- and up- cable
transmission enables the time transfer from each free-running DOM clock to the
surface clock and hence to UTC, without prior knowledge of the cable length
or its dispersive properties.

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.6\textwidth]{graphics/dom/rapcal/rapcal_symmetry.pdf}
 \caption{Diagram of the RAPCal time calibration scheme (not to scale).
   Each transmitted and received pair of signals is
   timestamped $(T_{\mathrm{tx}},T_{\mathrm{rx}})$ in the local clock domain, and by the symmetry of the
   situation the midpoints $T_C$ between transmitted and received timestamps are
   synchronous.} 
 \label{fig:rapcal_symmetry}
\end{figure}

\begin{figure}[h]
 \centering
 \includegraphics[width=0.6\textwidth]{graphics/dom/rapcal/dom_wf_zero_crossing.pdf}
 \caption{Digitized RAPCal pulse as received by a DOM, after cable dispersion.  The
   zero-crossing of the baseline-subtracted pulse is used as a fine-delay
   correction $f$ to the received timestamp $T_{\mathrm{rx}}$.}
 \label{fig:rapcal_zero_crossing}
\end{figure}

The received dispersed bipolar pulses are digitized by a 20 MHz communications ADC and 
timestamped using the local clock count.  The received pulse timestamp
corresponds to a fixed delay after the start of the final pulse recovery.  A RAPCal pulse sequence to and
from each DOM therefore results in a series of four timestamps 
$T_{\mathrm{tx}}^{\mathrm{DOR}}$, $T_{\mathrm{rx}}^{\mathrm{DOM}}$, 
$T_{\mathrm{tx}}^{\mathrm{DOM}}$,  and $T_{\mathrm{rx}}^{\mathrm{DOR}}$
(figure~\ref{fig:rapcal_symmetry}).  These timestamps initially count periods
of the DOM's 40 MHz clock and of the 20 MHz clock of the surface
electronics (DOR, section~\ref{sect:online:master_clock}); they are
translated into common units for the calculations described below.

Next, in order to improve the precision beyond the sampling speed of the
communications ADC, a ``fine delay'' correction $f$ to $T_{\mathrm{rx}}^{\mathrm{DOM}}$ and
$T_{\mathrm{rx}}^{\mathrm{DOR}}$ is calculated by interpolating 
to find the negative-going baseline crossing, with the baseline voltage
calculated using the initial samples of the waveform (figure~\ref{fig:rapcal_zero_crossing}):

\begin{equation}
  \tilde{T}_{\mathrm{rx}} = T_{\mathrm{rx}} - f\ .
\end{equation}


\noindent The midpoints $T_C^{\mathrm{DOR,DOM}}$ as shown in
figure~\ref{fig:rapcal_symmetry} are then determined, where

\begin{equation}
  T_C =  \frac{T_{\mathrm{tx}} + \tilde{T}_{\mathrm{rx}}}{2}\ .
\end{equation}

\noindent $T_C^{\mathrm{DOM}}$ and $T_C^{\mathrm{DOR}}$ then identify a
single point in time across the two clock domains.

The next stage of the process is to translate an arbitrary DOM hit timestamp
$t$ to UTC.  In typical operating conditions, the RAPCal procedure
is repeated once per second for each DOM.  We use the two nearest RAPCal
results before and after $t$ to derive a linear relationship

\begin{equation}
  \mathrm{UTC}(t) = (1+\epsilon)(t - T_C^{\mathrm{DOM}}) +
  T_C^{\mathrm{DOR}} + \Delta\ .
\end{equation}

\noindent The slope $(1+\epsilon)$ accounts for drifts in the DOM
clocks and is calculated from the $T_C$ pairs of the two neighboring RAPCal
results: 

\begin{equation}
  1+\epsilon = \frac{T_{C,2}^{\mathrm{DOR}} -
    T_{C,1}^{\mathrm{DOR}}}{T_{C,2}^{\mathrm{DOM}} -
    T_{C,1}^{\mathrm{DOM}}}\ .
\end{equation}

\noindent The median magnitude of $\epsilon$ is $1.34 \times 10^{-6}$.
Finally, because the timestamps $T^{\mathrm{DOR}}$ count the sub-second
time offset into the current UTC second, the UTC time offset $\Delta$ of the previous
1-second boundary, provided by the master clock, is added.  Details on the
master clock system are provided in section~\ref{sect:online:master_clock}.

The stability and repeatability of the calibration is monitored by
tracking the cable delay from RAPCal, determined by

\begin{equation}
  T_{\mathrm{cable}} = \frac{1}{2} \left( ( \tilde{T}_{\mathrm{rx}}^{\mathrm{DOR}} -
  T_{\mathrm{tx}}^{\mathrm{DOR}} ) - (1+\epsilon)(T_{\mathrm{tx}}^{\mathrm{DOM}} -
  \tilde{T}_{\mathrm{rx}}^{\mathrm{DOM}} )\right) \ .
\end{equation}

\noindent A representative distribution of $T_{\mathrm{cable}}$ from one DOM over an 8-hour
data-taking run is shown in figure~\ref{fig:rapcal_cable_len}, with a
standard deviation of 0.6 ns.  Individual RAPCal measurements in which $T_{\mathrm{cable}}$
deviates by more than 10 ns from a moving average are discarded.

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.6\textwidth]{graphics/dom/rapcal/tcal_hist_22-60.pdf}
 \caption{Distribution of one-way cable delays from multiple RAPCal
   measurements on DOM 22-60 (bottom of String 22), shown with a Gaussian fit.}
 \label{fig:rapcal_cable_len}
\end{figure}

The time calibration through the entire data acquisition and software
processing chain is verified using the LED flashers. During
commissioning, performed on each string after deployment, all 12~LEDs on
each DOM are flashed simultaneously at 
maximum brightness, and the arrival times of the first photons at the DOM above the flashing DOM
are recorded. Given the vertical spacing of 17~m on a standard IceCube
string and the group index of
refraction of  1.356 in ice at 400~nm \cite{price_woschnagg_ice}, the expected light travel time
from the flashing DOM to the DOM above is 77~ns. In DeepCore, the DOM
vertical spacings are 10~m and 7~m, corresponding to light travel
times of 45~ns and 32~ns respectively. The mean light travel
time to the DOM above for all flashing DOMs in ice is shown in
figure~\ref{fig:flashertiming}. The mean arrival time agrees with the
expectation for the DeepCore DOMs. For the standard DOMs, the observed
light travel time is about 3~ns longer than the expected light travel
time, due to the effects of scattering in the ice over the longer
distance. The accuracy of the photon arrival time with respect to the
arrival time of any other photon is measured using
the difference between arrival times at the two DOMs above the
flasher, which eliminates any uncertainty in the flasher source
time. Figure~\ref{fig:flashertiming} shows the distribution of the
difference in arrival times at the two DOMs above the flasher for the
DOMs with 7~m spacing, as the DOMs with larger spacing are more
affected by scattering in ice. The width of the distribution is
1.7~ns, so the measured timing accuracy is 1.7/$\sqrt{2}$~ns $=$ 1.2~ns. Muons are also used to
verify the time calibration~\cite{ICECUBE:DAQ}, including the absolute time difference
between the IceTop surface stations and the in-ice DOMs~\cite{IC3:perf}.

\begin{figure}[!h]
  \captionsetup[subfigure]{labelformat=empty}
  \centering
  \subfloat[]{\includegraphics[width=0.495\textwidth]{graphics/dom/rapcal/flashmean.pdf}}
  \subfloat[]{\includegraphics[width=0.5\textwidth]{graphics/dom/rapcal/oneabove.pdf}}
  \caption{Left: Time from flasher emission to detection at the DOM above for 17~m vertical spacing
    (red), 10~m vertical spacing (blue) and 7~m vertical spacing
    (green). The expected light travel times in ice for each distance are marked with
    vertical lines. Right: arrival time difference (blue) between the two
    DOMs above the flasher for DOMs with 7~m vertical spacing.  The
    vertical red line denotes expected light
    travel time in ice, and the mean of a Gaussian fit to the distribution
    (dashed blue line) is 33~ns, due to
    the effects of scattering in ice. The width of the distribution is 1.7~ns.}
  \label{fig:flashertiming}
\end{figure}

\subsection{\label{sec:domeff}DOM Optical Efficiency}

A baseline value for the photon detection efficiency has been established
by combining PMT measurements at 337~nm and a separate model of wavelength-
and angle-dependent effects.  Absolute sensitivity measurements were
performed on 13 IceCube PMTs, using single photons from a primary 337~nm laser beam of calibrated
intensity~\cite{ICECUBE:PMT}. The results agree well with independent
Hamamatsu measurements of sensitivity in the range 270--730~nm, which
were then normalized to the 337~nm measurement.  The resulting quantum
efficiency at the center of the PMT at 390~nm is 25\%.  A full
simulation model of the DOM includes the wavelength dependence of the PMT
response, optical absorption in the DOM glass and gel, discriminator
threshold effects, and photocathode non-uniformity.  The angular dependence is
dominated by the shape of the photocathode and its response variation away
from the center, which was measured for 135 bare
PMTs~\cite{ICECUBE:PMT}. The efficiency was also measured for 16 fully integrated
DOMs, including glass and gel; the response of an example DOM is shown
in figure~\ref{fig:goldendom}. The efficiency measurement used the
337~nm nitrogen laser, as well as LEDs at 365~nm, 470~nm, 520~nm, and
572~nm due to the low transparency of the glass at short wavelengths.

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.55\textwidth]{graphics/dom/performance/GDOM-NemuriNeko-2D-colz.pdf}
 \caption{Position dependence of the response of a DOM, normalized to
   the absolute efficiency at 365~nm. The x-y coordinates measure
   distance from the center of the DOM.}
 \label{fig:goldendom}
\end{figure}

The efficiency model as determined from the laboratory measurements was supplemented with \textit{in
  situ} measurements using Cherenkov light from muons in the ice. The
efficiency measured \textit{in situ} includes effects of the cable
shadowing the DOM and the refrozen ``hole ice'' in the vicinity of the
DOM. In one
study, low-energy muons (median energy 82~GeV) with well-understood light
emission were selected to illuminate the downward-facing PMTs in the ice as
directly as possible. The number of photons detected at different
distances from these muons was then compared to predictions of the
simulation model~\cite{IC3:ereco}.  Based on this and other \textit{in
  situ} analyses, the central value for the efficiency was adjusted upward
by 10\% in the simulation, compared to the baseline.  For physics analyses,
the normalization of the absolute DOM efficiency is generally included as a
nuisance parameter with prior uncertainty of 10\% that includes other uncertainties related to
generation and propagation of the light. Additional laboratory measurements
on assembled DOMs are in progress, including wavelength- and angle-dependent
effects, and are expected to reduce uncertainties~\cite{ICECUBE:DOMEFF}.

The absolute calibration at 337~nm was performed at room temperature on a
small subset of IceCube PMTs. The relative efficiency of all assembled DOMs
was separately measured as part of production testing
(section~\ref{sec:dom_prodtest}), using a 405~nm pulsed laser and a system of
fibers and diffusers to illuminate DOMs within 50$^{\circ}$ of the
PMT axis.  Using this system, the relative efficiency of DeepCore DOMs
(high quantum efficiency type) was measured to be higher by a factor
of 1.39
compared to standard IceCube DOMs, agreeing well with the
manufacturer-specified value of 1.40. Further \textit{in situ} studies
using muons yielded a factor of 1.35~\cite{ICECUBE:DC}, an
effective value including the Cherenkov spectrum folded with the different
wavelength sensitivity curves of the two types of PMTs.  The production
testing system also established that the efficiency change from room
temperature to $-40\ ^{\circ}\mathrm{C}$ is less than 1\% when gain is maintained at
the design value of $10^7$.

\subsection{\label{sec:baselines}Baseline Voltage Stability}

The beacon hits from which the digitizer baselines are derived
(section~\ref{sec:waveformcal}) are 
monitored continuously throughout the year. The average values of the
beacon baselines are very stable, with shifts of no more than
0.2~counts per year, corresponding to 0.018~mV in the fADC and
0.025~mV in the high-gain ATWD channel. The baseline shifts from May
2015 to April 2016 are shown in figure~\ref{fig:baseline_stability_2015}. 

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.6\textwidth]{graphics/dom/reliability/baseline_stability_2015.pdf}
 \caption{Distribution of shifts in baseline values in all ATWD
   channels and the fADC for all DOMs from May 2015 to April
   2016. The DOM configuration was unchanged during this period.}
 \label{fig:baseline_stability_2015}
\end{figure}

Every year when the detector
is recalibrated, adjustments in the SPE discriminator threshold can
cause shifts of up to 0.6~counts (0.54~mV) in the fADC
baselines, for reasons not completely understood. ATWD baselines are
unaffected by the SPE discriminator 
setting. Correcting recorded waveforms for the effect of transformer
coupling as described in section~\ref{sec:waveformcal} has
the side effect that small DC offset errors are converted into an
apparent PMT current that increases linearly in time over the course
of a few microseconds. The effect is stronger for DOMs with old-type toroid
transformers, where a baseline error of 0.6~counts can 
turn an actual deposited charge of 1~PE into a measured charge of over
2~PE with an unphysical time structure. The observed distortion in a
simulated single photoelectron charge due to fADC baseline shifts is
shown in figure~\ref{fig:charge_fadcshift}.  To avoid this problem, whenever the
discriminator thresholds are changed, the fADC baselines are re-measured,
and the values used for calibration are updated. As long as 
the discriminator thresholds are unchanged, the baselines are stable
to within 0.2~counts
as shown above, and no charge distortion is seen at that level of
baseline stability.

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.8\textwidth]{graphics/dom/reliability/charge_fadcshift.pdf}
 \caption{Reconstructed charge from a simulated single photoelectron
   deposit as a function of fADC baseline shift, for both old- and
   new-toroid DOMs. The shaded region indicates the observed range of 
   baseline variation from the nominal value in ice; no observable
   distortion in the charge spectrum is seen at these values.}
 \label{fig:charge_fadcshift}
\end{figure}

The baselines can be sensitive to radio-frequency interference (RFI). In
2009, RF signals from the COBRA meteor radar transmitter broadcasting at
46.3~MHz \cite{meteor_radar}
appeared as sinusoidal or spiky modulations in the waveform
baselines.  To mitigate this effect, one DOM was operated without PMT high
voltage and used to monitor the RF broadcasts and tag periods of RFI in
order to avoid data contamination.  Also
in 2009, RFI from DOM 68-42 affected nearby DOMs after it failed during DOM
calibration and appeared to begin sparking, resulting in sinusoidal distortions
in the baselines of neighboring DOMs. The meteor radar
transmitter is no longer operating, and investigations into RFI
from more recent radar activities at South Pole (SuperDARN
\cite{superdarn}) have not revealed any measurable interference, either in DOM
baselines or in RAPCal time stability. 

\subsection{\label{sec:gain_stability}Gain Stability}

The gain stability of the DOM, or the stability of the amplified charge
from converted photons, depends on a number of factors including stability
of the PMT high voltage, Main Board channel amplifiers, and the digitizers.
We can examine these subsystems using both historical calibration results
and by tracking the SPE charge during data-taking.

Variations of the front-end electronic amplifiers or the digitizers
themselves can potentially lead to changes in the overall gain of the DOM electronics.
The stability is demonstrated by comparing the Main Board channel amplifier gains
from sets of calibrations taken from 2011 to 2016
(figure~\ref{fig:domcal_ch_gain}).  From year to year, the amplifier gain 
calibration is repeatable to 0.1\%, 0.2\%, and 0.5\% in the high-gain,
medium-gain, and low-gain channels respectively.  Since detector completion
in 2011, a small systematic shift of $-0.3\%$ is visible in the low-gain
channel, but this is corrected by updating the calibration constants of
each DOM.

\begin{figure}[!h]
  \captionsetup[subfigure]{labelformat=empty}
  \centering
  \subfloat[]{\includegraphics[width=0.5\textwidth]{graphics/dom/reliability/channel_gain_shift_2016_2015.pdf}}
  \subfloat[]{\includegraphics[width=0.5\textwidth]{graphics/dom/reliability/channel_gain_shift_2016_2011.pdf}}
  \caption{Fractional ATWD channel amplifier gain shifts, from 2015 to
    2016 (left) and 2011 to 2016 (right).  Channels 0, 1, and 2 are
    high gain, medium gain, and low gain respectively.}
  \label{fig:domcal_ch_gain}
\end{figure}

The stability of the total gain, including both the PMT and
front-end electronics, is monitored during data-taking using the
single photoelectron peak of the charge distribution on each DOM from
cosmic ray muons. A
Gaussian + exponential function is fit to the peak region as in
figure~\ref{fig:spe_fit_thresh}, and the mean of the Gaussian is
tracked throughout the year. The threshold, defined as the bin in the histogram for which 99\% of the area of the histogram is contained in the sum of all higher bins, is also tracked through
the year. The peak position is calibrated to 1~PE
and is stable to within 0.01~PE for 95\% of all DOMs, as shown in
figure~\ref{fig:gain_spe_stability}. Over 99\% of DOMs show no
measurable change in the threshold as long as the discriminator
thresholds are unchanged; these settings are only changed once per year.

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.6\textwidth]{graphics/dom/reliability/chargedist.pdf}
 \caption{Charge distribution of a typical in-ice DOM. The threshold
   is marked in red, and a Gaussian + exponential fit to
   the SPE region is shown in blue. The mean of the Gaussian is used
   to monitor the gain stability.}
 \label{fig:spe_fit_thresh}
\end{figure}

\begin{figure}[!h]
  \captionsetup[subfigure]{labelformat=empty}
  \centering
  \subfloat[]{\includegraphics[width=0.5\textwidth]{graphics/dom/reliability/mean_value_2015.pdf}}
  \subfloat[]{\includegraphics[width=0.5\textwidth]{graphics/dom/reliability/meandiff_2015.pdf}}
  \caption{Distribution of the mean of the Gaussian fit to the SPE
    peak (left) and the shift in this value for each in-ice DOM (right) between May 2015 and April
   2016.}
  \label{fig:gain_spe_stability}
\end{figure} 

There are about 12~DOMs that show unpredictable, abrupt shifts in the SPE peak
position of 0.05~PE or more. Figure~\ref{fig:gainshift_spe} shows the time history of the
SPE peak position of one of these DOMs over 4~months. The peak shift
corresponds to increases or decreases in the multi-photoelectron (MPE)
scaler rate, where the scaler counts the discriminator crossings.  This
indicates that the SPE peak shift is indeed caused by a change in the DOM
gain. However, the SPE scaler rate is stable to within 2\%, indicating that the
probability to detect single photons is effectively unchanged.

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.8\textwidth]{graphics/dom/reliability/gainshift.pdf}
 \caption{Mean of the Gaussian fit to the SPE peak (top) and the SPE
   scaler rate (middle) and MPE
   scaler rate (bottom) from July 2015 to November 2015 for a DOM
   that shows unpredictable gain shift behavior. The DOM
   configuration was unchanged during this period.}
 \label{fig:gainshift_spe}
\end{figure}

Long-term stability of the PMTs can be tracked by examining any change in
the high voltage vs.~gain calibration over time.  The fractional change in gain for
all in-ice DOMs over a five-year time period is shown in
figure~\ref{fig:pmt_gainshift}.  For most DOMs, the PMT gain is stable to
within $\pm3\%$ over the time period shown, with a median gain shift of $0.3\%$, while approximately $1\%$ 
of DOMs have a gain shift of more than 10\%.  Any shifts are tracked with
regular calibration using the methods previously described.

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.7\textwidth]{graphics/dom/reliability/pmt_gainshift.pdf}
 \caption{Distribution of the fractional changes in PMT gain from 2011 to 2016 for all in-ice
   DOMs, at the 2011 operating high voltage for each DOM.}
 \label{fig:pmt_gainshift}
\end{figure}

\subsection{\label{sec:optical_stability}Optical Efficiency Stability}

The detector response in IceCube is verified with low energy muons as
described in ref.~\cite{IC3:ereco}. The detector response is monitored in each run using the track
detection probability (TDP) calculated from bright muon tracks
with more than 30 hits in IceCube. The muon 
tracks are reconstructed using the likelihood methods described in
ref.~\cite{Ahrens:2003fg}, but charge and time information from the DOM under
study are excluded from the reconstruction. The TDP is
defined for each DOM as the ratio of the number of detected tracks
within $\SI{100}{\meter}$ of the DOM to the total number of tracks within $\SI{100}{\meter}$ of
the DOM. This ratio depends both on the optical properties of the ice
near the DOM and the optical efficiency of the DOM. We do not attempt
to separate
these effects in the TDP, but rather use the TDP to monitor the
overall stability of the detector response. Figure~\ref{fig:tdp} shows the TDP on
String 80, which includes both standard and HQE DOMs; the TDP is
20--25\% higher for HQE DOMs than for neighboring standard
DOMs, whereas the quantum efficiency is 35\% higher. The TDP is stable to within 1\% since 2012, when the baselines
were stabilized by being set in the DAQ configuration. Figure~\ref{fig:tdp} shows
the difference in the TDP for all DOMs between a run in 2012 and a run
in 2015.

\begin{figure}[!h]
  \captionsetup[subfigure]{labelformat=empty}
  \centering
  \subfloat[]{\includegraphics[width=0.5\textwidth]{graphics/dom/reliability/tdp_onestring.pdf}}
  \subfloat[]{\includegraphics[width=0.5\textwidth]{graphics/dom/reliability/tdpcomparison.pdf}}
  \vspace{-1\baselineskip}  
  \caption{Left: track detection probability (TDP) in String 80, with a
    mixture of standard DOMs (blue crosses) and HQE DOMs (red
    circles). The variation with depth is due to the depth-dependent
    optical properties of the ice. Right: shift in TDP for all in-ice DOMs between 2012 and 2015; standard DOMs are in blue, and
    HQE DOMs are in red. The mean of the Gaussian fit to
    the TDP shift in the standard DOMs is $-0.1$\%, and the mean 
    TDP shift in the HQE DOMs is 0.05\%.}
  \label{fig:tdp}
\end{figure}

The detector response stability is also measured with the {\it in
  situ} light sources in IceCube. Both the in-ice calibration laser
\cite{IC3:SC} and the flasher LEDs show less than 1\% difference in the total
charge collected between 2012 and 2015. 

\subsection{\label{sect:darknoise}Dark Noise}

The vast majority of background hits result from dark noise, i.e. effects
that lead to the emission of an electron from the cathode of the PMT in the
absence of a photon source external to the DOM. 
Dark noise is a complex phenomenon with numerous possible sources,
including thermionic emission, electronic noise, field emission 
within the PMT, Cherenkov light from radioactive decays, and
scintillation / luminescence in the glass of the PMT and pressure sphere.
The average total in-ice hit rate is \SI{560}{\hertz} for DOMs with
standard PMTs and \SI{780}{\hertz} for high quantum efficiency DOMs. The 
contribution from cosmic-ray muons, estimated as the in-ice HLC hit rate,
is small and decreases with depth from 25 Hz to 5 Hz. 

The dark noise can be characterized as a combination of uncorrelated
(Poissonian) noise pulses with a rate between \SI{230}{\hertz} and
\SI{250}{\hertz}, and a correlated component, with a pulse rate from
\SI{280}{\hertz} to \SI{340}{\hertz}.  A comparison at low temperature of
DOM dark noise to that of a bare PMT suggests that the majority of the
noise originates from the glass pressure sphere.  Cherenkov light from
$^{40}\mathrm{K}$ decays contributes to the uncorrelated noise component,
and thus the potassium content of the glass was limited.
Measurements of early samples indicated a $\mathrm{K}_2\mathrm{O}$
concentration of 0.03\% by weight, roughly corresponding to 100 Bq of beta
decays per sphere. 

The correlated noise manifests itself as an overabundance of short time
intervals between hits in a single DOM compared to the Poisson expectation
(figure~\ref{fig:darknoise_deltaT}). 
The temperature dependence of the noise rate
(figure~\ref{fig:dom_darknoise_vs_temperature}) was determined by combining a
measured temperature profile of 
the South Pole ice cap \cite{price2002temperature} with a fit of the
Poissonian expectation of the total dark noise rate to every individual
DOM, and was verified in lab measurements.  The temperature of the in-ice
DOMs, measured on the Main Board, increases with depth from
$-31\ ^{\circ}\mathrm{C}$ to $-9\ ^{\circ}\mathrm{C}$
($10\ ^{\circ}\mathrm{C}$ above the ambient ice temperature). 

\begin{figure}
  \centering
  \includegraphics[width=0.7\textwidth]{graphics/dom/performance/darknoise/DarkNoise_Layer2Doms.pdf}
 \caption{Time interval between successive hits for all next-to-top layer
   DOMs (DeepCore excluded).  The line is an exponential fit to the
   Poissonian regime between 7 and 15 ms.}
 \label{fig:darknoise_deltaT}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=0.7\textwidth]{graphics/dom/performance/darknoise/HitRatevsTemp_inice_nomuons_nofit_bigfont.pdf}
  \caption{Dark noise rate of DOMs in ice as a function of temperature,
    obtained from untriggered HitSpool data. Each data point represents the
    average of 12 DOM layers from 78 strings (DeepCore excluded).} 
  \label{fig:dom_darknoise_vs_temperature}
\end{figure}

The short time intervals are clustered in bursts (uninterrupted sequences
of time intervals less than 3 ms) with an average number of
hits per burst increasing from \num{3.3} at \SI{-10}{\celsius} to \num{3.8} at
\SI{-30}{\celsius}. A study with forced-readout data shows that the
phenomenology of the correlated noise component in IceCube is in general in
good agreement with results reported in ref.~\cite{meyer_noise}, but an
unambiguous physical explanation is still to be confirmed.  Stimulation of
glass samples with radioactive sources results in scintillation
\cite{helbing_glass}, suggesting that luminescence of the glass triggered
by the radioactive decay of $^{40}\mathrm{K}$ and other elements in the
pressure sphere is responsible.  Naturally-occurring cerium in the glass is
a candidate for the active scintillator. 

The various sources of dark noise present in a DOM can be separated using
the time between successive untriggered hits in HitSpool 
data (section~\ref{sec:domhub_hitspool}), as shown in 
figure~\ref{fig:darknoise_deltaT_components}. An overview of the various 
noise components and their parameterizations is given in table \ref{tab:noise}.  Late-time
correlated afterpulses, a common feature of PMTs, is attributed to
ionization of residual gases by electrons that were accelerated between
the dynodes.  Although afterpulses occur at various time delays, this
component is parametrized here with a single average timescale.  A
noise model incorporating these various sources is used for detector
simulation \cite{larson2013simulation}. 

\begin{table}[h!]
\caption{Characteristics of noise components in IceCube DOMs, adapted from
  ref.~\cite{stanisha_noise_14}.}
  \centering
  \footnotesize
\begin{tabularx}{\textwidth}{lXXr}
\toprule
Noise Component& Origin & Distribution & Parameters \\
\midrule
afterpulses & PMT & Gaussian & $\mu = \SI{6}{\micro\second}$, $\sigma = \SI{2}{\micro\second}$\\
uncorrelated & thermal noise, \newline radioactive decay & Poissonian & $\lambda \simeq \SI{250}{\hertz}$\\
correlated & luminescence (?) & log-normal &
$\mu = \num{-6}\ [\log_{10}(\delta t/\mathrm{sec})]$, \\
& & & 
$\sigma = \num{0.85}\ [\log_{10}(\delta t/\mathrm{sec})]$ \\
\bottomrule
\end{tabularx}
\label{tab:noise}
\end{table}

\begin{figure}[!h]
 \centering
  \includegraphics[width=0.8\textwidth]{graphics/dom/performance/darknoise/SingleDOM_HitSpool_Hits_deltaT_fits_example.pdf}
 \caption{Histogram of time differences between successive hits from HitSpool data of
DOM 15 on string 27 (blue) on a logarithmic scale in order to visualize the
different noise components \cite{heereman2015hitspooling}.  A fourth
subdominant component centered at \SI{100}{\micro\second} is not parameterized and is still under study.}
 \label{fig:darknoise_deltaT_components}
\end{figure}

The evolution of the dark noise contribution to the total rate over time was investigated using the
supernova scaler stream \cite{IC3:supernova, briedel_phd}, effectively 
the summed dark noise rate of the detector with an artificial deadtime
applied to reduce the effect of correlated noise
(section~\ref{sect:SNDAQ}). A long-term exponential decay in 
the noise rate is visible; this may be caused by 
decreasing triboluminescence \cite{ice_tribo} arising from the initial ``freeze-in''
of newly deployed DOMs, impurities introduced during the drill
process, or a combination of the two effects.  The decay
is especially recognizable in the standard deviation of the scaler
distribution (figure~\ref{fig:noise_over_time_briedel}), which decreases by
25\% over the course of the three years, with the decrease increasing with
depth~\cite{Piegsa09}. Changes in the mean rate are
initially dominated by the decay component and later by the seasonal
variation of atmospheric muons. The noise
rate decay is most pronounced when selecting strings
that were deployed in the final construction season, as shown in
figure~\ref{fig:noise_over_time_briedel_lastseasondepoyed}.

\begin{figure}[!h]
 \centering
 \includegraphics[width=1.0\textwidth]{graphics/dom/performance/darknoise/SN_Scalers_Whole_Detector_Mean_Variance_IC86_2011_2012_2013_smaller_height.pdf}
 \caption{Mean and standard deviation of the supernova scaler distribution for the
   entire detector over the course of the first three years of the
   completed IceCube \cite{briedel_phd}. Spikes are due to calibration
 runs which injected light into the detector, and to fluctuations in
 individual DOMs.} 
 \label{fig:noise_over_time_briedel}
\end{figure}

\begin{figure}[!h]
 \centering
 \includegraphics[width=1.0\textwidth]{graphics/dom/performance/darknoise/SN_Scalers_IC86_mean_variance_Histo_IC86_2011_2012_2013_geomapping.pdf}
 \caption{Mean and standard deviation of the supernova scaler rate of strings
   deployed in the last deployment season (austral summer of 2010/2011)
   \cite{briedel_phd}, where changes in the mean rate are
still dominated by the decay component. Spikes are due to calibration
 runs which injected light into the detector, and to fluctuations in
 individual DOMs.} 
 \label{fig:noise_over_time_briedel_lastseasondepoyed}
\end{figure}

Since the dark noise components are not correlated between
DOMs, the dark noise rate change is not prominent in local coincidence
hits, which are dominated by atmospheric muons (figure
\ref{fig:hlc_over_time_briedel}).  Thus, the detector trigger rate as well
as many higher-level reconstruction variables are relatively isolated from the
noise rate variations.  Nevertheless, the total hit rates for each DOM are
tracked over time and updated yearly in a database for simulation and
reconstruction purposes.  Seasonal variations of these rates are
below the 1\% level.

\begin{figure}[!h]
 \centering
 \includegraphics[width=0.95\textwidth]{graphics/dom/performance/darknoise/HLC_Whole_Detector_Mean_Variance_IC86_2011_2012_2013.pdf}
 \caption{Mean and standard deviation of the local coincidence hit rate distribution
   \cite{briedel_phd}.  Changes in dark noise rate do not
   contribute significantly to changes in the local coincidence hit rate,
  which is dominated by seasonal and weather-related
   changes in the atmospheric muon flux.} 
 \label{fig:hlc_over_time_briedel}
\end{figure}