Writing correlations

[Olango1]

I am trying to use this code to create the correlation database and I keep getting the error below. Please assist.
import yaml

from seismic.db.corr_hdf5 import CorrelationDataBase
from seismic.correlate.stream import CorrStream

For this example, we are just gonna create an empty CorrStream

Of course, this will not really add any data to the file

cst = CorrStream()

Get your correlation dictionary

with open('/path/to/my/params.yaml') as file:
co = yaml.load(file, Loader=yaml.FullLoader)
with CorrelationDataBase('/path/to/myfile.h5', mode='w', corr_options=co) as cdb:
cdb.add_correlation(cst, tag='my_sensible_tag')

I am trying to recreate the results in the example tutorial by creating a new correlationdatabase for use but I am unable to.

KeyError Traceback (most recent call last)
File ~/miniforge3/envs/salvus/lib/python3.11/site-packages/seismic/db/corr_hdf5.py:210, in DBHandler.get_corr_options(self)
209 try:
--> 210 sco = str(self['co'].attrs['co'])
211 # Run once more through co_to_hdf5 to account for
212 # correlations that have been computed with older versions

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File ~/miniforge3/envs/salvus/lib/python3.11/site-packages/h5py/_hl/attrs.py:56, in AttributeManager.getitem(self, name)
54 """ Read the value of an attribute.
55 """
---> 56 attr = h5a.open(self._id, self._e(name))
57 shape = attr.shape

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File h5py/h5a.pyx:80, in h5py.h5a.open()

KeyError: "Can't open attribute (can't locate attribute: 'co')"

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last)
File ~/miniforge3/envs/salvus/lib/python3.11/site-packages/seismic/db/corr_hdf5.py:74, in DBHandler.init(self, path, mode, compression, co, force)
73 try:
---> 74 co_old = self.get_corr_options()
75 if co_old != co_to_hdf5(co) and not force:

File ~/miniforge3/envs/salvus/lib/python3.11/site-packages/seismic/db/corr_hdf5.py:216, in DBHandler.get_corr_options(self)
215 except KeyError:
--> 216 raise KeyError('No correlation options in file')
217 return co

KeyError: 'No correlation options in file'

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last)
Cell In[70], line 31
29 with open('/home/olangoea/Desktop/code_phd/velocity_change/data/params.yaml') as file:
30 co = yaml.load(file, Loader=yaml.FullLoader)
---> 31 with CorrelationDataBase('/home/olangoea/Desktop/code_phd/velocity_change/data/corr/myfile.h5', mode='a', corr_options=co) as cdb:
32 cdb.add_correlation(cst, tag='my_sensible_tag')

File ~/miniforge3/envs/salvus/lib/python3.11/site-packages/seismic/db/corr_hdf5.py:417, in CorrelationDataBase.enter(self)
416 def enter(self) -> DBHandler:
--> 417 self.db_handler = DBHandler(
418 self.path, self.mode, self.compression, self.co, self.force)
419 return self.db_handler

File ~/miniforge3/envs/salvus/lib/python3.11/site-packages/seismic/db/corr_hdf5.py:91, in DBHandler.init(self, path, mode, compression, co, force)
83 raise PermissionError(
84 f'The output file {path} already exists and contains'
85 ' data with'
(...)
88 f'\n{diff}'
89 )
90 except KeyError:
---> 91 self.add_corr_options(co)

File ~/miniforge3/envs/salvus/lib/python3.11/site-packages/seismic/db/corr_hdf5.py:97, in DBHandler.add_corr_options(self, co)
96 def add_corr_options(self, co: dict):
---> 97 sco = str(co_to_hdf5(co))
98 ds = self.create_dataset('co', data=np.empty(1))
99 ds.attrs['co'] = sco

File ~/miniforge3/envs/salvus/lib/python3.11/site-packages/seismic/db/corr_hdf5.py:522, in co_to_hdf5(co)
520 for key in remk:
521 coc.pop(key, None)
--> 522 coc['corr_args'].pop('combinations', None)
523 coc['subdivision'].pop('recombine_subdivision', None)
524 coc['subdivision'].pop('delete_subdivision', None)

KeyError: 'corr_args'

[PeterMakus]

Hi there,
thanks for getting in touch.
The DataBase has an internal protection to make sure that you do not randomly write Correlations with different processing parameters to the file. As I see it, you did not even specify a file.

If you like you wrote, really entered this code

with open('/path/to/my/params.yaml') as file: co = yaml.load(file, Loader=yaml.FullLoader) with CorrelationDataBase('/path/to/myfile.h5', mode='w', corr_options=co) as cdb: cdb.add_correlation(cst, tag='my_sensible_tag')

Then /path/to/my/parms.yaml and /path/to/myfile.h5 are probably not existing. So if you replace the path to the parameter file by an actual existing path with the properly formatted yaml file (see documentation), it should work.

Best,
Peter

[Olango1]

Hello,
I used the correct path for the earlier question which is as follows and I get the error I indicated earlier. The code snippet I shared was just to indicate what I used. So even with the correct path I still get the error
Collect all CorrTrace objects
all_corr_traces = []

Read all files from the directory and convert traces to CorrTrace
for file_name in os.listdir(data_dir):
full_path = os.path.join(data_dir, file_name)
stream = read(full_path) # Read file using ObsPy
for tr in stream: # Iterate over all traces in the stream
corr_trace = CorrTrace(data=tr.data, header1=None, header2=None)
all_corr_traces.append(corr_trace)

Create a CorrStream with all collected CorrTrace objects
cst = CorrStream(traces=all_corr_traces)

Load correlation dictionary
with open('/home/olangoea/Desktop/code_phd/velocity_change/data/params.yaml') as file:
co = yaml.load(file, Loader=yaml.FullLoader)

Use CorrelationDataBase to store the CorrStream data
with CorrelationDataBase('/home/olangoea/Desktop/code_phd/velocity_change/data/corr/myfile.h5', mode='a', corr_options=co) as cdb:
cdb.add_correlation(cst, tag='my_sensible_tag')

Secondly do you have an example you can share on how to add mseed data to corrstream?

Kindly assist.

Best,
Emerald

[PeterMakus]

Hey Emerald,
Ah that makes it a little clearer thanks.

So the reason why you are getting issues is that your CorrTraces do not have any headers. So you either have to provide the kwarg _header(containing the header of the CorrTrace itself, which follows the SEED combination logic (see docs).

This is because SeisMIC will store CorrTraces within the hdf5 file in a hierarchy that depends on the header (also discussed in the link above). If you only provide a numpy array, which is essentially what you are doing, it will not know what to do with the data. Also, you will have to ask yourself, how wise it is to store data without knowing the sampling rate, recording time, station name, etc..

So I don't have a direct example of adding data from MSEED. But in the end, you will have to make sure to build a correct header (CorrStats class in SeisMIC) and just provide the data for each of the Traces -> build a stream like you did in your example.

Hope that helps!

Cheers,
Peter