add QC workflow #46
Merged
add QC workflow #46
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was
linked to
issues
Nov 2, 2023
Closed
4 tasks
Adding this function directly to the Sonde class means that: 1. we can remove the separate qc module as it is no longer needed (all qc checks will be run in the sonde class itself) 2. the fullness without the weights is not going to be needed, so that doesn't need to be a qc function in the Sonde class 3. the weighted_fullness function is now changed so that the variable argument has a default value. This default value is now a dictionary where the keys are variable names expected in the dataset and the values are the corresponding sampling frequencies. Therefore, the sampling_frequency argument is no longer needed. 4. This also means that the separate RD41 config file can be removed. Reading through it will no longer be used. Instead if the user wants to provide different variables/sampling frequencies, they can do so using the main config file, which through the main() function in halodrops.__init__ will be replaced as a non-default value for this function. 5. Similarly, for the timestamp_frequency, the default value is now given as 4. The user can change it if they want. The value of 4 is no longer fixed in the function. 6. Docstrings have been changed to reflect the changes in the function
This commit introduces a new Quality Control (QC) check method `qc_check_profile_fullness` in the `Sonde` class. This method checks if the profile coverage of a `Sonde` instance is above a specified threshold. Key changes: 1. The `qc_check_profile_fullness` method takes an optional `qc_threshold` parameter, which defaults to 0.8. This parameter specifies the threshold for profile coverage. 2. The method checks if the `Sonde` instance has an attribute `profile_coverage`. If it does, the method checks if `profile_coverage` is greater than `qc_threshold`. If it is, the method returns `True`; otherwise, it returns `False`. 3. If the `Sonde` instance does not have a `profile_coverage` attribute, the method raises a `ValueError` with a message instructing the user to run the `weighted_fullness` method first. 4. The `apply_qc_checks` method in the `Sonde` class has been updated to include the new `qc_check_profile_fullness` method in the `qc_functions` dictionary. 5. A reason for the `qc_functions` dictionary to only say "profile fullness" instead of the whole function name is that I am thinking of having all the functions for "qc_checks" start with "qc_check_". This would make the dictionary keys unnecessarily long and also make it clear from the function names that they are QC checks.
This commit introduces two new methods in the `Sonde` class: `near_surface_coverage` and `qc_check_near_surface_coverage`. Key changes: 1. The `near_surface_coverage` method calculates the fraction of non-NaN values in specified variables near the surface. It takes a list of variables, altitude bounds, and the name of the altitude dimension as parameters. It raises a `ValueError` if the `aspen_ds` attribute does not exist. 2. The `qc_check_near_surface_coverage` method checks if the near surface coverage is above a specified threshold. It returns `True` if the coverage is above the threshold and `False` otherwise. It raises a `ValueError` if the `near_surface_coverage` attribute does not exist. 3. The `apply_qc_checks` method in the `Sonde` class now calls the `qc_check_near_surface_coverage` method.
The qc_check function now checks if all provided variables in the `weighted_fullness` function satisfy the qc_threshold. If any don't, the function returns False, else True.
logic same as commit efd1248
Geet-George
force-pushed
the
qc-workflow
branch
from
7fec127
to
cd83387
Compare
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Should close #45