Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature / columnar data optional id field support in update data #771

Merged
merged 141 commits into from
Nov 5, 2024
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
b9f2995
branch created
Jerry-Jinfeng-Guo Oct 1, 2024
3d1bfcc
added test with no ids
Jerry-Jinfeng-Guo Oct 7, 2024
340b740
[skip ci] functionality partially in place; wip
Jerry-Jinfeng-Guo Oct 8, 2024
1edc403
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 8, 2024
8ded552
[skip ci]
Jerry-Jinfeng-Guo Oct 8, 2024
ea6ba47
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 9, 2024
575e24d
[skip ci] added logic for has_id and id_nan
Jerry-Jinfeng-Guo Oct 9, 2024
0ea07fa
[skip ci] more component level properties
Jerry-Jinfeng-Guo Oct 10, 2024
75b1583
[skip ci] minor fix on the independence criteria
Jerry-Jinfeng-Guo Oct 14, 2024
721eec4
[skip ci] added elements per scenario of base; subsequent comparison …
Jerry-Jinfeng-Guo Oct 14, 2024
464ed64
[skip ci] added checks
Jerry-Jinfeng-Guo Oct 14, 2024
d2ce49d
[skip ci] functionality accepting sequence
Jerry-Jinfeng-Guo Oct 14, 2024
3280fa8
[skip ci] functionalities for get sequence in place; next to the upda…
Jerry-Jinfeng-Guo Oct 15, 2024
faa5966
[skip ci] removing id match assert: incompatible with optional id; wh…
Jerry-Jinfeng-Guo Oct 15, 2024
88610d2
[skip ci] empty columnar data input supported
Jerry-Jinfeng-Guo Oct 15, 2024
f584870
[skip ci] made test more complete; next to merge with `main`
Jerry-Jinfeng-Guo Oct 15, 2024
9a11e6a
[skip ci] removed commented code
Jerry-Jinfeng-Guo Oct 15, 2024
892ca09
[skip ci] address some comments
Jerry-Jinfeng-Guo Oct 16, 2024
ec5bc65
[skip ci] more comments addressed
Jerry-Jinfeng-Guo Oct 16, 2024
4d63387
[skip ci] one more comment addressed
Jerry-Jinfeng-Guo Oct 16, 2024
b06d8c5
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 17, 2024
890837a
[skip ci] branch created
Jerry-Jinfeng-Guo Oct 17, 2024
4ca60bd
[skip ci] intermediate commit marking several python tests as fail
Jerry-Jinfeng-Guo Oct 18, 2024
3b0c6b6
removed `id_exists` check and tests
Jerry-Jinfeng-Guo Oct 18, 2024
74e2225
Your code has been rated at 10.00/10, but I throw an error nonetheless
Jerry-Jinfeng-Guo Oct 18, 2024
6bbd9a8
`isort` has its own opinion
Jerry-Jinfeng-Guo Oct 18, 2024
9ea0f15
Merge branch 'main' into feature/update-python-validation-to-support-…
Jerry-Jinfeng-Guo Oct 18, 2024
8fef79c
Merge branch 'main' into feature/update-python-validation-to-support-…
Jerry-Jinfeng-Guo Oct 18, 2024
56cae49
New check logic
Jerry-Jinfeng-Guo Oct 18, 2024
4880c91
explicit optional
Jerry-Jinfeng-Guo Oct 21, 2024
f2a5177
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 21, 2024
fc9b7c8
unit test added
Jerry-Jinfeng-Guo Oct 21, 2024
22b0851
ignored two python tests
Jerry-Jinfeng-Guo Oct 21, 2024
d51d93e
typo
Jerry-Jinfeng-Guo Oct 21, 2024
23f6fcc
clang-tidy
Jerry-Jinfeng-Guo Oct 21, 2024
bf01d2e
clang-tidy 2.0
Jerry-Jinfeng-Guo Oct 21, 2024
64f3483
sonar cloud
Jerry-Jinfeng-Guo Oct 21, 2024
aee0dbb
clang-tidy
Jerry-Jinfeng-Guo Oct 21, 2024
c71493a
unit test added; next to columnar input
Jerry-Jinfeng-Guo Oct 21, 2024
33e2e62
added get id by idx2d
Jerry-Jinfeng-Guo Oct 22, 2024
bee2724
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 22, 2024
e7ff7f4
assert id check
Jerry-Jinfeng-Guo Oct 22, 2024
aa9a1e0
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 22, 2024
e035c3d
update error in calculate
Jerry-Jinfeng-Guo Oct 22, 2024
8599250
not necessarily a regular error type
Jerry-Jinfeng-Guo Oct 22, 2024
b411515
polished the cpi model test to further complete the wrong id input; p…
Jerry-Jinfeng-Guo Oct 23, 2024
d0e8c1f
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 23, 2024
b783aef
Merge branch 'main' into feature/update-python-validation-to-support-…
Jerry-Jinfeng-Guo Oct 23, 2024
5ca7124
repro UB in columnar batch update with unknown ID
mgovers Oct 23, 2024
caa8e1d
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
mgovers Oct 23, 2024
356c23f
extended unit test for validate ids
Jerry-Jinfeng-Guo Oct 23, 2024
f775c2b
extended unit test for validate ids
Jerry-Jinfeng-Guo Oct 23, 2024
5e33695
lambda to functions
Jerry-Jinfeng-Guo Oct 23, 2024
4f0fce4
address linter complain
Jerry-Jinfeng-Guo Oct 24, 2024
e57b446
thou shall not pass; linter 2.0
Jerry-Jinfeng-Guo Oct 24, 2024
4761123
there is his-py and mypy
Jerry-Jinfeng-Guo Oct 24, 2024
a43a365
addressed comment
Jerry-Jinfeng-Guo Oct 25, 2024
ac8082d
format
Jerry-Jinfeng-Guo Oct 25, 2024
2c08abd
revert `ComponentData`
Jerry-Jinfeng-Guo Oct 25, 2024
0fd17d0
fix last test
nitbharambe Oct 25, 2024
65fea67
[skip ci] debug in progress
Jerry-Jinfeng-Guo Oct 25, 2024
250b09f
bug fixed; next to merge with base branch
Jerry-Jinfeng-Guo Oct 26, 2024
384342d
Merge pull request #805 from PowerGridModel/feature/ub-update-unknown-id
Jerry-Jinfeng-Guo Oct 26, 2024
cae7b9c
has AddressSanitizer error on WSL gcc 13.0; corrected python tests
Jerry-Jinfeng-Guo Oct 27, 2024
bf02077
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 27, 2024
5878ad6
native api model remove const
Jerry-Jinfeng-Guo Oct 27, 2024
8f25bce
clang-tidy
Jerry-Jinfeng-Guo Oct 27, 2024
ad11736
change to get comp size
nitbharambe Oct 28, 2024
b672c04
in progress
Jerry-Jinfeng-Guo Oct 28, 2024
55bd9ae
all fixed; up for review
Jerry-Jinfeng-Guo Oct 28, 2024
1ccd250
Merge pull request #793 from PowerGridModel/feature/update-python-val…
TonyXiang8787 Oct 28, 2024
234be96
addressed comments
Jerry-Jinfeng-Guo Oct 28, 2024
10f1a38
wrap function call with _DEBUG
Jerry-Jinfeng-Guo Oct 28, 2024
0bdde42
_DEBUG to NDEBUG
Jerry-Jinfeng-Guo Oct 28, 2024
c880fc5
!NDEBUG
Jerry-Jinfeng-Guo Oct 28, 2024
f66e723
NDEBUG is still complained
Jerry-Jinfeng-Guo Oct 28, 2024
fbbb916
comments addressed
Jerry-Jinfeng-Guo Oct 28, 2024
add533f
removed unused lines in test
Jerry-Jinfeng-Guo Oct 28, 2024
b7e5450
sonar cloud issues
Jerry-Jinfeng-Guo Oct 29, 2024
454af2c
undo error introduced from last commit
Jerry-Jinfeng-Guo Oct 29, 2024
b79a4f9
re-added python validation
Jerry-Jinfeng-Guo Oct 29, 2024
bd459ef
mypy didn't catch mismatch, but his-py did
Jerry-Jinfeng-Guo Oct 29, 2024
c17cc2d
removed static test
Jerry-Jinfeng-Guo Oct 29, 2024
04fcf6a
comments addressed
Jerry-Jinfeng-Guo Oct 29, 2024
b3ec5e8
comments addressed
Jerry-Jinfeng-Guo Oct 29, 2024
1170719
sonar cloud
Jerry-Jinfeng-Guo Oct 29, 2024
c954095
python validation
Jerry-Jinfeng-Guo Oct 29, 2024
0e3bbb8
test deleted by mistake
Jerry-Jinfeng-Guo Oct 29, 2024
20054a3
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 29, 2024
add8802
add test
nitbharambe Oct 29, 2024
20787cf
add fix
nitbharambe Oct 29, 2024
c8e01ab
Merge pull request #812 from PowerGridModel/fix/update-input-data-opt…
Jerry-Jinfeng-Guo Oct 29, 2024
97e7c8b
[skip ci]
Jerry-Jinfeng-Guo Oct 29, 2024
963cac2
address comments
Jerry-Jinfeng-Guo Oct 30, 2024
2f063cf
add row based sparse test
nitbharambe Oct 30, 2024
679e9b8
addressed comments
Jerry-Jinfeng-Guo Oct 30, 2024
b439f33
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Oct 30, 2024
6ca043f
addressed comment
Jerry-Jinfeng-Guo Oct 31, 2024
d52d6ce
constexpr comment
Jerry-Jinfeng-Guo Oct 31, 2024
e29748e
addressed commits; handed over to Martijn and Nitish for final touch
Jerry-Jinfeng-Guo Oct 31, 2024
a8ea09d
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 31, 2024
76df59d
update ids checks
mgovers Nov 4, 2024
e20d96d
fix part of optional ids
mgovers Nov 4, 2024
3d5d0bb
make compile again
mgovers Nov 4, 2024
ff8f44b
most tests pass again
mgovers Nov 4, 2024
ca6960b
fix api model tests
mgovers Nov 4, 2024
4fedd67
attempt cleanup part 1
mgovers Nov 4, 2024
d2c8b03
clang-tidy
mgovers Nov 5, 2024
8ec4bbb
fix get attribute
mgovers Nov 5, 2024
fa71eda
resolve comments
mgovers Nov 5, 2024
a0a4523
remove TODO
mgovers Nov 5, 2024
344d5da
Merge pull request #818 from PowerGridModel/feature/resolve-comments-…
Jerry-Jinfeng-Guo Nov 5, 2024
f0d4baa
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
1e4145b
sonar cloud
Jerry-Jinfeng-Guo Nov 5, 2024
ed8d0a0
format
Jerry-Jinfeng-Guo Nov 5, 2024
7a37897
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
ab3a4d7
minor
mgovers Nov 5, 2024
71f3ee0
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
161161f
use std map find
mgovers Nov 5, 2024
d1b2ccf
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
ca2c1d2
fix validator tests
nitbharambe Nov 5, 2024
2ca0fdb
channeling done; test fixed
Jerry-Jinfeng-Guo Nov 5, 2024
d1f2220
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
0513ffa
Apply suggestions from code review
mgovers Nov 5, 2024
f098446
sonar-cloud
mgovers Nov 5, 2024
1faab9f
minor
mgovers Nov 5, 2024
314af8a
Apply suggestions from code review
mgovers Nov 5, 2024
1344c9f
more resolve comments
mgovers Nov 5, 2024
d53d3ea
fix unused captured variable
mgovers Nov 5, 2024
02bf930
fix variable name
mgovers Nov 5, 2024
aa63dc8
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
8762766
fix typo
mgovers Nov 5, 2024
cfb4d1a
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
mgovers Nov 5, 2024
dba5644
fix error introduced from merging
Jerry-Jinfeng-Guo Nov 5, 2024
80d25eb
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Nov 5, 2024
0a1ea5d
fix merge error
Jerry-Jinfeng-Guo Nov 5, 2024
69b736b
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
2c76f9c
Merge pull request #813 from PowerGridModel/feature/row-update-data-s…
Jerry-Jinfeng-Guo Nov 5, 2024
a6320cf
Revert "remove optional ids tests"
mgovers Nov 5, 2024
849d293
remove pytest mark xfail
mgovers Nov 5, 2024
74f2fef
format
mgovers Nov 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ constexpr double numerical_tolerance = 1e-8;
constexpr double nan = std::numeric_limits<double>::quiet_NaN();
constexpr IntS na_IntS = std::numeric_limits<IntS>::min();
constexpr ID na_IntID = std::numeric_limits<ID>::min();
constexpr Idx na_Idx = std::numeric_limits<Idx>::min();

// power grid constant
constexpr double base_power_3p = 1e6;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,6 @@ class LoadGen final : public std::conditional_t<is_generator_v<appliance_type_>,

// update for load_gen
UpdateChange update(LoadGenUpdate<loadgen_symmetry> const& update_data) {
assert(update_data.id == this->id());
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
this->set_status(update_data.status);
set_power(update_data.p_specified, update_data.q_specified);
// change load connection and/or value will not change topology or parameters
Expand All @@ -112,8 +111,6 @@ class LoadGen final : public std::conditional_t<is_generator_v<appliance_type_>,
LoadGenUpdate<loadgen_symmetry> inverse(LoadGenUpdate<loadgen_symmetry> update_data) const {
double const scalar = direction_ * base_power<loadgen_symmetry>;

assert(update_data.id == this->id());
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved

set_if_not_nan(update_data.status, static_cast<IntS>(this->status()));
set_if_not_nan(update_data.p_specified, real(s_specified_) * scalar);
set_if_not_nan(update_data.q_specified, imag(s_specified_) * scalar);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,6 @@ template <symmetry_tag power_sensor_symmetry_> class PowerSensor : public Generi
}

PowerSensorUpdate<power_sensor_symmetry> inverse(PowerSensorUpdate<power_sensor_symmetry> update_data) const {
assert(update_data.id == this->id());
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved

auto const scalar = convert_direction() * base_power<power_sensor_symmetry>;

set_if_not_nan(update_data.p_measured, real(s_measured_) * scalar);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,6 @@ template <symmetry_tag sym> class VoltageSensor : public GenericVoltageSensor {
u_angle_measured_{voltage_sensor_input.u_angle_measured} {};

UpdateChange update(VoltageSensorUpdate<sym> const& update_data) {
assert(update_data.id == this->id());
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved

double const scalar = 1 / (u_rated_ * u_scale<sym>);

update_real_value<sym>(update_data.u_measured, u_measured_, scalar);
Expand All @@ -95,8 +93,6 @@ template <symmetry_tag sym> class VoltageSensor : public GenericVoltageSensor {
}

VoltageSensorUpdate<sym> inverse(VoltageSensorUpdate<sym> update_data) const {
assert(update_data.id == this->id());
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved

double const scalar = u_rated_ * u_scale<sym>;

set_if_not_nan(update_data.u_measured, u_measured_ * scalar);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,10 @@ class Container<RetrievableTypes<GettableTypes...>, StorageableTypes...> {
}
return result;
}
template <supported_type_c<StorageableTypes...> Storageable> Idx get_group_idx() const {
return static_cast<Idx>(get_cls_pos_v<Storageable, StorageableTypes...>);
}
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved

// get item based on ID
template <supported_type_c<GettableTypes...> Gettable> Gettable& get_item(ID id) {
Idx2D const idx = get_idx_by_id<Gettable>(id);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,12 @@ inline Idx2D get_component_idx_by_id(MainModelState<ComponentContainer> const& s
return state.components.template get_idx_by_id<ComponentType>(id);
}

template <typename ComponentType, class ComponentContainer>
requires model_component_state_c<MainModelState, ComponentContainer, ComponentType>
inline Idx get_component_group_idx(MainModelState<ComponentContainer> const& state) {
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
return state.components.template get_group_idx<ComponentType>();
}

template <typename ComponentType, class ComponentContainer>
requires model_component_state_c<MainModelState, ComponentContainer, ComponentType>
inline Idx get_component_sequence(MainModelState<ComponentContainer> const& state, auto const& id_or_index) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,21 +33,30 @@ template <component_c Component, class ComponentContainer,
std::output_iterator<Idx2D> OutputIterator>
requires model_component_state_c<MainModelState, ComponentContainer, Component>
inline void get_component_sequence(MainModelState<ComponentContainer> const& state, ForwardIterator begin,
ForwardIterator end, OutputIterator destination) {
ForwardIterator end, OutputIterator destination, Idx n_comp_elements) {
figueroa1395 marked this conversation as resolved.
Show resolved Hide resolved
using UpdateType = typename Component::UpdateType;

auto idx_getter_func = [&state, n_comp_elements](UpdateType const& update, auto index) {
if (n_comp_elements == na_Idx) {
mgovers marked this conversation as resolved.
Show resolved Hide resolved
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
return get_component_idx_by_id<Component>(state, update.id);
} else {
Idx group = get_component_group_idx<Component>(state);
return Idx2D{group, index % n_comp_elements};
}
};

std::transform(begin, end, destination,
[&state](UpdateType const& update) { return get_component_idx_by_id<Component>(state, update.id); });
[&, index = 0](UpdateType const& update) mutable { return idx_getter_func(update, index++); });
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
}

template <component_c Component, class ComponentContainer,
forward_iterator_like<typename Component::UpdateType> ForwardIterator>
requires model_component_state_c<MainModelState, ComponentContainer, Component>
inline std::vector<Idx2D> get_component_sequence(MainModelState<ComponentContainer> const& state, ForwardIterator begin,
ForwardIterator end) {
ForwardIterator end, Idx n_comp_elements = na_Idx) {
std::vector<Idx2D> result;
result.reserve(std::distance(begin, end));
get_component_sequence<Component>(state, begin, end, std::back_inserter(result));
get_component_sequence<Component>(state, begin, end, std::back_inserter(result), n_comp_elements);
return result;
}

Expand Down
Loading