Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Centralize data year logic [DHIS2-16705] #19281

Merged
merged 126 commits into from
Nov 25, 2024
Merged
Show file tree
Hide file tree
Changes from 120 commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
b8a1966
fix: Update code
larshelge Nov 13, 2024
81e861b
fix: Update code
larshelge Nov 13, 2024
952657b
Merge branch 'master' into lars-dev
larshelge Nov 13, 2024
5eaeb3d
fix: Update code
larshelge Nov 13, 2024
2ac1d9d
fix: Update code
larshelge Nov 13, 2024
1a7ecbe
fix: Update code
larshelge Nov 13, 2024
3b1d516
fix: Update code
larshelge Nov 13, 2024
91e44ac
fix: Update code
larshelge Nov 13, 2024
1b46af6
fix: Update code
larshelge Nov 13, 2024
334379c
Merge branch 'master' into lars-dev
larshelge Nov 13, 2024
32de857
fix: Update code
larshelge Nov 13, 2024
23408ba
fix: Update code
larshelge Nov 13, 2024
10990c4
fix: Update code
larshelge Nov 13, 2024
99afbad
fix: Update code
larshelge Nov 13, 2024
f16ef96
fix: Update code
larshelge Nov 13, 2024
e292a23
fix: Update code
larshelge Nov 13, 2024
486776f
Merge branch 'master' into lars-dev
larshelge Nov 13, 2024
ca741d1
fix: Update code
larshelge Nov 14, 2024
8fdc695
fix: Update code
larshelge Nov 14, 2024
2c8abe8
fix: Update code
larshelge Nov 14, 2024
1da4c6a
fix: Update code
larshelge Nov 14, 2024
7dab403
fix: Update code
larshelge Nov 14, 2024
cf9e3b6
fix: Update code
larshelge Nov 14, 2024
a025ce6
fix: Update code
larshelge Nov 14, 2024
5a19093
Merge branch 'master' into lars-dev
larshelge Nov 14, 2024
b48fea5
fix: Update code
larshelge Nov 14, 2024
5910541
Merge branch 'master' into lars-dev
larshelge Nov 14, 2024
5c98cbe
Merge branch 'master' into lars-dev
larshelge Nov 15, 2024
9a8f4e4
fix: Update code
larshelge Nov 15, 2024
33dd76a
fix: Update code
larshelge Nov 15, 2024
f3847d1
fix: Update code
larshelge Nov 15, 2024
e3e2215
fix: Update code
larshelge Nov 15, 2024
4213d99
Merge branch 'master' into lars-dev
larshelge Nov 15, 2024
612c587
fix: Update code
larshelge Nov 15, 2024
c4ecc24
Merge branch 'master' into lars-dev
larshelge Nov 15, 2024
fd347f5
fix: Update code
larshelge Nov 15, 2024
3df3f70
fix: Update code
larshelge Nov 15, 2024
127f25a
fix: Update code
larshelge Nov 17, 2024
2af1fec
fix: Update code
larshelge Nov 17, 2024
7649246
Merge branch 'master' into lars-dev
larshelge Nov 17, 2024
221c724
Merge branch 'master' into lars-dev
larshelge Nov 17, 2024
06bffce
fix: Update code
larshelge Nov 17, 2024
655fee2
fix: Update code
larshelge Nov 18, 2024
8cbda99
fix: Update code
larshelge Nov 18, 2024
0eb8bcc
fix: Update code
larshelge Nov 18, 2024
d26bcb3
fix: Update code
larshelge Nov 18, 2024
c0722ec
Merge branch 'master' into lars-dev
larshelge Nov 18, 2024
e9423c3
fix: Update code
larshelge Nov 18, 2024
2c20571
fix: Update code
larshelge Nov 18, 2024
40d6c33
fix: Update code
larshelge Nov 18, 2024
6f5a1d6
Merge branch 'master' into lars-dev
larshelge Nov 18, 2024
f532475
fix: Update code
larshelge Nov 18, 2024
51e26bc
fix: Update code
larshelge Nov 18, 2024
6a2ace3
fix: Update code
larshelge Nov 18, 2024
3ceff8d
fix: Update code
larshelge Nov 18, 2024
fa4ffb9
Merge branch 'master' into lars-dev
larshelge Nov 18, 2024
13f8a01
fix: Update code
larshelge Nov 18, 2024
e27caad
fix: Update code
larshelge Nov 18, 2024
2ae8e58
Merge branch 'master' into lars-dev
larshelge Nov 18, 2024
5ed2410
fix: Update code
larshelge Nov 18, 2024
c828aa6
fix: Update code
larshelge Nov 18, 2024
8ce9a2d
fix: Update code
larshelge Nov 18, 2024
aaa7557
fix: Update code
larshelge Nov 18, 2024
b5a0cd2
Merge branch 'master' into lars-dev
larshelge Nov 18, 2024
fbcc596
fix: Update code
larshelge Nov 18, 2024
15dba5a
fix: Update code
larshelge Nov 18, 2024
b4ebd28
fix: Update code
larshelge Nov 18, 2024
e55866f
fix: Update code
larshelge Nov 18, 2024
001db10
Merge branch 'master' into lars-dev
larshelge Nov 18, 2024
387e0f6
fix: Update code
larshelge Nov 18, 2024
c9ac624
Merge branch 'master' into lars-dev
larshelge Nov 19, 2024
ff84d2a
fix: Update code
larshelge Nov 19, 2024
e13eabb
fix: Update code
larshelge Nov 19, 2024
5969b6a
fix: Update code
larshelge Nov 19, 2024
7c23c8b
fix: Update code
larshelge Nov 19, 2024
f892775
fix: Update code
larshelge Nov 19, 2024
c7dacee
Merge branch 'master' into lars-dev
larshelge Nov 19, 2024
7496a38
Merge branch 'master' into lars-dev
larshelge Nov 19, 2024
463c511
fix: Update code
larshelge Nov 19, 2024
7107ded
fix: Update code
larshelge Nov 19, 2024
d388a71
fix: Update code
larshelge Nov 19, 2024
30c7a56
fix: Update code
larshelge Nov 19, 2024
fcc344c
fix: Update code
larshelge Nov 19, 2024
ef53068
fix: Update code
larshelge Nov 19, 2024
d7aeca2
fix: Update code
larshelge Nov 19, 2024
8a3e048
fix: Update code
larshelge Nov 19, 2024
a1e925d
Merge branch 'master' into lars-dev
larshelge Nov 19, 2024
0e88ebb
fix: Update code
larshelge Nov 19, 2024
26805ce
fix: Update code
larshelge Nov 19, 2024
ade5f9e
fix: Update code
larshelge Nov 20, 2024
0d80d0d
Merge branch 'master' into lars-dev
larshelge Nov 20, 2024
36309e9
fix: Update code
larshelge Nov 20, 2024
0f53e47
Merge branch 'master' into lars-dev
larshelge Nov 20, 2024
403a868
Merge branch 'master' into lars-dev
larshelge Nov 20, 2024
36e3f11
fix: Update code
larshelge Nov 21, 2024
d2f49d5
fix: Update code
larshelge Nov 21, 2024
9f44151
Merge branch 'master' into lars-dev
larshelge Nov 21, 2024
d9ebb3c
Merge branch 'master' into lars-dev
larshelge Nov 22, 2024
c554378
fix: Update code
larshelge Nov 23, 2024
e2b4ccf
fix: Update code
larshelge Nov 23, 2024
1661309
fix: Update code
larshelge Nov 23, 2024
824df0a
fix: Update code
larshelge Nov 23, 2024
dbe3169
fix: Update code
larshelge Nov 23, 2024
b87bb9b
fix: Update code
larshelge Nov 23, 2024
f5d9436
fix: Update code
larshelge Nov 23, 2024
fa22cce
fix: Update code
larshelge Nov 23, 2024
84c1022
fix: Update code
larshelge Nov 23, 2024
38eabcc
fix: Update code
larshelge Nov 23, 2024
8e1ec27
fix: Update code
larshelge Nov 23, 2024
d947ea6
fix: Update code
larshelge Nov 23, 2024
710e4f5
Merge branch 'master' into lars-dev
larshelge Nov 23, 2024
16445cd
Merge branch 'master' into lars-dev
larshelge Nov 24, 2024
879b90c
fix: Update code
larshelge Nov 25, 2024
17322b7
Merge branch 'master' into lars-dev
larshelge Nov 25, 2024
5840c47
Merge branch 'master' into lars-dev
larshelge Nov 25, 2024
6ab60b0
fix: Update code
larshelge Nov 25, 2024
9f9c559
fix: Update code
larshelge Nov 25, 2024
446333d
Merge branch 'master' into lars-dev
larshelge Nov 25, 2024
6ac28e2
fix: Update code
larshelge Nov 25, 2024
785146f
fix: Update code
larshelge Nov 25, 2024
41bb367
fix: Update code
larshelge Nov 25, 2024
7133dc9
fix: Update code
larshelge Nov 25, 2024
86a8a98
fix: Update code
larshelge Nov 25, 2024
bb70007
fix: Update code
larshelge Nov 25, 2024
5c70bde
fix: Update code
larshelge Nov 25, 2024
9e7738c
Merge branch 'master' into lars-dev
larshelge Nov 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,6 @@
import static org.hisp.dhis.db.model.DataType.GEOMETRY;
import static org.hisp.dhis.db.model.DataType.INTEGER;
import static org.hisp.dhis.db.model.DataType.TEXT;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.SYSTEM_DEFINED;
import static org.hisp.dhis.util.DateUtils.toLongDate;
import static org.hisp.dhis.util.DateUtils.toMediumDate;

Expand Down Expand Up @@ -150,8 +148,7 @@ public List<AnalyticsTable> getAnalyticsTables(AnalyticsTableUpdateParams params
isSpatialSupport());

List<Integer> availableDataYears =
periodDataProvider.getAvailableYears(
analyticsTableSettings.getMaxPeriodYearsOffset() == null ? SYSTEM_DEFINED : DATABASE);
periodDataProvider.getAvailableYears(analyticsTableSettings.getPeriodSource());

return params.isLatestUpdate()
? getLatestAnalyticsTables(params)
Expand Down Expand Up @@ -330,8 +327,7 @@ protected List<String> getPartitionChecks(Integer year, Date endDate) {
@Override
public void populateTable(AnalyticsTableUpdateParams params, AnalyticsTablePartition partition) {
List<Integer> availableDataYears =
periodDataProvider.getAvailableYears(
analyticsTableSettings.getMaxPeriodYearsOffset() == null ? SYSTEM_DEFINED : DATABASE);
periodDataProvider.getAvailableYears(analyticsTableSettings.getPeriodSource());
Integer firstDataYear = availableDataYears.get(0);
Integer latestDataYear = availableDataYears.get(availableDataYears.size() - 1);
Program program = partition.getMasterTable().getProgram();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@
import static org.hisp.dhis.db.model.DataType.VARCHAR_50;
import static org.hisp.dhis.db.model.constraint.Nullable.NOT_NULL;
import static org.hisp.dhis.db.model.constraint.Nullable.NULL;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.SYSTEM_DEFINED;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.SYSTEM_DEFINED;
import static org.hisp.dhis.util.DateUtils.toLongDate;
import static org.hisp.dhis.util.DateUtils.toMediumDate;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@
import static org.hisp.dhis.external.conf.ConfigurationKey.ANALYTICS_TABLE_SKIP_COLUMN;
import static org.hisp.dhis.external.conf.ConfigurationKey.ANALYTICS_TABLE_SKIP_INDEX;
import static org.hisp.dhis.external.conf.ConfigurationKey.ANALYTICS_TABLE_UNLOGGED;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.SYSTEM_DEFINED;
import static org.hisp.dhis.util.ObjectUtils.isNull;

import com.google.common.collect.Lists;
Expand All @@ -50,6 +52,7 @@
import org.hisp.dhis.db.model.Database;
import org.hisp.dhis.db.model.Logged;
import org.hisp.dhis.external.conf.DhisConfigurationProvider;
import org.hisp.dhis.period.PeriodDataProvider.PeriodSource;
import org.hisp.dhis.setting.SystemSettings;
import org.hisp.dhis.setting.SystemSettingsProvider;
import org.springframework.stereotype.Component;
Expand Down Expand Up @@ -92,6 +95,15 @@ public Integer getMaxPeriodYearsOffset() {
return yearsOffset < 0 ? null : yearsOffset;
}

/**
* Returns the {@link PeriodSource} based on the max years offset.
*
* @return the {@link PeriodSource}.
*/
public PeriodSource getPeriodSource() {
return getMaxPeriodYearsOffset() == null ? SYSTEM_DEFINED : DATABASE;
}

/**
* Indicates whether an analytics database instance is configured.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,12 @@ public final class AnalyticsPeriodCriteriaUtils {
*
* @param criteria {@link EventsAnalyticsQueryCriteria} query criteria.
* @param periodDataProvider {@link EventsAnalyticsQueryCriteria} period data provider.
* @param dataSource {@link PeriodDataProvider.DataSource} source of data.
* @param dataSource {@link PeriodDataProvider.PeriodSource} source of data.
*/
public static void defineDefaultPeriodForCriteria(
EnrollmentAnalyticsQueryCriteria criteria,
PeriodDataProvider periodDataProvider,
PeriodDataProvider.DataSource dataSource) {
PeriodDataProvider.PeriodSource dataSource) {
List<Integer> availableYears = periodDataProvider.getAvailableYears(dataSource);

if (PeriodCriteriaUtils.hasPeriod(criteria)) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,6 @@

import static java.time.temporal.ChronoUnit.YEARS;
import static java.util.Comparator.reverseOrder;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.SYSTEM_DEFINED;
import static org.hisp.dhis.scheduling.JobProgress.FailurePolicy.SKIP_ITEM;

import com.google.common.collect.Lists;
Expand Down Expand Up @@ -185,8 +183,7 @@ private final List<ResourceTable> getApprovalResourceTables() {
*/
List<Integer> getAndValidateAvailableDataYears() {
List<Integer> availableYears =
periodDataProvider.getAvailableYears(
analyticsTableSettings.getMaxPeriodYearsOffset() == null ? SYSTEM_DEFINED : DATABASE);
periodDataProvider.getAvailableYears(analyticsTableSettings.getPeriodSource());
validateYearsOffset(availableYears);
return availableYears;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
import static org.hisp.dhis.db.model.DataType.TEXT;
import static org.hisp.dhis.db.model.DataType.TIMESTAMP;
import static org.hisp.dhis.db.model.Table.STAGING_TABLE_SUFFIX;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.DATABASE;
import static org.hisp.dhis.test.TestBase.createDataElement;
import static org.hisp.dhis.test.TestBase.createProgram;
import static org.hisp.dhis.test.TestBase.createProgramStage;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
import static org.hisp.dhis.db.model.DataType.TIMESTAMP;
import static org.hisp.dhis.db.model.Table.STAGING_TABLE_SUFFIX;
import static org.hisp.dhis.db.model.constraint.Nullable.NULL;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.DATABASE;
import static org.hisp.dhis.system.util.SqlUtils.quote;
import static org.hisp.dhis.test.TestBase.createCategory;
import static org.hisp.dhis.test.TestBase.createCategoryCombo;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ void testDefineDefaultPeriodForCriteria_without_period() {

// when
AnalyticsPeriodCriteriaUtils.defineDefaultPeriodForCriteria(
criteria, periodDataProvider, PeriodDataProvider.DataSource.SYSTEM_DEFINED);
criteria, periodDataProvider, PeriodDataProvider.PeriodSource.SYSTEM_DEFINED);

// then
assertEquals(
Expand All @@ -69,7 +69,7 @@ void testDefineDefaultPeriodForCriteria_with_period() {

// when
AnalyticsPeriodCriteriaUtils.defineDefaultPeriodForCriteria(
criteria, periodDataProvider, PeriodDataProvider.DataSource.SYSTEM_DEFINED);
criteria, periodDataProvider, PeriodDataProvider.PeriodSource.SYSTEM_DEFINED);

// then
assertNull(criteria.getStartDate());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
package org.hisp.dhis.resourcetable;

import static java.time.temporal.ChronoUnit.YEARS;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.DATABASE;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
import static java.util.Collections.sort;
import static java.util.Collections.unmodifiableList;
import static org.apache.commons.collections4.CollectionUtils.isEmpty;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.SYSTEM_DEFINED;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.SYSTEM_DEFINED;

import java.util.ArrayList;
import java.util.List;
Expand All @@ -56,30 +56,30 @@ public class PeriodDataProvider {

private final JdbcTemplate jdbcTemplate;

public enum DataSource {
public enum PeriodSource {
/** Fixed boundaries. Backwards compatible. */
SYSTEM_DEFINED,
/** Dynamic from data in database. */
DATABASE
}

/**
* Returns a distinct union of all years available in the "event" table + "period" table, both
* from aggregate and tracker, with 5 years previous and future additions.
*
* <p>ie: [extra_5_previous_years, data_years, extra_5_future_year]
* <p><code>[extra_5_previous_years, data_years, extra_5_future_year]</code>
*
* @param source the source ({@link DataSource}) where the years wil be retrieved from.
* @param source the source ({@link PeriodSource}) where the years wil be retrieved from.
* @return an unmodifiable list of distinct years and the respective additions.
*/
public List<Integer> getAvailableYears(DataSource source) {
public List<Integer> getAvailableYears(PeriodSource source) {
List<Integer> availableDataYears = new ArrayList<>();

if (source == SYSTEM_DEFINED) {
// Add default hard-coded years (keeps it backward compatible).
for (int year = DEFAULT_FIRST_YEAR_SUPPORTED; year <= DEFAULT_LATEST_YEAR_SUPPORTED; year++) {
availableDataYears.add(year);
}
} else {
// Add years dynamically, based on the database.
availableDataYears.addAll(fetchAvailableYears());
addSafetyBuffer(availableDataYears, BEFORE_AND_AFTER_DATA_YEARS_SUPPORTED);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@
import static org.hisp.dhis.period.PeriodDataProvider.BEFORE_AND_AFTER_DATA_YEARS_SUPPORTED;
import static org.hisp.dhis.period.PeriodDataProvider.DEFAULT_FIRST_YEAR_SUPPORTED;
import static org.hisp.dhis.period.PeriodDataProvider.DEFAULT_LATEST_YEAR_SUPPORTED;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.SYSTEM_DEFINED;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.SYSTEM_DEFINED;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@
import static org.hisp.dhis.common.RequestTypeAware.EndpointAction.QUERY;
import static org.hisp.dhis.common.RequestTypeAware.EndpointItem.ENROLLMENT;
import static org.hisp.dhis.common.cache.CacheStrategy.RESPECT_SYSTEM_SETTING;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.DataSource.SYSTEM_DEFINED;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.DATABASE;
import static org.hisp.dhis.period.PeriodDataProvider.PeriodSource.SYSTEM_DEFINED;
import static org.hisp.dhis.security.Authorities.F_PERFORM_ANALYTICS_EXPLAIN;
import static org.hisp.dhis.system.grid.GridUtils.toCsv;
import static org.hisp.dhis.system.grid.GridUtils.toHtml;
Expand Down
Loading