Skip to content

Commit

Permalink
Lint things
Browse files Browse the repository at this point in the history
  • Loading branch information
Luke Sikina committed Oct 11, 2024
1 parent e749611 commit 702c264
Show file tree
Hide file tree
Showing 42 changed files with 709 additions and 895 deletions.
31 changes: 31 additions & 0 deletions .github/workflows/linting.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven

# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.

name: Java CI with Maven

on:
push:
branches: [ "main", "release" ]
pull_request:
branches: [ "main", "release" ]

jobs:
build:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
- name: Set up JDK
uses: actions/setup-java@v4
with:
java-version: '21'
distribution: 'temurin'
cache: maven
- name: Lint Project
run: mvn spotless:check
58 changes: 14 additions & 44 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,20 @@ The Dictionary Project's goal is to create a unified process for enabling the fi
of a query. It needs to work across our current projects, and, in anticipation of our push to productize PIC-SURE, it
needs to be workable for new PIC-SURE installations.

## Contribution

### Requirements

- Java 21
- Maven

### Setup

1. Copy the linting commit hook:
`cp code-formatting/pre-commit.sh .git/hooks/pre-commit`
2. Do a clean build of the project:
`mvn clean install`

## Usage

### Create Env File
Expand All @@ -39,47 +53,3 @@ docker exec -ti dictionary-db psql -U picsure dictionary
```shell
docker compose up -d --build
```

## Solution

### UI

The Dictionary Project's end result will be an overhaul of the PIC-SURE Query Builder tab. It will provide the user with
three ways to find variables they wish to query on:

- An expandable tree of variables and their concept paths
- A series of facets on the page's left gutter
- A search bar

The tree of variables can be filtered using the facets or the search bar. Users might also need additional information
to support the filtering options available to them. The UI will have modals that provide meta information about variables, facets, facet categories, and, in relevant environments, datasets.

### API

The frontend will be supported by a new Dictionary API. The API will have four core resources:

- Concepts (`/concepts/`): Variables, the concept paths to them, and associated metadata. The concepts API will support listing (`/concepts/`), viewing details (`/concepts/<dataset>/<concept path>`), and viewing a tree (`/concepts/tree/<dataset>/<concept path>?depth=[1-9]+`). The listing API will return a paginated list of concepts; the response will also be filtered using a universal filter object. That object will be shared among all API endpoints, and will be
described later. The tree API returns a hierarchy of concept nodes descending from the requested node, limiting the response to a depth of `depth`. The details API will be used for viewing details such as meta fields, type information, related facets, and related harmonized variables.
- Facets (`/facets/`): Facets, their categories, and associated metadata. The facets API will support listing (`/facets/`), and viewing details (`/facets/<facet title>/<facet id>`). The listing API will be filtered using the universal filter object. While facets are technically two-dimensional, their hierarchies are quite shallow and broad, so a traditional listing structure will be used, with no pagination. The details API will be used for viewing meta details, related facets, and potentially variable counts for a facet.
- datasets (`/datasets/`): datasets, and their metadata. There datasets API will support viewing details (`/datasets/<dataset id>`)
- updates (`/updates/`): update all business objects. This will not be a published, user facing API to start. Instead, it will be for ETL purposes. In the future, we may allow admin users access to this API so that they can update dictionaries in live environments.

The universal filter object can be `POST`ed to filterable listing APIs to filter results. It has the following structure:

```json
{
"search": "search terms",
"facets": [
{
"facet_category": "facet title",
"facet_id": "facet id"
}
]
}
```

### Architecture

The Dictionary project has a UI, API, database, and an ETL. The UI will be part of the unified, next generation monolith currently in progress. The API will be isolated to its own docker container, reachable via the proxy endpoints in the PIC-SURE API. The database will be handled on a per-project basis; in some instances it will make more sense to isolate the database to its own container, but in others, it will remain part of the PIC-SURE relational database monolith.

The queries we will be making to the relational database are in many instances complex, and well outside the standards set out by SQL. This means that this project will likely be locked into whatever database we choose, as the tech debt for moving to a different RDMS would not be justifiable. We would like to use Postgres for this project.
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
@SpringBootApplication
public class DictionaryApplication {

public static void main(String[] args) {
SpringApplication.run(DictionaryApplication.class, args);
}
public static void main(String[] args) {
SpringApplication.run(DictionaryApplication.class, args);
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -22,25 +22,19 @@ public class ConceptController {
private Integer MAX_DEPTH;


public ConceptController(
@Autowired ConceptService conceptService
) {
public ConceptController(@Autowired ConceptService conceptService) {
this.conceptService = conceptService;
}


@PostMapping(path = "/concepts")
public ResponseEntity<Page<Concept>> listConcepts(
@RequestBody Filter filter,
@RequestParam(name = "page_number", defaultValue = "0", required = false) int page,
@RequestBody Filter filter, @RequestParam(name = "page_number", defaultValue = "0", required = false) int page,
@RequestParam(name = "page_size", defaultValue = "10", required = false) int size
) {
PageRequest pagination = PageRequest.of(page, size);
PageImpl<Concept> pageResp = new PageImpl<>(
conceptService.listConcepts(filter, pagination),
pagination,
conceptService.countConcepts(filter)
);
PageImpl<Concept> pageResp =
new PageImpl<>(conceptService.listConcepts(filter, pagination), pagination, conceptService.countConcepts(filter));

return ResponseEntity.ok(pageResp);
}
Expand All @@ -52,35 +46,26 @@ public ResponseEntity<Page<Concept>> dumpConcepts(
) {
PageRequest pagination = PageRequest.of(page, size);
PageImpl<Concept> pageResp = new PageImpl<>(
conceptService.listDetailedConcepts(new Filter(List.of(), "", List.of()), pagination),
pagination,
conceptService.listDetailedConcepts(new Filter(List.of(), "", List.of()), pagination), pagination,
conceptService.countConcepts(new Filter(List.of(), "", List.of()))
);

return ResponseEntity.ok(pageResp);
}

@PostMapping(path = "/concepts/detail/{dataset}")
public ResponseEntity<Concept> conceptDetail(
@PathVariable(name = "dataset") String dataset,
@RequestBody() String conceptPath
) {
return conceptService.conceptDetail(dataset, conceptPath)
.map(ResponseEntity::ok)
.orElse(ResponseEntity.notFound().build());
public ResponseEntity<Concept> conceptDetail(@PathVariable(name = "dataset") String dataset, @RequestBody() String conceptPath) {
return conceptService.conceptDetail(dataset, conceptPath).map(ResponseEntity::ok).orElse(ResponseEntity.notFound().build());
}

@PostMapping(path = "/concepts/tree/{dataset}")
public ResponseEntity<Concept> conceptTree(
@PathVariable(name = "dataset") String dataset,
@RequestBody() String conceptPath,
@PathVariable(name = "dataset") String dataset, @RequestBody() String conceptPath,
@RequestParam(name = "depth", required = false, defaultValue = "2") Integer depth
) {
if (depth < 0 || depth > MAX_DEPTH) {
return ResponseEntity.badRequest().build();
}
return conceptService.conceptTree(dataset, conceptPath, depth)
.map(ResponseEntity::ok)
.orElse(ResponseEntity.notFound().build());
return conceptService.conceptTree(dataset, conceptPath, depth).map(ResponseEntity::ok).orElse(ResponseEntity.notFound().build());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -34,40 +34,35 @@ AND concept_node_meta.KEY IN (:disallowed_meta_keys)
""";

private static final String CONSENT_QUERY = """
dataset.dataset_id IN (
SELECT
consent.dataset_id
FROM consent
LEFT JOIN dataset ON dataset.dataset_id = consent.dataset_id
WHERE
concat(dataset.ref, '.', consent.consent_code) IN (:consents) OR
(dataset.ref IN (:consents) AND consent.consent_code = '')
UNION
SELECT
dataset_harmonization.harmonized_dataset_id
FROM consent
JOIN dataset_harmonization ON dataset_harmonization.source_dataset_id = consent.dataset_id
LEFT JOIN dataset ON dataset.dataset_id = dataset_harmonization.source_dataset_id
WHERE
concat(dataset.ref, '.', consent.consent_code) IN (:consents) OR
(dataset.ref IN (:consents) AND consent.consent_code = '')
) AND
""";
dataset.dataset_id IN (
SELECT
consent.dataset_id
FROM consent
LEFT JOIN dataset ON dataset.dataset_id = consent.dataset_id
WHERE
concat(dataset.ref, '.', consent.consent_code) IN (:consents) OR
(dataset.ref IN (:consents) AND consent.consent_code = '')
UNION
SELECT
dataset_harmonization.harmonized_dataset_id
FROM consent
JOIN dataset_harmonization ON dataset_harmonization.source_dataset_id = consent.dataset_id
LEFT JOIN dataset ON dataset.dataset_id = dataset_harmonization.source_dataset_id
WHERE
concat(dataset.ref, '.', consent.consent_code) IN (:consents) OR
(dataset.ref IN (:consents) AND consent.consent_code = '')
) AND
""";

@Autowired
public ConceptFilterQueryGenerator(
@Value("${filtering.unfilterable_concepts}") List<String> disallowedMetaFields
) {
public ConceptFilterQueryGenerator(@Value("${filtering.unfilterable_concepts}") List<String> disallowedMetaFields) {
this.disallowedMetaFields = disallowedMetaFields;
}

/**
* This generates a query that will return a list of concept_node IDs for the given filter.
* <p>
* A filter object has a list of facets, each belonging to a category. Within a category,
* facets are ORed together. Between categories, facets are ANDed together.
* In SQL, this is represented as N clauses for N facets, all INTERSECTed together. Search
* also acts as its own special facet here.
* This generates a query that will return a list of concept_node IDs for the given filter. <p> A filter object has a list of facets,
* each belonging to a category. Within a category, facets are ORed together. Between categories, facets are ANDed together. In SQL,
* this is represented as N clauses for N facets, all INTERSECTed together. Search also acts as its own special facet here.
*
* @param filter universal filter object for the page
* @param pageable pagination, if applicable
Expand Down Expand Up @@ -113,8 +108,7 @@ ORDER BY max((1 + rank) * coalesce(rank_adjustment, 1)) DESC, q.concept_node_id
LIMIT :limit
OFFSET :offset
""";
params.addValue("limit", pageable.getPageSize())
.addValue("offset", pageable.getOffset());
params.addValue("limit", pageable.getPageSize()).addValue("offset", pageable.getOffset());
}

superQuery = " concepts_filtered_sorted AS (\n" + superQuery + "\n)";
Expand Down Expand Up @@ -150,26 +144,26 @@ private String createValuelessNodeFilter(String search, List<String> consents) {
continuous_max.value <> '' OR
categorical_values.value <> ''
)
""".formatted(rankQuery, rankWhere, consentWhere);
"""
.formatted(rankQuery, rankWhere, consentWhere);
}

private List<String> createFacetFilter(Filter filter, MapSqlParameterSource params) {
String consentWhere = CollectionUtils.isEmpty(filter.consents()) ? "" : CONSENT_QUERY;
return filter.facets().stream()
.collect(Collectors.groupingBy(Facet::category))
.entrySet().stream()
.map(facetsForCategory -> {
params
// The templating here is to namespace the params for each facet category in the query
.addValue("facets_for_category_%s".formatted(facetsForCategory.getKey()), facetsForCategory.getValue().stream().map(Facet::name).toList())
.addValue("category_%s".formatted(facetsForCategory.getKey()), facetsForCategory.getKey());
String rankQuery = "0";
String rankWhere = "";
if (StringUtils.hasLength(filter.search())) {
rankQuery = "ts_rank(searchable_fields, (phraseto_tsquery(:search)::text || ':*')::tsquery)";
rankWhere = "concept_node.searchable_fields @@ (phraseto_tsquery(:search)::text || ':*')::tsquery AND";
}
return """
return filter.facets().stream().collect(Collectors.groupingBy(Facet::category)).entrySet().stream().map(facetsForCategory -> {
params
// The templating here is to namespace the params for each facet category in the query
.addValue(
"facets_for_category_%s".formatted(facetsForCategory.getKey()),
facetsForCategory.getValue().stream().map(Facet::name).toList()
).addValue("category_%s".formatted(facetsForCategory.getKey()), facetsForCategory.getKey());
String rankQuery = "0";
String rankWhere = "";
if (StringUtils.hasLength(filter.search())) {
rankQuery = "ts_rank(searchable_fields, (phraseto_tsquery(:search)::text || ':*')::tsquery)";
rankWhere = "concept_node.searchable_fields @@ (phraseto_tsquery(:search)::text || ':*')::tsquery AND";
}
return """
(
SELECT
facet__concept_node.concept_node_id AS concept_node_id,
Expand All @@ -187,8 +181,7 @@ facet.name IN (:facets_for_category_%s ) AND facet_category.name = :category_%s
facet__concept_node.concept_node_id
)
""".formatted(rankQuery, rankWhere, consentWhere, facetsForCategory.getKey(), facetsForCategory.getKey());
})
.toList();
}).toList();
}

}
Loading

0 comments on commit 702c264

Please sign in to comment.