Skip to content

Commit

Permalink
feat(api): rework the sdk api to a more class-based approach (#8)
Browse files Browse the repository at this point in the history
* Create common building blocks to interact with the RDS API: RdsServer, RdsCatalog, and RdsDataProduct.
* Add initial testing for these three classes.
* Add dependencies for existing MTNA model libraries


BREAKING CHANGE: No longer singletons, must instantiate the RdsServer, RdsCatalog, and RdsDataProduct via the constructor.

RJ-3
  • Loading branch information
wallace41290 committed May 21, 2020
1 parent 48f3697 commit 35cc217
Show file tree
Hide file tree
Showing 38 changed files with 1,007 additions and 155 deletions.
128 changes: 83 additions & 45 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@
[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg?style=for-the-badge)](https://github.com/semantic-release/semantic-release)
[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=for-the-badge)](https://github.com/prettier/prettier)

<a href="https://www2.richdataservices.com"><img src="https://www2.richdataservices.com/assets/logo.svg" align="left" target="_blank" hspace="10" vspace="6" style="max-width: 200px"></a>
## Checkout our awesome [examples/showcases][examples] repo.
<br>

<a href="https://www.richdataservices.com"><img src="./resources/rds-logo.png" align="left" width="200"></a>

**Rich Data Services** (or **RDS**) is a suite of REST APIs designed by Metadata Technology North America (MTNA) to meet various needs for data engineers, managers, custodians, and consumers. RDS provides a range of services including data profiling, mapping, transformation, validation, ingestion, and dissemination. For more information about each of these APIs and how you can incorporate or consume them as part of your work flow please visit the MTNA website.

Expand All @@ -18,7 +21,7 @@
Make RDS queries easy. Write strongly typed code. Use RDS-JS.

## References
[RDS SDK Documentation](https://mtna.github.io/rds-js/) | [RDS API Documentation](https://covid19.richdataservices.com/rds/swagger/) | [Examples](https://github.com/mtna/rds-js-examples) | [Contributing](CONTRIBUTING.md) | [Developer Documentation](DEVELOPER.md) | [Changelog](https://github.com/mtna/rds-js/releases)
[RDS SDK Documentation][docs] | [RDS API Documentation](https://covid19.richdataservices.com/rds/swagger/) | [Examples][examples] | [Contributing](CONTRIBUTING.md) | [Developer Documentation](DEVELOPER.md) | [Changelog](https://github.com/mtna/rds-js/releases)
|---|---|---|---|---|---|

## Quick start
Expand All @@ -31,80 +34,115 @@ Install the sdk into your project.
npm install @rds/sdk
```

#### Initialization

Import `RdsServer` and initialize to indicate where the RDS API is hosted. This must be done a single time before performing any queries.
### The setup

```typescript
import { RdsServer } from '@rds/sdk';
RdsServer.init('https://', 'covid19.richdataservices.com');
import { RdsServer, RdsCatalog, RdsDataProduct } from '@rds/sdk';

// Instantiate a new server to define where the RDS API is hosted.
const server = new RdsServer('https://covid19.richdataservices.com/rds');
// Instantiate a catalog that exists on the server
const catalog = new RdsCatalog(server, 'int');
// Instantiate a data product that exists on the catalog
const dataProduct = new RdsDataProduct(catalog, 'jhu_country');
```

### RDS Query Controller
These are the basic, foundational building blocks of the RDS SDK. From here, we can explore what catalogs/data products exist on the server, details about them, subset the data through various queries, and downloading customized data packages.

This service is used to query data products for both record level and aggregate data.
See the [documentation][docs] for the full SDK API.

#### Count
---

> Get the number of records in a dataset
### RdsServer

```typescript
import { HttpResponse, RdsQueryController } from '@rds/sdk';
Represents a single RDS API server, provides methods to query server-level information.

> Get the root catalog on the server
```ts
import { RdsServer } from '@rds/sdk';
const server = new RdsServer('https://covid19.richdataservices.com/rds');
server
.getRootCatalog()
.then(res=>
console.log(`There are ${res.parsedBody.catalogs.length} catalogs on this server!`)
);
```

const CATALOG_ID = 'covid19';
const DATA_PRODUCT_ID = 'us_jhu_ccse_country';
---

RdsQueryController
.count(CATALOG_ID, DATA_PRODUCT_ID)
.then((res: HttpResponse<number>) =>
{ console.log(`Found ${res.parsedBody} records!`); }
### RdsCatalog

Represents a single catalog on a server, provides methods to query catalog related information.

> Resolve properties about the catalog
```ts
import { RdsCatalog } from '@rds/sdk';
// Given a previously instantiated server, like in the examples above
const catalog = new RdsCatalog(server, 'int');
catalog
.resolve()
.then(()=>
catalog.name; // Name of catalog
catalog.description; // Catalog description
catalog.dataProducts; // All the data products on this catalog
// See the docs for all the possible properties
);
```

#### Select
---

> Running a select query on the specified data product returns record level microdata.
### RdsDataProduct

```typescript
import { AmchartsDataSet, HttpResponse, RdsQueryController, RdsSelectParameters } from '@rds/sdk';
Represents a single data product within a catalog, provides methods to query data product related information.

> Run a **select** query to get record level microdata.
const CATALOG_ID = 'covid19';
const DATA_PRODUCT_ID = 'us_jhu_ccse_country';
const PARAMS: RdsSelectParameters = {
```ts
import { AmchartsDataSet, HttpResponse, RdsDataProduct, RdsSelectParameters } from '@rds/sdk';

// Given the catalog from the above examples
const dataProduct = new RdsDataProduct(catalog, 'jhu_country');
// Specify some parameters
const params: RdsSelectParameters = {
cols: 'date_stamp,cnt_confirmed,cnt_death,cnt_recovered',
where: '(iso3166_1=US)',
metadata: true,
limit: 5000,
format: 'amcharts'
};

RdsQueryController
.select<AmchartsDataSet>(CATALOG_ID, DATA_PRODUCT_ID, PARAMS)
dataProduct
.select<AmchartsDataSet>(params)
.then((res: HttpResponse<AmchartsDataSet>) =>
{ /** Make a cool visualization */ }
{ /* Make a cool visualization */ }
);
```

#### Tabulate

> Running tabulations on the specified data product returns aggregate level data about the dimensions and measures specified.
> Run a **tabulation** to get aggregate level data about the dimensions and measures specified.
```typescript
import { AmchartsDataSet, HttpResponse, RdsQueryController, RdsTabulateParameters } from '@rds/sdk';

const CATALOG_ID = 'covid19';
const DATA_PRODUCT_ID = 'us_jhu_ccse_country';
const PARAMS: RdsTabulateParameters = {
dims: 'iso3166_1',
measure: 'cnt_confirmed:SUM(cnt_confirmed),cnt_death:SUM(cnt_death),cnt_recovered:SUM(cnt_recovered)',
orderby: 'cnt_confirmed DESC',
import { PlotlyDataSet, HttpResponse, RdsDataProduct, RdsTabulateParameters } from '@rds/sdk';

// Given the catalog from the above examples
const dataProduct = new RdsDataProduct(catalog, 'jhu_country');
// Specify some parameters
const params: RdsTabulateParameters = {
dims: 'date_stamp,iso3166_1',
measure: 'cnt_confirmed:SUM(cnt_confirmed)',
where: '(year_stamp=2020) AND (iso3166_1=US OR iso3166_1=CA OR iso3166_1=ES OR iso3166_1=IT OR iso3166_1=CN)',
orderby: 'date_stamp ASC,iso3166_1 ASC',
metadata: true,
limit: 10
inject: true,
totals: true,
format: 'plotly_heatmap'
};

RdsQueryController
.tabulate<AmchartsDataSet>(CATALOG_ID, DATA_PRODUCT_ID, PARAMS)
.then((res: HttpResponse<AmchartsDataSet>) =>
{ /** Make a cool visualization */ }
dataProduct
.tabulate<PlotlyDataSet>(params)
.then((res: HttpResponse<PlotlyDataSet>) =>
{ /* Make a cool visualization */ }
);
```

[docs]: https://mtna.github.io/rds-js/
[examples]: https://github.com/mtna/rds-js-examples
63 changes: 63 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 10 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@
"prepare": "npm run snyk-protect"
},
"dependencies": {
"@mtna/model-base-ui": "^9.0.6",
"@mtna/model-core-ui": "^10.0.0",
"@mtna/model-predefined-data-ui": "^1.1.6",
"@mtna/pojo-base-ui": "^9.0.2",
"@mtna/pojo-consumer-ui": "^1.1.9",
"@types/google.visualization": "0.0.52",
"snyk": "^1.321.0"
},
Expand All @@ -68,6 +73,7 @@
"husky": "^1.0.1",
"jest": "^26.0.1",
"jest-config": "^26.0.1",
"jest-fetch-mock": "^3.0.3",
"lint-staged": "^8.0.0",
"lodash.camelcase": "^4.3.0",
"prettier": "^1.14.3",
Expand Down Expand Up @@ -111,7 +117,7 @@
"transform": {
".(ts|tsx)": "ts-jest"
},
"testEnvironment": "node",
"testEnvironment": "jsdom",
"testRegex": "(/__tests__/.*|\\.(test|spec))\\.(ts|tsx|js)$",
"moduleFileExtensions": [
"ts",
Expand All @@ -132,6 +138,9 @@
},
"collectCoverageFrom": [
"src/*.{js,ts}"
],
"setupFiles": [
"./tools/setupJest.js"
]
},
"prettier": "@mtna/prettier-config-ui",
Expand Down
Binary file added resources/rds-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
72 changes: 72 additions & 0 deletions src/models/async/async-resource.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
import { ResolutionListener } from './resolution-listener';

/**
* An asyncronous resource that can be resolved.
*/
export abstract class AsyncResource {
/** Whether this resource is resolved */
private resolved = false;
/** Whether this resource is resolving */
private resolving = false;
/** Array of resolution listeners */
private resolutionListeners: ResolutionListener[] = [];

/**
* Create a new AsyncResource
*
* @param resolve whether to resolve the resource now, defaults to false
*/
constructor(resolve: boolean) {
if (resolve) {
this.resolve().catch(error => console.error('[@rds/sdk] AsyncResource: failed to resolve', error));
}
}

/** @returns whether this resource is resolved */
isResolved(): boolean {
return this.resolved;
}

/** @returns whether this resource is resolving */
isResolving(): boolean {
return this.resolving;
}

/**
* Register a listener that will emit when this resource has been resolved sucessfully.
* @param listener the listener for when this resource has been resolved
*/
registerResolutionListener(listener: ResolutionListener) {
this.resolutionListeners.push(listener);
}

/**
* Resolve this resource.
*
* @returns a promise that completes once the resource is resolved
*/
abstract async resolve(): Promise<void>;

/**
* Setter for `resolved`.
* @param resolved whether this resource is resolved
*/
protected setResolved(resolved: boolean) {
this.resolved = resolved;
// When the resource is resolved,
// notify any resolution listeners
if (this.resolved) {
while (this.resolutionListeners.length) {
this.resolutionListeners.pop()?.resolved();
}
}
}

/**
* Setter for `resolving`.
* @param resolving whether the resource is resolving
*/
protected setResolving(resolving: boolean) {
this.resolving = resolving;
}
}
2 changes: 2 additions & 0 deletions src/models/async/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
export * from './async-resource';
export * from './resolution-listener';
3 changes: 3 additions & 0 deletions src/models/async/resolution-listener.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export interface ResolutionListener {
resolved(): void;
}
Loading

0 comments on commit 35cc217

Please sign in to comment.