Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement data store service to fetch entity data from HTTP or file system #156

Merged
merged 2 commits into from
Mar 7, 2024

Conversation

wcalderipe
Copy link
Collaborator

@wcalderipe wcalderipe commented Mar 6, 2024

This PR introduces the data store service responsible for fetching data from various stores based on the tenant's configuration.

image

Image 1: part of the diagram done in this PR.

The goals were to allow data to be stored in any store you'd like, plus simple extensibility for new protocols.

image

Image 2: data store design.

This PR doesn't solve any specific configuration per data store. For example, authentication will be likely different from repository to repository (e.g. basic auth for HTTP).

Note on key-value storage model

The simplicity of key-value storage comes with a few trade-offs:

  • Schema validation is handled at the application layer. These need to be manually checked before writing and after reading.
  • Schemas don't support versioning. Changing a schema will introduce a breaking change to any engine with historical data using the old schema.
  • The key-value model is not designed for use cases that require listing data (e.g., finding all the tenants). To address this, we need to maintain an index of tenants in a separate key managed by the application. This approach is error-prone and won't scale (though scalability isn't an issue right now).

Note on Dependency Inversion Principle (DIP)

The design of the data store service follows the DIP, making it the most hexagonal/clean architecture implementation in our codebase today.

A quick refresher on DIP: high-level modules should not depend on low-level modules. Both should depend on abstractions.

Applied to our structure, this means the core should not depend on anything outside the core. You can see an example of this in core/repository/data-store.repository.ts, which is just an interface, but in other places where we just need one repository the interface step is ditched.

By now, you might be wondering why we don't apply this approach everywhere. The reason is simplicity. It's tedious and difficult to abstract something when you have only a single sample. More importantly, code navigation becomes challenging.

Regarding code navigation, if your services only depend on interfaces, attempting to navigate to the implementation of an interface will lead you to the interface itself—which is correct but probably not what you want. You're likely interested in the actual implementation at runtime. However, to find the actual implementation, you'll have to figure out how the runtime injects the right implementation in the abstraction.

@wcalderipe wcalderipe self-assigned this Mar 6, 2024
@wcalderipe wcalderipe changed the title Implement data store service for the file system and HTTP Implement data store service to fetch entity data from HTTP or file system Mar 6, 2024
@wcalderipe wcalderipe merged commit 8998310 into main Mar 7, 2024
3 checks passed
@wcalderipe wcalderipe deleted the feature/data-source-fetch branch May 27, 2024 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant