Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ New PUBSUB - NATS JetStream support #1002

Open
wants to merge 110 commits into
base: development
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
ba3e661
✨ initial
mfreeman451 Sep 8, 2024
df7da72
🤡 added mockgen generated tests
mfreeman451 Sep 8, 2024
b3b78fe
✅ adding health test
mfreeman451 Sep 8, 2024
4b93955
✅ adding message test
mfreeman451 Sep 8, 2024
7991516
🤡 added mocked jetstream
mfreeman451 Sep 8, 2024
9ebd1f7
🚧 WIP natsclient test
mfreeman451 Sep 9, 2024
ba51980
✅ updated nats test
mfreeman451 Sep 10, 2024
bc613e2
🤡 added mocked nats connection
mfreeman451 Sep 12, 2024
f8f97c7
Merge branch 'development' into new/nats_jetstream_support
mfreeman451 Sep 12, 2024
a227f93
🚨 fixing linter issues
mfreeman451 Sep 13, 2024
2be4273
🐛 fixing kafka thing
mfreeman451 Sep 13, 2024
5ed9e4e
🔧 renaming NATSClient
mfreeman451 Sep 13, 2024
52a77be
✨ adding some missing files
mfreeman451 Sep 13, 2024
2e0feac
✅ updated tests
mfreeman451 Sep 13, 2024
da70ca5
🚨 fixing linter issues
mfreeman451 Sep 13, 2024
eb47ade
✨ adding go.mod
mfreeman451 Sep 13, 2024
ab6d892
📝 updated docs for NATS
mfreeman451 Sep 13, 2024
fb40358
Merge branch 'development' into new/nats_jetstream_support
mfreeman451 Sep 13, 2024
d09e6c6
👷‍♂️ WIP
mfreeman451 Sep 14, 2024
cad63b0
🚧 WIP
mfreeman451 Sep 14, 2024
96e38b6
🚧 updating tests
mfreeman451 Sep 14, 2024
097dcdd
🚧 fixing tests
mfreeman451 Sep 14, 2024
f3e678b
✅ fixd subscribesuccess stuff
mfreeman451 Sep 14, 2024
a18bfbb
🚧 updating health tests
mfreeman451 Sep 14, 2024
fe1ce59
✅ updated tests
mfreeman451 Sep 14, 2024
c7ecc20
🚧 WIP example
mfreeman451 Sep 14, 2024
0f781f9
🔨 working out import cycle issues
mfreeman451 Sep 15, 2024
691654d
✅ updated tests
mfreeman451 Sep 15, 2024
4726aa8
🚧 WIP nats test updates
mfreeman451 Sep 15, 2024
3ed1b35
🚧 WIP updated validateconfigs test
mfreeman451 Sep 15, 2024
a4399c3
✅ updated tests
mfreeman451 Sep 15, 2024
c420d3f
♻️ cleanup
mfreeman451 Sep 15, 2024
c98d5bc
♻️ cleanup
mfreeman451 Sep 15, 2024
926c574
🚨 fixing linter issues
mfreeman451 Sep 15, 2024
7cfd16e
♻️ refactored nats.go file too big
mfreeman451 Sep 15, 2024
70151c3
✅ updated tests
mfreeman451 Sep 16, 2024
4db53ab
🎉 example working
mfreeman451 Sep 16, 2024
9ffb056
🚨 fixing linter issues
mfreeman451 Sep 16, 2024
7605698
✅ updated tests
mfreeman451 Sep 16, 2024
7c0767f
🚨 fixing linter issues
mfreeman451 Sep 16, 2024
8e53572
🚨 fixing code quality issues
mfreeman451 Sep 16, 2024
c8f164e
🚨 fixing code quality issues
mfreeman451 Sep 16, 2024
8f16da2
📝 missing comment
mfreeman451 Sep 16, 2024
087b6da
🔨 cleaning up commit
mfreeman451 Sep 16, 2024
f3a4959
📝 updating comments
mfreeman451 Sep 16, 2024
79e8c45
📝 updating comments
mfreeman451 Sep 16, 2024
9763e32
✅ adding test
mfreeman451 Sep 16, 2024
8690d7f
🚨 fixing linter errors
mfreeman451 Sep 16, 2024
54a224d
🐛 fixed commiter create func
mfreeman451 Sep 16, 2024
7ab1e2d
✅ adding test
mfreeman451 Sep 16, 2024
909b201
🚧 WIP tests
mfreeman451 Sep 16, 2024
a64b762
🚧 WIP tests
mfreeman451 Sep 16, 2024
044e92a
🔧 working on test coverage
mfreeman451 Sep 16, 2024
c74e82c
🚨 fixing linter errors
mfreeman451 Sep 17, 2024
79faff3
🚨 fixing linter errors
mfreeman451 Sep 17, 2024
5b3869d
⚰️ removing unused code
mfreeman451 Sep 17, 2024
ed9c04f
♻️ cleanup and remove nats_mocks
mfreeman451 Sep 17, 2024
d68efff
✨ added support for nats creds file
mfreeman451 Sep 17, 2024
7ee49ea
🚨 fixing linter errors
mfreeman451 Sep 17, 2024
ab6925b
📝 adding comments
mfreeman451 Sep 17, 2024
f2340b7
🐛 accidently removed a log statement
mfreeman451 Sep 17, 2024
3448c74
Merge branch 'development' into new/nats_jetstream_support
mfreeman451 Sep 17, 2024
ce997b3
🌱 missing some files from git
mfreeman451 Sep 17, 2024
ff50121
Merge branch 'development' into new/nats_jetstream_support
Umang01-hash Sep 17, 2024
b38fe25
🔧 adding back comment
mfreeman451 Sep 20, 2024
f91ba16
🔧 updates
mfreeman451 Sep 20, 2024
980b761
✅ updating tests
mfreeman451 Sep 20, 2024
131ac92
🔧 updated new and tests
mfreeman451 Sep 20, 2024
118eb0b
🔧 unexported processMessages
mfreeman451 Sep 20, 2024
e178433
🔧 update log message
mfreeman451 Sep 20, 2024
1cbe7c2
💄 updated constants to camelcase
mfreeman451 Sep 20, 2024
9e65dcb
🔧 updated func
mfreeman451 Sep 20, 2024
3ec754a
♻️ cleanup
mfreeman451 Sep 20, 2024
3606942
Merge branch 'development' into new/nats_jetstream_support
mfreeman451 Sep 20, 2024
0cf8bc7
🔧 update
mfreeman451 Sep 20, 2024
7c645e5
🔧 sync
mfreeman451 Sep 20, 2024
40fc408
🔧 remove alias
mfreeman451 Sep 20, 2024
ed07e4a
🔧 cleanup
mfreeman451 Sep 20, 2024
b12dede
🔧 adding temporary replace directice to pass tests
mfreeman451 Sep 20, 2024
b082486
Merge branch 'development' into new/nats_jetstream_support
mfreeman451 Sep 23, 2024
34a1d9e
🔧 fixing import shadow and subscriber test
mfreeman451 Sep 23, 2024
b430815
🔧 updating kafka test
mfreeman451 Sep 24, 2024
a435cea
Merge branch 'development' into new/nats_jetstream_support
Umang01-hash Sep 24, 2024
09cda43
🔧 fixed import order
mfreeman451 Sep 25, 2024
5fc2fa6
🔧 reverting back to datasource
mfreeman451 Sep 25, 2024
f76fb90
🔧 updates
mfreeman451 Sep 25, 2024
250ffd5
🔧 updated errors pkg
mfreeman451 Sep 25, 2024
baf660e
🔧 fixed test, adding missing file
mfreeman451 Sep 25, 2024
272857d
🔧 sync
mfreeman451 Sep 25, 2024
c1cc795
🔧 fixing kafka test
mfreeman451 Sep 25, 2024
cebc607
🔧 cleanup mqtt
mfreeman451 Sep 25, 2024
34ecce8
🔧 cleanup mqtt test
mfreeman451 Sep 25, 2024
96eb2c5
🔧 unexporting stuff
mfreeman451 Sep 25, 2024
a3fbc8c
🔧 cleanup
mfreeman451 Sep 25, 2024
b14b9fd
🔧 updated subscriber_test
mfreeman451 Sep 25, 2024
3e5f8df
🔧 goimport cleanup
mfreeman451 Sep 25, 2024
ee3287b
🐛 fixing export of client interface
mfreeman451 Sep 25, 2024
8a34dd5
🔧 fixing go mod
mfreeman451 Sep 25, 2024
8340347
Merge branch 'development' into new/nats_jetstream_support
mfreeman451 Sep 26, 2024
0427c8f
Merge branch 'development' into new/nats_jetstream_support
mfreeman451 Sep 30, 2024
f2b36db
Revert container.go to dev branch version
mfreeman451 Sep 30, 2024
632262f
✨ cleanup and docs update
mfreeman451 Sep 30, 2024
a4b941d
🔧 container.go update
mfreeman451 Sep 30, 2024
10ab8a5
📝 typo
mfreeman451 Sep 30, 2024
95189e8
🔧 go.mod cleanup
mfreeman451 Sep 30, 2024
3be3dab
🔧 go.mod cleanup
mfreeman451 Sep 30, 2024
23ce87f
📝 fixing comments
mfreeman451 Sep 30, 2024
789d74f
fix dependencies
Umang01-hash Oct 1, 2024
ad7259e
Merge branch 'development' into new/nats_jetstream_support
mfreeman451 Oct 1, 2024
eae9558
🔧 sync
mfreeman451 Oct 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 77 additions & 2 deletions docs/advanced-guide/using-publisher-subscriber/page.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ scaled and maintained according to its own requirement.

## Design choice

In GoFr application if a user wants to use the Publisher-Subscriber design, it supports two message brokers—Apache Kafka
and Google PubSub.
In GoFr application if a user wants to use the Publisher-Subscriber design, it supports several message brokers,
including Apache Kafka, Google PubSub, MQTT, and NATS JetStream.
The initialization of the PubSub is done in an IoC container which handles the PubSub client dependency.
With this, the control lies with the framework and thus promotes modularity, testability, and re-usability.
Users can do publish and subscribe to multiple topics in a single application, by providing the topic name.
Expand Down Expand Up @@ -175,6 +175,81 @@ docker run -d \
eclipse-mosquitto:latest
```
> **Note**: find the default mosquitto config file {% new-tab-link title="here" href="https://github.com/eclipse/mosquitto/blob/master/mosquitto.conf" /%}

### NATS JetStream

NATS JetStream is supported as an external pubsub provider, meaning if you're not using it, it won't be added to your binary.

#### Configs
```dotenv
PUBSUB_BACKEND=NATS
PUBSUB_BROKER=nats://localhost:4222
NATS_STREAM=mystream
NATS_SUBJECTS=orders.*,shipments.*
NATS_MAX_WAIT=5s
NATS_BATCH_SIZE=100
NATS_MAX_PULL_WAIT=500ms
NATS_CONSUMER=my-consumer
NATS_CREDS_FILE=/path/to/creds.json
```

#### Setup

To set up NATS JetStream, follow these steps:

1. Import the external driver for NATS JetStream:

```bash
go get gofr.dev/pkg/gofr/datasources/pubsub/nats
```

2. Use the `AddPubSub` method to add the NATS JetStream driver to your application:

```go
app := gofr.New()

app.AddPubSub(nats.New(nats.Config{
Server: "nats://localhost:4222",
Stream: nats.StreamConfig{
Stream: "mystream",
Subjects: []string{"orders.*", "shipments.*"},
},
MaxWait: 5 * time.Second,
BatchSize: 100,
MaxPullWait: 500 * time.Millisecond,
Consumer: "my-consumer",
CredsFile: "/path/to/creds.json",
}))
```

#### Docker setup
```shell
docker run -d \
--name nats \
-p 4222:4222 \
-p 8222:8222 \
-v <path-to>/nats.conf:/nats/config/nats.conf \
nats:2.9.16
```

#### Configuration Options

| Name | Description | Required | Default | Example |
|------|-------------|----------|---------|---------|
| `PUBSUB_BACKEND` | Set to "NATS" to use NATS JetStream as the message broker | Yes | - | `NATS` |
| `PUBSUB_BROKER` | NATS server URL | Yes | - | `nats://localhost:4222` |
| `NATS_STREAM` | Name of the NATS stream | Yes | - | `mystream` |
| `NATS_SUBJECTS` | Comma-separated list of subjects to subscribe to | Yes | - | `orders.*,shipments.*` |
| `NATS_MAX_WAIT` | Maximum wait time for batch requests | No | - | `5s` |
| `NATS_BATCH_SIZE` | Maximum number of messages to pull in a single request | No | 0 | `100` |
| `NATS_MAX_PULL_WAIT` | Maximum wait time for individual pull requests | No | 0 | `500ms` |
| `NATS_CONSUMER` | Name of the NATS consumer | No | - | `my-consumer` |
| `NATS_CREDS_FILE` | Path to the credentials file for authentication | No | - | `/path/to/creds.json` |

#### Usage

When subscribing or publishing using NATS JetStream, make sure to use the appropriate subject name that matches your stream configuration.
For more information on setting up and using NATS JetStream, refer to the official NATS documentation.

### Azure Eventhub
GoFr supports eventhub starting gofr version v1.22.0.
Expand Down
24 changes: 23 additions & 1 deletion docs/references/configs/page.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ This document lists all the configuration options supported by the GoFr framewor

- PUBSUB_BACKEND
- Pub/Sub message broker backend
- kafka, google, mqtt
- kafka, google, mqtt, nats

{% /table %}

Expand Down Expand Up @@ -338,3 +338,25 @@ This document lists all the configuration options supported by the GoFr framewor
- Sends regular messages to check the link is active. May not work as expected if handling func is blocking execution

{% /table %}

**NATS JetStream**

{% table %}

- Name
- Description
- Default Value

---

- NATS_SERVER
- URL of the NATS server
- nats://localhost:4222

---

- NATS_CREDS_FILE
- File containing the NATS credentials
- creds.json

{% /table %}
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module gofr.dev

go 1.22
go 1.22.3

require (
cloud.google.com/go/pubsub v1.42.0
Expand Down
1 change: 0 additions & 1 deletion pkg/gofr/container/mock_container.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ import (
"testing"

"go.uber.org/mock/gomock"

"gofr.dev/pkg/gofr/datasource"
"gofr.dev/pkg/gofr/datasource/file"
"gofr.dev/pkg/gofr/datasource/pubsub"
Expand Down
4 changes: 3 additions & 1 deletion pkg/gofr/datasource/file/ftp/go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
module gofr.dev/pkg/gofr/datasource/file/ftp

go 1.22
go 1.22.3

toolchain go1.23.1

replace gofr.dev => ../../../../../../gofr

Expand Down
6 changes: 4 additions & 2 deletions pkg/gofr/datasource/file/sftp/go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
module gofr.dev/pkg/gofr/datasource/file/sftp

go 1.22
go 1.22.3

toolchain go1.23.1

replace gofr.dev => ../../../../../../gofr

Expand All @@ -20,4 +22,4 @@ require (
github.com/pmezard/go-difflib v1.0.0 // indirect
golang.org/x/sys v0.25.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
)
Loading