Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: ATL-6983 ZIO Stream Kafka PoC in background jobs #1339

Merged
merged 89 commits into from
Oct 9, 2024
Merged

feat: ATL-6983 ZIO Stream Kafka PoC in background jobs #1339

merged 89 commits into from
Oct 9, 2024

Conversation

bvoiturier
Copy link
Contributor

Description:

ATL-6983

bvoiturier and others added 30 commits August 6, 2024 10:16
@bvoiturier @mineme0110
feat: introduce draft MessagingService interface and Kafka impl
47488f4 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: add zio-kafka implementation
6f85bbb 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: allow passing Kafka bootstrapServers and groupId as parameters
2838aa2 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
chore: exclude ZIO 2.1.2 transitive dependency from zio-kafka as it m…
f3fb835 
…akes compilation fail

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: allow passing an 'R' to the consumer handler method
0c03775 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: add kafka and zookeeper service to shared docker compose config
55cde88 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: early draft (but functional) implementation of messaging servic…
f1bf42b 
…e as connect bg replacement

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: manually create topics with custom partitions number
e8d7146 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: add layer for Kafka consumer and producer
3939e3c 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: rather use producer singleton in connection service
730f185 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: fork multiple consumers as replacement for connect bg job polli…
cad7d89 
…ng mechanism

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
chore: comment out obsolete code and log when invalid records are rec…
46b3ef3 
…eived for processing

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
chore: explicitly set important consumer settings parameters
a80dd8e 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: create a 'connect-retry' Kafka topic
ba858ef 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: introduce ByteArrayWrapper to eliminate dotty compilation error…
8965c84 
… + add timestamp to Message case class

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: implement retry mechanism using dedicated connect-retry topic
da681fc 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
chore: use const
be10565 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier @mineme0110
feat: implement configurable consumer retry strategy based on topic s…
c2c5507 
…equence (with DLQ)

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@mineme0110
WIP Imemorykafka
374477e 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@mineme0110
In memory kafka messaging service
47c3a6b 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@mineme0110
created a seperate docker-compose for kafka
9d9fbce 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@mineme0110
created a seperate docker-compose for kafka
0def74a 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@mineme0110
Add config for kafka inmemrory enable
074363b 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@mineme0110
Issue background jobs to use messaging service
e6a0014 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@mineme0110
Issue background jobs to use messaging service
313bae6 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@mineme0110
Moved present background jobs to use the kafka inmemory
b9ff594 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@bvoiturier @mineme0110
chore(cloud-agent): minor fixes and clean-up
d08d8d7 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@mineme0110
Moved sync background jobs to use the kafka inmemory
4a815e1 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@mineme0110
Moved sync background jobs to use the kafka inmemory
93bfc42 
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
@bvoiturier
chore(cloud-agent): restructure status list backgound job code
9037bd0 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
chore: increase scalac max-inlines to 50 to fix DeriveConfig macro co…
d4b17a2 
…mpilation error

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
Merge branch 'main' into feat/ATL-6983-zio-stream-kafka-poc-in-connec…
f8cd974 
…t-bg-jobs
@bvoiturier
chore: run scalafmt
73a79f7 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
feat: solve race condition issue with VC status list index allocation
b35cbff 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
Merge branch 'main' into feat/ATL-6983-zio-stream-kafka-poc-in-connec…
7f5d2c5 
…t-bg-jobs

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
fix: for kafka retry strategy generation from config
4f3663c 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
chore: clean-up obsolete background jobs related config params
f933e25 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
chore: refactor DID state background job logic
a8b49a5 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
chore: set status list & DID state sync recurrence delay to 5 sec in …
29601ac 
…integration tests

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
Merge branch 'main' into feat/ATL-6983-zio-stream-kafka-poc-in-connec…
e182254 
…t-bg-jobs
@bvoiturier
feat: use unique adivosry lock id for the same wallet Id
d6a89fb 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
chore: log at debug level
93e0dab 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
Merge branch 'main' into feat/ATL-6983-zio-stream-kafka-poc-in-connec…
29bd636 
…t-bg-jobs

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
chore: add recordId when dying with "record not found"
0fc30ec 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
feat: support in-memory queue capacity configuration
f1132a7 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
chore: refactor messaging service config and migrate to messaging ser…
e4c4260 
…vice package

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
chore: refactor/simplify MessagingService-related layers creation
bb5ea8f 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier
chore: run scalafmt
af1ace6 
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier bvoiturier marked this pull request as ready for review October 8, 2024 15:37
@bvoiturier bvoiturier requested a review from a team as a code owner October 8, 2024 15:37
@yshyn-iohk yshyn-iohk enabled auto-merge (squash) October 9, 2024 07:35
@yshyn-iohk yshyn-iohk merged commit 19ab426 into main Oct 9, 2024
13 checks passed
@yshyn-iohk yshyn-iohk deleted the feat/ATL-6983-zio-stream-kafka-poc-in-connect-bg-jobs branch October 9, 2024 07:49
@bvoiturier bvoiturier mentioned this pull request Oct 22, 2024
Open
bytewizard42i pushed a commit to bytewizard42i/identus-cloud-agent that referenced this pull request Nov 13, 2024
@hyperledger-bot
chore(release): cut the Identus Cloud agent 1.40.0 release
94e7ec0 
# [1.40.0](hyperledger/identus-cloud-agent@cloud-agent-v1.39.0...cloud-agent-v1.40.0) (2024-11-05)

### Bug Fixes

* Add key_id missing field ([hyperledger#1403](hyperledger#1403)) ([cbd1a03](hyperledger@cbd1a03))
* adjust Kotlin and TypeScript HTTP client to use the `schemaId` f… ([hyperledger#1388](hyperledger#1388)) ([c2da492](hyperledger@c2da492))
* cannot reuse the same credential-offer in oid4vci ([hyperledger#1361](hyperledger#1361)) ([6a0a3ea](hyperledger@6a0a3ea))
* handle unsupported PIURI found in DIDComm messages accordingly ([hyperledger#1399](hyperledger#1399)) ([9b64793](hyperledger@9b64793))
* key id for jwt and sdjwt ([hyperledger#1420](hyperledger#1420)) ([5830a7e](hyperledger@5830a7e))
* oas to use any schema for json ast node ([hyperledger#1372](hyperledger#1372)) ([95d328e](hyperledger@95d328e))
* oid4vci endpoints error statuses and negative input validation ([hyperledger#1384](hyperledger#1384)) ([65cc9a7](hyperledger@65cc9a7))
* Preserve Presentation Format ([hyperledger#1363](hyperledger#1363)) ([c18385c](hyperledger@c18385c))
* return 404 when create credConfig on non-existing issuer ([hyperledger#1379](hyperledger#1379)) ([e532ba6](hyperledger@e532ba6))

### Features

* Add KID to the credential-offers API - ATL-7704 ([hyperledger#1320](hyperledger#1320)) ([56200cf](hyperledger@56200cf))
* add presentation-exchange endpoints ([hyperledger#1365](hyperledger#1365)) ([49f7ab3](hyperledger@49f7ab3))
* ATL-6983 ZIO Stream Kafka PoC in background jobs ([hyperledger#1339](hyperledger#1339)) ([19ab426](hyperledger@19ab426))
* Default Backend API to Array Of Credential Schema ([hyperledger#1366](hyperledger#1366)) ([693dcc4](hyperledger@693dcc4))
* Default Object As Issuer ([hyperledger#1349](hyperledger#1349)) ([d29eebb](hyperledger@d29eebb))
* Implement prism anoncreds method for schemas and credential definitions ([hyperledger#1385](hyperledger#1385)) ([fbee055](hyperledger@fbee055))
* Issuer Replace Either By Union Type ([hyperledger#1374](hyperledger#1374)) ([8fc2fe3](hyperledger@8fc2fe3))
* presentation_submission validation logic ([hyperledger#1332](hyperledger#1332)) ([f80b3c3](hyperledger@f80b3c3))
* Support Array Of Credential Schema ([hyperledger#1351](hyperledger#1351)) ([948e314](hyperledger@948e314))
* Test JWT OBJECT as Issuer ([hyperledger#1343](hyperledger#1343)) ([7208d95](hyperledger@7208d95))
* VC support for Array of credential Status ([hyperledger#1383](hyperledger#1383)) ([ad946cf](hyperledger@ad946cf))
* VCVerification API support ARRAY or OBJECT as Credential Sc… ([hyperledger#1355](hyperledger#1355)) ([91cb4e7](hyperledger@91cb4e7))

 [skip ci]

Signed-off-by: Hyperledger Bot <hyperledger-bot@hyperledger.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yshyn-iohk yshyn-iohk yshyn-iohk approved these changes

@FabioPinheiro FabioPinheiro Awaiting requested review from FabioPinheiro

@mineme0110 mineme0110 Awaiting requested review from mineme0110

@amagyar-iohk amagyar-iohk Awaiting requested review from amagyar-iohk

Assignees
No one assigned
Labels
build cloud-agent connect infra pollux shared
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@bvoiturier @coveralls @yshyn-iohk @mineme0110