Turborepo Shared typed kafka client

This is an example to show how to share one kafka client with different backend services. Imagine you have a kafka cluster and two backend services(backend1, backend2) want to producer and consume the messages from the cluster. You dont want to repeat the same codes for kafka implementation in two backend 1 and 2 codebases. Then a kafka-client package comes in handy since you can import it and use it directly. One problem is how can we make it type safe and easy to use. I try to write the types that can autocomplete.

Highlight

const kafkaClient = new KafkaClient("backend1", []); // you get the autocomplete for which services are "backend1" | "backend2"

kafkaClient.sendMessage("payment-request", {}); // auto suggest which topic to choose and get the correct message type to pass in

kafkaClient.startConsumer(["user-request"], (message, topic) => {}); // auto suggest what topics can be subscribed

Limiations and Improvements

How I make the sendMessage and startConsumer to get the autocomplete working. I have a predefined topics in the kafka-client

type TopicMessage = {
  "user-request": UserRequestMessage;
  "payment-request": PaymentRequestMessage;
};
type Topic = keyof TopicMessage;
// we can get the message type like this
// T is generic
T extends Topic
TopicMessage[T]

And I do some not elegant conditional check to make it work

type Message<T extends Topic> = T extends "user-request"
  ? UserRequestMessage
  : T extends "payment-request"
  ? PaymentRequestMessage
  : never;

One down side is if I want to add more topics, I need to do the check for each topic to get the correct message types I am still learning typescript. I would like to get some help if everyone happen to view this repo. Thanks in advance

Using this example

Run the following command:

yarn
yarn dev

What's inside?

This Turborepo includes the following packages/apps:

Apps and Packages

• backend1: a nodejs app
• backend2: another nodejs app
• kafka-client: a kakfka client package used by backend1 and backend2
• eslint-config-custom: eslint configurations (includes eslint-config-next and eslint-config-prettier)
• tsconfig: tsconfig.jsons used throughout the monorepo

Each package/app is 100% TypeScript.

Utilities

This Turborepo has some additional tools already setup for you:

• TypeScript for static type checking
• ESLint for code linting
• Prettier for code formatting

Build

To build all apps and packages, run the following command:

yarn build

Develop

To develop all apps and packages, run the following command:

yarn dev

Remote Caching

Turborepo can use a technique known as Remote Caching to share cache artifacts across machines, enabling you to share build caches with your team and CI/CD pipelines.

By default, Turborepo will cache locally. To enable Remote Caching you will need an account with Vercel. If you don't have an account you can create one, then enter the following commands:

cd my-turborepo
npx turbo login

This will authenticate the Turborepo CLI with your Vercel account.

Next, you can link your Turborepo to your Remote Cache by running the following command from the root of your Turborepo:

npx turbo link

Useful Links

Learn more about the power of Turborepo:

• Tasks
• Caching
• Remote Caching
• Filtering
• Configuration Options
• CLI Usage