Skip to content

Commit

Permalink
Remove unnecessary files and updated README
Browse files Browse the repository at this point in the history
  • Loading branch information
mostafa committed Nov 19, 2020
1 parent 963caa0 commit 53be418
Show file tree
Hide file tree
Showing 4 changed files with 31 additions and 290 deletions.
222 changes: 0 additions & 222 deletions Gopkg.lock

This file was deleted.

15 changes: 0 additions & 15 deletions Gopkg.toml

This file was deleted.

74 changes: 31 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,50 +1,38 @@
# Disclaimer
# xk6-kafka

The [k6](https://github.com/loadimpact/k6) [plugin system](https://github.com/loadimpact/k6/issues/1353) is currently experimental. This plugin is a proof of concept, and it isn't supported by the k6 team, and may break in the future. USE IT AT YOUR OWN RISK!
This is a [k6](https://github.com/loadimpact/k6) extension using the [xk6](https://github.com/loadimpact/k6) system.

This project is also a WIP, so it is not feature-complete, nor something to rely on. Over time, I'll try to add a better API that is more natural to both Go and JavaScript.
| :exclamation: This is a proof of concept, isn't supported by the k6 team, and may break in the future. USE AT YOUR OWN RISK! |
| ---------------------------------------------------------------------------------------------------------------------------- |

---
This project is a k6 extension that can be used to load test Kafka, using a producer. Per each connection to Kafka, many messages can be sent. These messages are an array of objects containing a key and a value. There is also a consumer for testing purposes, i.e. to make sure you send the correct data to Kafka. The consumer is not meant to be used for testing Kafka under load. The extension supports producing and consuming messages in Avro format, given a schema for key and/or value.

# k6-plugin-kafka
The real purpose of this extension is not only to test Apache Kafka, but also the system you've designed that uses Apache Kafka. So, you can test your consumers, and hence your system, by auto-generating messages and sending them to your system via Apache Kafka.

This project is a k6 plugin that can be used to load test Kafka, using a producer. Per each connection to Kafka, many messages can be sent, which is basically an array of objects containing a key and a value. There's also a consumer for testing purposes, e.g. to make sure you send the correct data to Kafka. The consumer is not meant to be used for testing Kafka under load. The plugin supports producing and consuming messages in Avro format, given a schema for key and/or value.
In order to build the source, you should have the latest version of Go (go1.15) installed. I recommend you to have [gvm](https://github.com/moovweb/gvm) installed.

The real purpose of this plugin is not only to test Apache Kafka, but also the system you've designed that uses Apache Kafka. So, you can test your consumers, and hence your system, by auto-generating messages and sending them to your system via Apache Kafka.
## Build

In order to build the source, you should have the latest version of Go installed, which I recommend you to have [gvm](https://github.com/moovweb/gvm), Go version manager, installed.
To build a `k6` binary with this extension, first ensure you have the prerequisites:

<!--
## Changelog
- [gvm](https://github.com/moovweb/gvm)
- [Git](https://git-scm.com/)

* v0.0.1
- [feat] Added a slightly better API to work with the plugin
-->
Then, install [xk6](https://github.com/k6io/xk6) and build your custom k6 binary with the Kafka extension:

## Build k6 from source (with plugin support)
1. Install `xk6`:
```shell
$ go get -u github.com/k6io/xk6/cmd/xk6
```

This step will be removed once [the plugin support PR](https://github.com/loadimpact/k6/pull/1396) is merged and in production.

```bash
$ go get -d github.com/loadimpact/k6
$ cd $GOPATH/src/github.com/loadimpact/k6
$ git checkout -b andremedeiros-feature/plugins tags/v0.26.2
$ git pull -f https://github.com/andremedeiros/k6.git feature/plugins
$ make
```

## Build plugin from source

```bash
$ go get -d github.com/mostafa/k6-plugin-kafka
$ cd $GOPATH/src/github.com/mostafa/k6-plugin-kafka
$ ./build.sh
$ cp $GOPATH/src/github.com/loadimpact/k6/k6 $GOPATH/src/github.com/mostafa/k6-plugin-kafka
```
2. Build the binary:
```shell
$ xk6 build v0.29.0 --with github.com/mostafa/xk6-kafka
```

## Run & Test

First, you need to have your Kafka development environment setup. I recommend you to use [Lenses.io fast-data-dev Docker image](https://github.com/lensesio/fast-data-dev), which is a complete Kafka setup for development that includes: Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors. It is fairly easy to setup, if you have Docker installed. Just make sure to monitor Docker logs to have a working setup, before attempting to test.
First, you need to have your Kafka development environment setup. I recommend you to use [Lenses.io fast-data-dev Docker image](https://github.com/lensesio/fast-data-dev), which is a complete Kafka setup for development that includes: Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors. It is fairly easy to setup, if you have Docker installed. Just make sure to monitor Docker logs to have a working setup, before attempting to test. Initial setup, leader election and test data ingestion takes time.

### Development Environment

Expand All @@ -57,24 +45,24 @@ $ sudo docker logs -f lensesio

### k6 Test

The following k6 test script is used to test this plugin and Apache Kafka in turn. The script is availale as `test.js` with more code and commented sections. The script has 4 parts:
The following k6 test script is used to test this extension and Apache Kafka in turn. The script is available as `test.js` with more code and commented sections. The script has 4 parts:

1. The __imports__ at the top shows the exposed functions that are imported from k6 and the plugin, `check` from k6 and the `writer`, `produce`, `reader`, `consume` from the plugin using the `k6-plugin/kafka` plugin loading convention.
1. The __imports__ at the top shows the exposed functions that are imported from k6 and the extension, `check` from k6 and the `writer`, `produce`, `reader`, `consume` from the extension using the `k6/x/kafka` extension loading convention.
2. The __Avro schema__ defines a value schema that is used by both producer and consumer, according to the [Avro schema specification](https://avro.apache.org/docs/current/spec.html).
3. The __Avro message producer__:
1. The `writer` function is used to open a connection to the bootstrap servers. The first arguments is an array of string that signifies the bootstrap server addresses and the second is the topic you want to write to. You can reuse this writer object to produce as many messages as you want.
1. The `writer` function is used to open a connection to the bootstrap servers. The first argument is an array of strings that signifies the bootstrap server addresses and the second is the topic you want to write to. You can reuse this writer object to produce as many messages as you want.
2. The `produce` function is used to send a list of messages to Kafka. The first argument is the `producer` object, the second is the list of messages (with key and value), the third and the fourth are the key schema and value schema in Avro format. If the schema are not passed to the function, the values are treated as normal strings, as in the key schema, where an empty string, `""`, is passed.
The produce function returns an `error` if it fails. The check is optional, but `error` being `undefined` means that `produce` function successfully sent the message.
3. The `producer.close()` function closes the `producer` object.
4. The __Avro message consumer__:
1. The `reader` function is used to open a connection to the bootstrap servers. The first arguments is an array of string that signifies the bootstrap server addresses and the second is the topic you want to reader from.
1. The `reader` function is used to open a connection to the bootstrap servers. The first argument is an array of strings that signifies the bootstrap server addresses and the second is the topic you want to reader from.
2. The `consume` function is used to read a list of messages from Kafka. The first argument is the `consumer` object, the second is the number of messages to read in one go, the third and the fourth are the key schema and value schema in Avro format. If the schema are not passed to the function, the values are treated as normal strings, as in the key schema, where an empty string, `""`, is passed.
The consume function returns an empty array if it fails. The check is optional, but it checks to see if the length of the message array is exactly 10.
3. The `consumer.close()` function closes the `consumer` object.

```javascript
import { check } from 'k6';
import { writer, produce, reader, consume } from 'k6-plugin/kafka'; // import kafka plugin
import { writer, produce, reader, consume } from 'k6/x/kafka'; // import kafka extension

// Avro value schema
const value_schema = JSON.stringify({
Expand All @@ -92,18 +80,18 @@ export default function () {
// Avro message producer
const producer = writer(
["localhost:9092"], // bootstrap servers
"test-k6-plugin-topic", // Kafka topic
"test-k6-extension-topic", // Kafka topic
)

for (let index = 0; index < 100; index++) {
let error = produce(producer,
[{
key: "DA KEY!",
value: JSON.stringify({
"name": "k6-plugin-kafka",
"name": "k6-extension-kafka",
"version": "0.0.1",
"author": "Mostafa Moradian",
"description": "k6 Plugin to Load Test Apache Kafka"
"description": "k6 Extension to Load Test Apache Kafka"
})
}], "", value_schema);

Expand All @@ -116,7 +104,7 @@ export default function () {
// Avro message consumer
const consumer = reader(
["localhost:9092"], // bootstrap servers
"test-k6-plugin-topic", // Kafka topic
"test-k6-extension-topic", // Kafka topic
)

// Read 10 messages only
Expand All @@ -132,7 +120,7 @@ export default function () {
You can run k6 with the Kafka extension using the following command:

```bash
$ ./k6 run --vus 1 --duration 10s test.js
$ ./k6 run --vus 50 --duration 60s test.js
```

And here's the test result output:
Expand Down
10 changes: 0 additions & 10 deletions build.sh

This file was deleted.

0 comments on commit 53be418

Please sign in to comment.