Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ISSUE #523 & OSPP2023] Feature: support archetype to create connectors #525

Merged
merged 11 commits into from
Oct 17, 2023
Merged

[ISSUE #523 & OSPP2023] Feature: support archetype to create connectors #525

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5117ede
initial commit
joeCarf May 5, 2023
25e7ba5
commit
joeCarf May 5, 2023
54c2517
rename
joeCarf May 5, 2023
1957ce8
finish source connector
joeCarf May 6, 2023
eb5033d
finish sink connector
joeCarf May 6, 2023
969b395
add license
joeCarf May 6, 2023
ca0d454
Update README.md
joeCarf May 6, 2023
31cd5e4
Update connect-standalone.conf
joeCarf May 6, 2023
d5c7461
Merge branch 'apache:master' into dev/clickhouse-source
joeCarf Jul 25, 2023
264ae81
support archetype
joeCarf Aug 20, 2023
869a989
Delete README.md
joeCarf Aug 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions connectors/rocketmq-connect-connectors-archetype/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
## How to Use Connnector-Archetype

1. 进入脚手架文件夹

```shell
cd rocketmq-connect-connectors-archetype/
```

2. 将脚手架安装到本地

```shell
mvn -e clean install
```

3. 创建connector模版工程

```shell
cd connectors/
mvn archetype:generate \
-DarchetypeGroupId=org.apache.rocketmq \
-DarchetypeArtifactId=rocketmq-connect-connectors-archetype \
-DarchetypeVersion=1.0-SNAPSHOT \
-DdatabaseName=<databasename>
```

例：创建Clickhouse-Connector

```shell
mvn archetype:generate \
-DarchetypeGroupId=org.apache.rocketmq \
-DarchetypeArtifactId=rocketmq-connect-connectors-archetype \
-DarchetypeVersion=1.0-SNAPSHOT \
-DdatabaseName=clickhouse
```

4. 如上指令将创建一个connector的框架，开发者主要关心`helper/xxxHelperClient`以及`xxxxSourceTask`,`xxxSinkTask`的实现即可，剩余配置可以按需修改。
37 changes: 37 additions & 0 deletions connectors/rocketmq-connect-connectors-archetype/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-connect-connectors-archetype</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>maven-archetype</packaging>

<name>rocketmq-connect-connectors-archetype</name>

<build>
<extensions>
<extension>
<groupId>org.apache.maven.archetype</groupId>
<artifactId>archetype-packaging</artifactId>
<version>3.2.1</version>
</extension>
</extensions>

<pluginManagement>
<plugins>
<plugin>
<artifactId>maven-archetype-plugin</artifactId>
<version>3.2.1</version>
</plugin>
</plugins>
</pluginManagement>
</build>

<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
</license>
</licenses>
</project>
66 changes: 66 additions & 0 deletions ...tmq-connect-connectors-archetype/src/main/resources/META-INF/maven/archetype-metadata.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
<?xml version="1.0" encoding="UTF-8"?>
<archetype-descriptor xsi:schemaLocation="https://maven.apache.org/plugins/maven-archetype-plugin/archetype-descriptor/1.1.0 http://maven.apache.org/xsd/archetype-descriptor-1.1.0.xsd" name="rocketmq-connect-connectors-archetype"
xmlns="https://maven.apache.org/plugins/maven-archetype-plugin/archetype-descriptor/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<requiredProperties>
<!--必填属性-->
<requiredProperty key="groupId">
<defaultValue>org.apache.rocketmq</defaultValue>
</requiredProperty>
<requiredProperty key="artifactId">
<defaultValue>rocketmq-connect-${dbNameToLowerCase}</defaultValue>
</requiredProperty>
<requiredProperty key="version">
<defaultValue>1.0.0-SNAPSHOT</defaultValue>
</requiredProperty>
<requiredProperty key="package">
<defaultValue>org.apache.rocketmq.connect.${dbNameToLowerCase}</defaultValue>
</requiredProperty>

<!-- The database to connect -->
<requiredProperty key="databaseName"/>

<requiredProperty key="dbNameToUpperCase" >
<defaultValue>${databaseName.toUpperCase()}</defaultValue>
</requiredProperty>

<requiredProperty key="dbNameToCamel" >
<defaultValue>${databaseName.toLowerCase().substring(0,1).toUpperCase()}${databaseName.toLowerCase().substring(1)}</defaultValue>
</requiredProperty>

<requiredProperty key="dbNameToLowerCase" >
<defaultValue>${databaseName.toLowerCase()}</defaultValue>
</requiredProperty>
</requiredProperties>

<fileSets>
<fileSet filtered="true" encoding="UTF-8">
<directory/>
<includes>
<include>.reviewboardrc</include>
<include>README.md</include>
</includes>
</fileSet>
<fileSet encoding="UTF-8">
<directory/>
<includes>
<include>.gitignore</include>
<include>TODO.md</include>
</includes>
</fileSet>
<fileSet encoding="UTF-8">
<directory/>
<includes>
<include>.gitignore</include>
</includes>
</fileSet>
<fileSet filtered="true" packaged="true" encoding="UTF-8">
<directory>src/main/java</directory>
</fileSet>
<fileSet filtered="true" packaged="true" encoding="UTF-8">
<directory>src/test/java</directory>
</fileSet>
</fileSets>

</archetype-descriptor>
55 changes: 55 additions & 0 deletions ...q-connect-connectors-archetype/src/main/resources/archetype-resources/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
## FIXME: fix document

##### ${dbNameToCamel}SourceConnector fully-qualified name

org.apache.rocketmq.connect.${dbNameToLowerCase}.source.${dbNameToCamel}SourceConnector

**${dbNameToLowerCase}-source-connector** start

```
POST http://${runtime-ip}:${runtime-port}/connectors/${dbNameToLowerCase}SourceConnector
{
"connector.class":"org.apache.rocketmq.connect.${dbNameToLowerCase}.source.${dbNameToCamel}SourceConnector",
"${dbNameToLowerCase}host":"localhost",
"${dbNameToLowerCase}port":8123,
"database":"default",
"username":"default",
"password":"123456",
"table":"tableName",
"topic":"test${dbNameToCamel}Topic",
"value.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter",
"key.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter"
}
```

##### ${dbNameToCamel}SinkConnector fully-qualified name

org.apache.rocketmq.connect.${dbNameToLowerCase}.sink.${dbNameToCamel}SinkConnector

**${dbNameToLowerCase}-sink-connector** start

```
POST http://${runtime-ip}:${runtime-port}/connectors/${dbNameToLowerCase}SinkConnector
{
"connector.class":"org.apache.rocketmq.connect.${dbNameToLowerCase}.sink.${dbNameToCamel}SinkConnector",
"${dbNameToLowerCase}host":"localhost",
"${dbNameToLowerCase}port":8123,
"database":"clickhouse",
"username":"default",
"password":"123456",
"connect.topicnames":"test${dbNameToCamel}Topic",
"value.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter",
"key.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter"
}
```

##### parameter configuration

| parameter | effect | required | default |
|--------------------------|---------------------------------------------------|-------------------|---------|
| ${dbNameToLowerCase}host | The Host of the ${dbNameToCamel} server | yes | null |
| ${dbNameToLowerCase}port | The Port of the ${dbNameToCamel} server | yes | null |
| database | The database to read or write | yes | null |
| table | The source table to read | yes (source only) | null |
| topic | RocketMQ topic for source connector to write into | yes (source only) | null |
| connect.topicnames | RocketMQ topic for sink connector to read from | yes (sink only) | null |
188 changes: 188 additions & 0 deletions ...tors/rocketmq-connect-connectors-archetype/src/main/resources/archetype-resources/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for additional
information regarding copyright ownership. The ASF licenses this file to
You under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of
the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
by applicable law or agreed to in writing, software distributed under the
License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
OF ANY KIND, either express or implied. See the License for the specific
language governing permissions and limitations under the License. --><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>${groupId}</groupId>
<artifactId>${artifactId}</artifactId>
<version>${version}</version>

<name>connect-${dbNameToLowerCase}</name>

<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
</license>
</licenses>

<issueManagement>
<system>jira</system>
<url>https://issues.apache.org/jira/browse/RocketMQ</url>
</issueManagement>

<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>versions-maven-plugin</artifactId>
<version>2.3</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>clirr-maven-plugin</artifactId>
<version>2.7</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<compilerVersion>${maven.compiler.source}</compilerVersion>
<showDeprecation>true</showDeprecation>
<showWarnings>true</showWarnings>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<argLine>-Xms512m -Xmx1024m</argLine>
<forkMode>always</forkMode>
<includes>
<include>**/*Test.java</include>
</includes>
</configuration>
</plugin>
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.6</version>
<configuration>
<locales>en_US</locales>
<outputEncoding>UTF-8</outputEncoding>
<inputEncoding>UTF-8</inputEncoding>
</configuration>
</plugin>
<plugin>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.4</version>
<configuration>
<charset>UTF-8</charset>
<locale>en_US</locale>
<excludePackageNames>io.openmessaging.internal</excludePackageNames>
</configuration>
<executions>
<execution>
<id>aggregate</id>
<goals>
<goal>aggregate</goal>
</goals>
<phase>site</phase>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<encoding>${project.build.sourceEncoding}</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>3.0.4</version>
</plugin>
<plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<version>0.12</version>
<configuration>
<excludes>
<exclude>README.md</exclude>
<exclude>README-CN.md</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>io.openmessaging</groupId>
<artifactId>openmessaging-connector</artifactId>
<version>0.1.4</version>
<scope>compile</scope>
</dependency>

<!-- FIXME: Write dependencies here-->

<dependency>
<groupId>org.lz4</groupId>
<artifactId>lz4-java</artifactId>
<version>1.8.0</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.83</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>RELEASE</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
</dependencies>

</project>
Loading