Skip to content

Commit

Permalink
chore: Clean up (#6)
Browse files Browse the repository at this point in the history
  • Loading branch information
edgarrmondragon authored Dec 30, 2024
1 parent 45e54ec commit 7d1d80e
Show file tree
Hide file tree
Showing 9 changed files with 56 additions and 67 deletions.
1 change: 1 addition & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ permissions:

jobs:
build:
name: Build and Inspect Python Package
runs-on: ubuntu-latest
outputs:
package_version: ${{ steps.baipp.outputs.package_version }}
Expand Down
45 changes: 16 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,23 @@
# target-starrocks

`target-starrocks` is a Singer target for [Starrocks].
`target-starrocks` is a Singer target for [StarRocks].

Build with the [Meltano Target SDK][Singer SDK].

<!--
Developer TODO: Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPi repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
## Installation

Install from PyPi:
Install from PyPI:

```bash
pipx install target-starrocks
uv tool install target-starrocks
```

Install from GitHub:

```bash
pipx install git+https://github.com/ORG_NAME/target-starrocks.git@main
uv tool install git+https://github.com/reservoir-data/target-starrocks.git@main
```

-->

## Supported Python Versions

* 3.9
Expand All @@ -38,11 +32,11 @@ pipx install git+https://github.com/ORG_NAME/target-starrocks.git@main

| Setting | Required | Default | Description |
| :-------------------------------- | :------- | :---------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| user | False | None | User for the Starrocks database |
| password | False | None | Password for the Starrocks database |
| host | False | None | Host for the Starrocks database |
| port | False | 9030 | Port for the Starrocks database |
| database | True | None | Database name for the Starrocks database |
| user | False | None | User for the StarRocks database |
| password | False | None | Password for the StarRocks database |
| host | False | None | Host for the StarRocks database |
| port | False | 9030 | Port for the StarRocks database |
| database | True | None | Database name for the StarRocks database |

#### Additional Config Options

Expand Down Expand Up @@ -75,10 +69,6 @@ environment variable is set either in the terminal context or in the `.env` file

### Authentication and Authorization

<!--
Developer TODO: If your target requires special access on the destination system, or any special authentication requirements, provide those here.
-->

## Usage

You can easily run `target-starrocks` by itself or in a pipeline using [Meltano](https://meltano.com/).
Expand All @@ -99,7 +89,7 @@ Follow these instructions to contribute to this project.
### Initialize your Development Environment

```bash
pipx install poetry
uv tool install poetry
poetry install
```

Expand All @@ -123,17 +113,13 @@ poetry run target-starrocks --help
_**Note:** This target will work in any Singer environment and does not require Meltano.
Examples here are for convenience and to streamline end-to-end orchestration scenarios._

<!--
Developer TODO:
Your project comes with a custom `meltano.yml` project file already created. Open the `meltano.yml` and follow any "TODO" items listed in
the file.
-->

Next, install Meltano (if you haven't already) and any needed plugins:

```bash
# Install meltano
pipx install meltano
uv tool install meltano
meltano config meltano set venv.backend uv

# Initialize meltano within this directory
cd target-starrocks
meltano install
Expand All @@ -144,7 +130,8 @@ Now you can test and orchestrate using Meltano:
```bash
# Test invocation:
meltano invoke target-starrocks --version
# OR run a test `elt` pipeline with the Smoke Test sample tap:

# OR run a test EL pipeline with the Smoke Test sample tap:
meltano run tap-smoke-test target-starrocks
```

Expand All @@ -153,5 +140,5 @@ meltano run tap-smoke-test target-starrocks
See the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the Meltano Singer SDK to
develop your own Singer taps and targets.

[Starrocks]: https://starrocks.io
[StarRocks]: https://starrocks.io
[Singer SDK]: https://sdk.meltano.com
11 changes: 6 additions & 5 deletions meltano.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ plugins:
- name: tap-fedidb
variant: edgarrmondragon
pip_url: tap-fedidb

loaders:
- name: target-starrocks
namespace: target_starrocks
Expand All @@ -33,29 +34,29 @@ plugins:
- name: user
kind: string
label: User
description: User for the Starrocks database
description: User for the StarRocks database

- name: password
kind: string
label: Password
sensitive: true
description: Password for the Starrocks database
description: Password for the StarRocks database

- name: host
kind: string
label: Host
description: Host for the Starrocks database
description: Host for the StarRocks database

- name: port
kind: integer
label: Port
value: 9030
description: Port for the Starrocks database
description: Port for the StarRocks database

- name: database
kind: string
label: Database
description: Database for the Starrocks database
description: Database for the StarRocks database

settings_group_validation:
- [database]
Expand Down
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
[tool.poetry]
name = "target-starrocks"
version = "0.0.0"
description = "Singer target for Starrocks, built with the Meltano Singer SDK."
description = "Singer target for StarRocks, built with the Meltano Singer SDK."
readme = "README.md"
authors = ["Edgar Ramírez-Mondragón <edgarrm358@gmail.com>"]
keywords = [
"ELT",
"Starrocks",
"StarRocks",
]
classifiers = [
"Intended Audience :: Developers",
Expand Down Expand Up @@ -65,4 +65,4 @@ requires = ["poetry-core>=1.0.0", "poetry-dynamic-versioning>=1.0.0,<2.0.0"]
build-backend = "poetry_dynamic_versioning.backend"

[tool.poetry.scripts]
target-starrocks = 'target_starrocks.target:TargetStarrocks.cli'
target-starrocks = 'target_starrocks.target:TargetStarRocks.cli'
2 changes: 1 addition & 1 deletion target_starrocks/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
"""Target for Starrocks."""
"""Target for StarRocks."""
6 changes: 3 additions & 3 deletions target_starrocks/__main__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""Starrocks entry point."""
"""StarRocks entry point."""

from __future__ import annotations

from target_starrocks.target import TargetStarrocks
from target_starrocks.target import TargetStarRocks

TargetStarrocks.cli()
TargetStarRocks.cli()
24 changes: 12 additions & 12 deletions target_starrocks/sinks.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Starrocks target sink class, which handles writing streams."""
"""StarRocks target sink class, which handles writing streams."""

from __future__ import annotations

Expand All @@ -12,14 +12,14 @@
from starrocks.datatype import JSON


class JSONSchemaToStarrocks(JSONSchemaToSQL):
"""Convert a JSON schema to a Starrocks-compatible SQL type."""
class JSONSchemaToStarRocks(JSONSchemaToSQL):
"""Convert a JSON schema to a StarRocks-compatible SQL type."""

def __init__(self, *args: Any, **kwargs: Any) -> None:
"""Initialize the type converter."""
super().__init__(*args, **kwargs)
if self._max_varchar_length is None:
msg = "max_varchar_length is required for Starrocks"
msg = "max_varchar_length is required for StarRocks"
raise ValueError(msg)

def handle_raw_string(self, schema: dict) -> sa.types.TypeEngine:
Expand All @@ -32,8 +32,8 @@ def handle_raw_string(self, schema: dict) -> sa.types.TypeEngine:
return sa.types.VARCHAR(length=max_length or self._max_varchar_length)


class StarrocksConnector(SQLConnector):
"""The connector for Starrocks.
class StarRocksConnector(SQLConnector):
"""The connector for StarRocks.
This class handles all DDL and type conversions.
"""
Expand All @@ -52,15 +52,15 @@ class StarrocksConnector(SQLConnector):
max_varchar_length: int = 65_533

@functools.cached_property
def jsonschema_to_sql(self) -> JSONSchemaToStarrocks:
def jsonschema_to_sql(self) -> JSONSchemaToStarRocks:
"""The JSON-to-SQL type mapper object for this SQL connector."""
to_sql = JSONSchemaToStarrocks(max_varchar_length=self.max_varchar_length)
to_sql = JSONSchemaToStarRocks(max_varchar_length=self.max_varchar_length)
to_sql.register_type_handler("array", JSON)
to_sql.register_type_handler("object", JSON)
return to_sql

def get_sqlalchemy_url(self, config: dict) -> str:
"""Generates a SQLAlchemy URL for Starrocks.
"""Generates a SQLAlchemy URL for StarRocks.
Args:
config: The configuration for the connector.
Expand All @@ -76,7 +76,7 @@ def get_sqlalchemy_url(self, config: dict) -> str:
).render_as_string(hide_password=False)


class StarrocksSink(SQLSink):
"""Starrocks target sink class."""
class StarRocksSink(SQLSink):
"""StarRocks target sink class."""

connector_class = StarrocksConnector
connector_class = StarRocksConnector
22 changes: 11 additions & 11 deletions target_starrocks/target.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
"""Starrocks target class."""
"""StarRocks target class."""

from __future__ import annotations

from singer_sdk import typing as th
from singer_sdk.target_base import SQLTarget

from target_starrocks.sinks import (
StarrocksSink,
StarRocksSink,
)


class TargetStarrocks(SQLTarget):
"""Sample target for Starrocks."""
class TargetStarRocks(SQLTarget):
"""Sample target for StarRocks."""

name = "target-starrocks"

Expand All @@ -20,39 +20,39 @@ class TargetStarrocks(SQLTarget):
"user",
th.StringType,
title="User",
description="User for the Starrocks database",
description="User for the StarRocks database",
),
th.Property(
"password",
th.StringType,
title="Username",
description="Password for the Starrocks database",
description="Password for the StarRocks database",
secret=True,
),
th.Property(
"host",
th.StringType,
title="Host",
description="Host for the Starrocks database",
description="Host for the StarRocks database",
),
th.Property(
"port",
th.IntegerType,
title="Port",
description="Port for the Starrocks database",
description="Port for the StarRocks database",
default=9030,
),
th.Property(
"database",
th.StringType,
title="Database",
description="Database name for the Starrocks database",
description="Database name for the StarRocks database",
required=True,
),
).to_dict()

default_sink_class = StarrocksSink
default_sink_class = StarRocksSink


if __name__ == "__main__":
TargetStarrocks.cli()
TargetStarRocks.cli()
6 changes: 3 additions & 3 deletions tests/test_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@
import pytest
from singer_sdk.testing import get_target_test_class

from target_starrocks.target import TargetStarrocks
from target_starrocks.target import TargetStarRocks

# Run standard built-in target tests from the SDK:
StandardTargetTests = get_target_test_class(target_class=TargetStarrocks, config={})
StandardTargetTests = get_target_test_class(target_class=TargetStarRocks, config={})


class TestTargetStarrocks(StandardTargetTests): # type: ignore[misc, valid-type]
class TestTargetStarRocks(StandardTargetTests): # type: ignore[misc, valid-type]
"""Standard Target Tests."""

@pytest.fixture(scope="class")
Expand Down

0 comments on commit 7d1d80e

Please sign in to comment.