Skip to content

Commit

Permalink
docs: Add Meltano instructions (#28)
Browse files Browse the repository at this point in the history
  • Loading branch information
edgarrmondragon authored Feb 23, 2023
1 parent 4cb599d commit 94f1ff6
Show file tree
Hide file tree
Showing 12 changed files with 92 additions and 11 deletions.
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# Config files
*.json
*.yaml
!*.example.*

# Data files
Expand Down
26 changes: 26 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
ci:
autofix_prs: false
autoupdate_schedule: weekly
autoupdate_commit_msg: 'chore(deps): pre-commit autoupdate'

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: check-added-large-files
- id: check-json
- id: check-toml
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace

- repo: https://github.com/pre-commit/pre-commit
rev: v3.0.4
hooks:
- id: validate_manifest

- repo: https://github.com/python-jsonschema/check-jsonschema
rev: 0.21.0
hooks:
- id: check-dependabot
- id: check-github-workflows
58 changes: 57 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,63 @@ go build -o target-jsonl-blob

## Usage with Meltano

_TODO_
1. [Download the appropriate asset](#1-download-the-appropriate-asset)
1. [Allow execution of the downloaded binary](#2-allow-execution-of-the-downloaded-binary)
1. [Add a custom Meltano plugin to your project](#3-add-a-custom-meltano-plugin-to-your-project)
1. [Run a pipeline](#4-run-a-pipeline)

### 1. Download the appropriate asset

You can see the full list of assets in the release page: https://github.com/MeltanoLabs/target-jsonl-blob/releases/latest.

The [`gh`](https://cli.github.com/) tool makes downloading an asset easy:

```bash
gh release download v0.0.4 \
-R MeltanoLabs/target-jsonl-blob \
-p '*darwin-amd64' \
--clobber \
-O target-jsonl-blob
```

### 2. Allow execution of the downloaded binary

```bash
chmod +x target-jsonl-blob
```

### 3. Add a custom Meltano plugin to your project

```yaml
# meltano.yml
plugins:
loaders:
- name: target-jsonl-blob
namespace: target_jsonl_blob
executable: ./target-jsonl-blob
settings:
- name: bucket
label: Bucket
description: Target directory (local, S3, Azure Blob)
- name: key_template
label: Key Template
description: Template string for file keys
config:
bucket: file://./output/my-bucket
key_template: $MELTANO_EXTRACTOR_NAMESPACE/{{.StreamName}}.jsonl
```
You also need to ensure the local "bucket" exists:
```bash
mkdir output/my-bucket
```

### 4. Run a pipeline

```bash
meltano run tap-github target-jsonl-blob
```

## Roadmap

Expand Down
2 changes: 1 addition & 1 deletion cmd/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
Expand Down
2 changes: 1 addition & 1 deletion main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
Expand Down
2 changes: 1 addition & 1 deletion target/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
Expand Down
2 changes: 1 addition & 1 deletion target/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
Expand Down
2 changes: 1 addition & 1 deletion target/messages.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
Expand Down
2 changes: 1 addition & 1 deletion target/messages_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
Expand Down
2 changes: 1 addition & 1 deletion target/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
Expand Down
2 changes: 1 addition & 1 deletion target/testdata/config.good.example.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"bucket": "s3://my-bucket"
}
}
2 changes: 1 addition & 1 deletion target/testdata/config.no-bucket.example.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"someOtherKey": "value"
}
}

0 comments on commit 94f1ff6

Please sign in to comment.