Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: A field referencing record in streams maps gets incorrectly mapped to a string type in the schema #1882

Closed
1 task
Tracked by #1883
edgarrmondragon opened this issue Jul 28, 2023 · 0 comments · Fixed by #1885
Closed
1 task
Tracked by #1883
Labels
kind/Bug Something isn't working valuestream/SDK

Comments

@edgarrmondragon
Copy link
Collaborator

Singer SDK Version

0.30.0

Is this a regression?

  • Yes

Python Version

NA

Bug scope

Mapping (stream maps, flattening, etc.)

Operating System

NA

Description

Consider the countries sample in the SDK repo and the following stream maps configuration:

{
    "stream_maps": {
        "countries": {
            "_data": "record",
            "__key_properties__": [],
            "__else__": null
        }
    },
    "stream_map_config": {}
}

Then run

poetry run python samples/sample_tap_countries --config stream_maps.json > mapped_countries.singer.jsonl

Note that in the resulting SCHEMA message, the _data field is of type string. This is because record is not recognized as a special name and the type inference logic falls back to the default string type:

sdk/singer_sdk/mapper.py

Lines 370 to 384 in 759c77b

default = default or StringType()
if expr.startswith("float("):
return NumberType()
if expr.startswith("int("):
return IntegerType()
if expr.startswith("str("):
return StringType()
if expr[0] == "'" and expr[-1] == "'":
return StringType()
return default

This should be relatively easy to fix by adding a new branch

if expr == "record":
    return ObjectType()

Code

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/Bug Something isn't working valuestream/SDK
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant