Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding ground truth labeling module #252

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

EthanBunce
Copy link
Collaborator

Describe your changes

Added the sagemaker ground truth labeling module. Module provides a workflow to label items using sagemaker ground truth labeling. Users upload the data to a bucket, which is then sent to an SQS queue. A state machine pulls data from the queue, runs a labeling job and a verification job (if the task type supports it), before saving the labels to a sagemaker feature store.

Module currently supports all built in task types for image and text labeling, but could be expanded to video and 3d point in the future.

Checklist before requesting a review

  • I updated CHANGELOG.MD with a description of my changes
  • If the change was to a module, I ran the code validation script (scripts/validate.sh)
  • If the change was to a module, I have added thorough tests
  • If the change was to a module, I have added/updated the module's README.md
  • If a module was added, I added a reference to the module to the repository's README.md
  • I verified that my code deploys successfully using seedfarmer apply

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: AIOpsModuleTestsInfrastruct-doMLXEYsnmxr
  • Commit ID: f228d22
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

return {"FeatureName": feature_name, "ValueAsString": feature_value}


def multi_threaded_batch_delete_message_from_sqs_and_save_to_feature_store(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decided to multi-thread here due to sagemaker feature store not having a batch put operation. Running single threaded would cause issues at scale with lambda time limits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants