Welcome to rtdl Discussions!

[thtmnisamnstr]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[misterbykl]

Hi. I am a senior data engineer with a background of streaming and batch analytics. I am very interested in this project and would like to contribute as much as I can. Although I don't have any expertise in Go language, I would be happy to be part of this project in some way. I would be glad if you could guide me. Thanks.

[thtmnisamnstr]

That would be awesome. We’d love to find a way for you to contribute. What programming languages are you most comfortable with?

[thtmnisamnstr]

Looking at your background, it might be good for you to see if you can build a Flink Stateful Function (StateFun) to extend rtdl’s processing functionality. StateFun functions can be written in Java, Python, Go, or Javascript.

The provided ingester stateful function reads JSON, converts to Parquet, and writes to storage. The next one on our list is PII detection and removal/obfuscation.

If you’re interested in helping with that, let us know. Once we hear back, @dipanjanb will prep a small requirements doc explaining what we are looking for and how it should work with rtdl.

[misterbykl]

Thanks!. That would be great. I am experienced in Java, Javascript and Python. I feel more comfortable in Java. And Flink is one of my favorite framework. Let's begin with it. Looking forward for the details.

[thtmnisamnstr]

Dipanjan replied with some design guidance in the main discussion thread.

[dipanjanb]

Hi Emre, Attaching a document where I've tried to capture my thoughts. Please go through it and let me know in case you have any questions. What I'd suggest is that you fork the repo, create a branch therein and keep adding your code to that branch. Once you're satisfied - you can raise a PR - which we can merge to the main repo post review and subsequently merge the branch you created to the main branch as part of a release. Please feel free to reach out at any time should you have any questions. Thanks & Regards, Dipanjan

…

On Mon, 28 Feb 2022 at 21:21, Emre Baykal ***@***.***> wrote: Thanks!. That would be great. I am experienced in Java, Javascript and Python. I feel more comfortable in Java. And Flink is one of my favorite framework. Let's begin with it. Looking forward for the details. — Reply to this email directly, view it on GitHub <#3 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDSZX4DBEDFYYCAFB4SDU5OKW7ANCNFSM5NVLMLDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dipanjanb]

Sorry - realized now that I replied on mail and the attachment isn't reflecting - uploaded now
Guide to Adding Stateful Function to RTDL.docx

[misterbykl]

Thanks. I'll let you know once I review the doc.

[misterbykl]

@dipanjanb I reviewed the doc. It might take some time for me to understand the structure of the code. Do you have deadlines for feature developments like this?

[thtmnisamnstr]

Dioanjan replied in the main thread.

Hi Emre,

We value community engagement.
Hence please take your time to understand the code and let us know what
kind of timeline you anticipate for developing the functionality outlined.

On Tue, 1 Mar 2022 at 01:04, Emre Baykal @.***> wrote:

@dipanjanb https://github.com/dipanjanb I reviewed the doc. It might
take some time for me to understand the structure of the code. Do you have
deadlines for feature developments like this?

[dipanjanb]

Hi Emre, We value community engagement. Hence please take your time to understand the code and let us know what kind of timeline you anticipate for developing the functionality outlined.

…

On Tue, 1 Mar 2022 at 01:04, Emre Baykal ***@***.***> wrote: @dipanjanb <https://github.com/dipanjanb> I reviewed the doc. It might take some time for me to understand the structure of the code. Do you have deadlines for feature developments like this? — Reply to this email directly, view it on GitHub <#3 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDSYBEUAFN2PARM52RSDU5PE3XANCNFSM5NVLMLDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[misterbykl]

[misterbykl]

I tried to reproduce the issue but everything is working now. Strange. My bad I guess.

[misterbykl]

Hi Dipanjan. I've done the following so far.

• Implemented pii-detection module in Java using state-fun structure.
• Added the functionality of http-server, stateful-function and message type.

I am currently working on module.yaml , docker-compose.yml and Dockerfile. Next step is to deploy the system and run some tests.
Meantime, can you inform me about the pii-detection logic? Thanks!

[dipanjanb]

Hi Emre, You can refer to this article https://medium.com/snowflake/basic-pii-detection-and-masking-in-snowflake-using-java-1689ae63aa69 which contains sample PII detection code. This is for creating a Java-based PII detection function for Snowflake - you can integrate the detection logic (coded in Java) into the Stateful Function you're building.

…

On Fri, 4 Mar 2022 at 11:59, Emre Baykal ***@***.***> wrote: Hi Dipanjan. I've done the following so far. - Implemented pii-detection module in Java using state-fun structure. - Added the functionality of http-server, stateful-function and message type. I am currently working on module.yaml , docker-compose.yml and Dockerfile. Next step is to deploy the system and run some tests. Meantime, can you inform me about the pii-detection logic? Thanks! — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDSZA5OIW7BJWHGTMKVDU6GUWLANCNFSM5NVLMLDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[misterbykl]

Hi @dipanjanb @thtmnisamnstr. Thanks again for all your help on my first PR. I can't wait for my next challenge. Let me know if there are more tasks waiting (new feature, blog post, documentation).

[dipanjanb]

You're most welcome Emre :-) Also - you did a fine job and would definitely love to involve you in upcoming initiatives. Thanks & Regards, Dipanjan <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

…

On Thu, 17 Mar 2022 at 21:40, Emre Baykal ***@***.***> wrote: Hi @dipanjanb <https://github.com/dipanjanb> @thtmnisamnstr <https://github.com/thtmnisamnstr>. Thanks again for all your help on my first PR. I can't wait for my next challenge. Let me know if there are more tasks waiting (new feature, blog post, documentation). — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDS6LXWIPTPCQXTKYQPDVANKQTANCNFSM5NVLMLDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>