WHALE

Social Sensing environment with Experience Sampling capabilities.

Tasks

• Import Experience Sampling elements from Excel tables
• Allow to sample on lock/unlock and continuously in a set interval (maybe we can define some sensors to run in a periodic interval and others to sample on interaction)
• Implement API in app (pushing data, showing study name)
• Add notification sensor
• Add interaction tracking bubble
• Add onboarding interface (permissions, enrolment) and basic main activity for the app
• Use WorkManager to schedule backend pushes
• Add radio frequency (Bluetooth and WiFi) sensor
• Detect if a conversation happens (see openSMILE)
• Revamp AppSensor so it actually retrieves the currently open app (probably with AccessibilityService)
• Sample whenever the device gets unlocked
• Create simple backend that receives pushed data (audio and other sample readings)
• Add (crude) study enrolment interface
• Integrate AudioSampleService to use default content pipeline

Components

• Android App: The app that records sensor data. Needs to be installed on participant's devices
• Backend: The backend that allows for data storage. Deployed on a server.

Note

See the component's respective READMEs for more information.

Interaction Bubble

We want to track if an interaction is happening and relying on sensors alone would not be sufficient. We use the following approach:

• a sticky and unobtrusive widget flows around the screen
• when no interaction was marked, we ask the user on unlock, if they are in an interaction
• on each unlock, we ask if this interaction is still ongoing
• if they mark the interaction as ended, an experience sampling form is triggered

---
title: Interaction Bubble State
---
stateDiagram-v2
    state "No Interaction" as NoInteraction
    state "In Interaction" as InInteraction
    state "Ask on Unlock" as AskOnUnlock
    state "Ask if Ongoing" as AskIfOngoing
    state "Trigger ESM Form" as TriggerForm
    
    [*] --> NoInteraction
    NoInteraction --> AskOnUnlock: Unlock
    AskOnUnlock --> InInteraction: Yes Interaction
    AskOnUnlock --> NoInteraction: No interaction
    
    InInteraction --> AskIfOngoing: Unlock
    AskIfOngoing --> InInteraction: Still ongoing
    AskIfOngoing --> TriggerForm: Ended
    TriggerForm --> NoInteraction: Form completed

Loading

Experience Sampling

We want to trigger an experience sampling form when specific events happen:

• important: user marks an interaction ended
• possibly in a specific interval

A sampling could include the following:

• content separated in steps (title, position)
  • visual/content
    • (rich) text
  • input
    • text
    • radio
    • checkbox
    • Likert scale
    • slider
    • affect grid
    • time/duration

---
title: ESM Model
---
erDiagram
    Study
    Questionnaire {
        bool enabled
        string name
        number version
    }
    Trigger {
        string eventType
        bool enabled
        object configuration
    }
    Element {
        string type
        number step
        number position
        object configuration
    }
    Questionnaire ||--|{ Trigger : "has"
    Questionnaire ||--o{ Study : "belongs to"
    Element ||--o{ Questionnaire : "belongs to"

Loading

After updating any element of the questionnaire, its version needs to be updated. The app will only update the questionnaire if the version is higher than the one currently cached.

Acknowledgements & References

Based on the SenseEverything app.

• Weber, D., & Mayer, S. (2014). LogEverything. GitHub Repository. Retrieved from https://github.com/hcilab-org/LogEverything/