QoE Metrics Reporting - Investigate and derive metrics via PlaybackStatsListener #27

dsilhavy · 2023-06-01T09:59:29Z

Feature description
From the media3 Exoplayer documentation:

PlaybackStatsListener is an AnalyticsListener that implements on-device event processing. It calculates PlaybackStats, with counters and derived metrics including:

Summary metrics, for example the total playback time.
Adaptive playback quality metrics, for example the average video resolution.
Rendering quality metrics, for example the rate of dropped frames.
Resource usage metrics, for example the number of bytes read over the network.

Goal

Investigate which metrics are exposed via the PlaybackStatsListener and if it makes sense to use them directly with a newly defined metrics reporting scheme.
Investigate the relationship to the metrics scheme referenced in 26.512 linking to 26.247: For downlink media streaming, TS 26.247 [7] clauses 10.6.1 and 10.6.2 specify the required MIME content type and metrics report format for the 3GPP urn:3GPP:ns:PSS:DASH:QM10 metrics reporting scheme.

The text was updated successfully, but these errors were encountered:

rjb1000 · 2023-06-01T10:40:50Z

Investigate the relationship to the metrics scheme referenced in 26.512 linking to 26.247: For downlink media streaming, TS 26.247 [7] clauses 10.6.1 and 10.6.2 specify the required MIME content type and metrics report format for the 3GPP urn:3GPP:ns:PSS:DASH:QM10 metrics reporting scheme.

Yes. Putting together a mapping table on a Wiki page is probably the first thing I would do. This would be a very useful piece of analysis that would be of benefit to others following in your footsteps.

dsilhavy · 2023-06-19T11:21:11Z

Using this ticket to also collect open questions that I need to check and we potentially need to discuss:

Should the buffer level be derived once the metrics are requested or at a certain interval. For the latter, we can then return a list of all the collected bufferlevel entries since the last request.
Discuss whether we serialize the metrics in the MediaSessionHandler or already in the MediaStreamHandler. Serializing them in the MediaSessionHandler centralizes this step. However, some players might already support the target XML format in which case we can simply forward it from the MediaSessionHandler to the AF.
RepSwitchList does not contain a media type. Provider has to infer from the Representation@id what the media type is.

dsilhavy · 2023-06-19T14:03:21Z

Added initial support for:

Deriving the buffer level via ExoPlayerAdapter : playerInstance.totalBufferedDuration
Added a new class QoEMetricsExoPlayer to derive metrics from the Exoplayer and serialize them according to the urn:3GPP:ns:PSS:DASH:QM10 profile. Currently only BufferLevel is supported
Added model classes for BufferLevel and QoEMetricsReport in the 5GMSCommonLibrary
Integrated simpleframework.xml for serialization to XML
Added messages to request and send the metrics between MediaStreamHandler and MediaSessionHandler

Currently, a metrics message that is received by the MediaSessionHandler looks like this:

<QoeReport>
   <QoeMetric class="java.util.ArrayList">
      <object class="com.fivegmag.a5gmscommonlibrary.qoeMetricsModels.threeGPP.BufferLevel">
         <BufferLevel class="java.util.ArrayList">
            <BufferLevelEntry level="3982" t="1687183773586"/>
            <BufferLevelEntry level="3984" t="1687183773586"/>
         </BufferLevel>
      </object>
   </QoeMetric>
</QoeReport>

The above messages needs some cleanup:

Remove references to Android classes
As it does not seem to be possible to assign multiple elements with the same name (we need multiple QoeMetric elements) I introduced a workaround with @field:ElementList(name = "QoeMetric") val entries: List<Any> for keeping all metrics.

All of the above serves as a starting point to derive additional metrics and serialize them.

rjb1000 · 2023-06-19T14:40:32Z

@dsilhavy: A couple of things missing from your QoE metrics report:

XML processing instruction. All XML instance documents are required to provide this on the first line.
Root element. The root element of a QoE metrics report document is <ReceptionReport>, as specified in clause 10.6.2 of TS 26.247.
- See clause 10.6.3 of TS 26.247 for an example.
- This root element wraps around <QoeReport> in your example above.
- You'll also need to specify the correct name space for this root element as urn:3gpp:metadata:2011:HSD:receptionreport.
- There are a couple more attributes to set in this root elememt:
  - @contentURI (mandatory). The Media Player Entry Point URL. For 5GMSd this is the M4 URL of the DASH MPD.
  - @clientID (optional in TS 26.247; recommended in TS 26.512). This is not properly specified in TS 26.247, but clause 11.4.3 of TS 26.512 fills this gap. If you can't get hold of the UE's GPSI via an Android API, I suggest using a persistent UUID, created by the Media Session Handler when it is first run on a new client, and remains the same until reset.
- I would not declare any xsi:schemaLocation in the root element: it's the document recipient's task to map the declared schema name space to the correct local XSD file to perform syntax validation.

dsilhavy · 2023-06-19T16:39:10Z

Thanks @rjb1000 the example above was limited to the relevant information for BufferLevel. I will collect several more metrics before wrapping everything in the right root element(s) and adding the missing attributes. Your comments provide a good overview of what would be required.

dsilhavy · 2023-06-19T17:43:15Z

Added basic support for RepresentationSwitchList. Note that this is event-driven. I am currently saving all the RepSwitchEvent entries as an ArrayList in the ExoPlayerListener. This list is available via the ExoPlayerAdapter. At this point, the list is cleared once a new streaming session is started. What is currently missing is the logic to only report "new" RepresentationSwitch events. For that reason, we have some options:

Include a timestamp in the request from the Media Session Handler and filter all entries that are older than this timestamp
Clear the list of events once it has been requested by the Media Session Handler
Filter/Remove non-required data in the Media Session Handler

I prefer option 1) The Media Session Handler knows when the last metric request to the media player was issued. By sending this timestamp, we can filter all old values.

Moreover, we cannot simply omit values for Android data classes and defaulting to null causes an error with the XML conversion. As a workaround, I used -1 for lto in the XML below. This is not ideal, and we should remove invalid/unused values when sending the final XML string.

For means of completion, an example how the reporting looks. Note this is not completed at this point, see @rjb1000 and my comments above.

<QoeReport>
   <QoeMetric class="java.util.ArrayList">
      <object class="com.fivegmag.a5gmscommonlibrary.qoeMetricsModels.threeGPP.BufferLevel">
         <BufferLevel>
            <BufferLevelEntry level="3928" t="1687196339408"/>
         </BufferLevel>
      </object>
      <object class="com.fivegmag.a5gmscommonlibrary.qoeMetricsModels.threeGPP.RepresentationSwitchList">
         <RepSwitchList>
            <RepSwitchEvent lto="-1" mt="268000" t="1687196338309" to="V300"/>
            <RepSwitchEvent lto="-1" mt="268000" t="1687196338717" to="A48"/>
         </RepSwitchList>
      </object>
   </QoeMetric>
</QoeReport>

dsilhavy · 2023-07-10T17:01:34Z

Current status incorporating @rjb1000 comments in #35

<?xml version="1.0"?>
<ReceptionReport xmlns:sv="urn:3gpp:metadata:2016:PSS:schemaVersion"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    contentURI="https://livesim.dashif.org/livesim/testpic_2s/Manifest.mpd"
    xmlns="urn:3gpp:metadata:2011:HSD:receptionreport"
    xsi:schemaLocation="urn:3gpp:metadata:2011:HSD:receptionreport DASH-QoE-Report.xsd">
    <QoeReport periodID="p0" reportPeriod="10000" reportTime="2023-07-10T16:55:32.106Z">
        <QoeMetric>
            <BufferLevel>
                <BufferLevelEntry level="0" t="2023-07-10T16:55:28.130Z" />
                <BufferLevelEntry level="0" t="2023-07-10T16:55:29.898Z" />
                <BufferLevelEntry level="0" t="2023-07-10T16:55:30.901Z" />
                <BufferLevelEntry level="0" t="2023-07-10T16:55:31.301Z" />
                <BufferLevelEntry level="0" t="2023-07-10T16:55:31.343Z" />
                <BufferLevelEntry level="2000" t="2023-07-10T16:55:31.667Z" />
                <BufferLevelEntry level="2000" t="2023-07-10T16:55:31.710Z" />
                <BufferLevelEntry level="3965" t="2023-07-10T16:55:32.053Z" />
                <BufferLevelEntry level="3921" t="2023-07-10T16:55:32.089Z" />
            </BufferLevel>
        </QoeMetric>
        <QoeMetric>
            <HttpList>
                <HttpListEntry
                    actualurl="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/Manifest.mpd"
                    interval="1731" range="" responsecode="200" trequest="2023-07-10T16:55:28.139Z"
                    tresponse="2023-07-10T16:55:28.153Z" type="MPD"
                    url="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/Manifest.mpd">
                    <Trace b="1955" d="1731" s="2023-07-10T16:55:28.153Z" />
                </HttpListEntry>
                <HttpListEntry
                    actualurl="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/V300/init.mp4"
                    interval="200" range="" responsecode="200" trequest="2023-07-10T16:55:30.681Z"
                    tresponse="2023-07-10T16:55:30.691Z" type="InitializationSegment"
                    url="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/V300/init.mp4">
                    <Trace b="715" d="200" s="2023-07-10T16:55:30.691Z" />
                </HttpListEntry>
                <HttpListEntry
                    actualurl="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/V300/844504049.m4s"
                    interval="332" range="" responsecode="200" trequest="2023-07-10T16:55:30.950Z"
                    tresponse="2023-07-10T16:55:30.959Z" type="MediaSegment"
                    url="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/V300/844504049.m4s">
                    <Trace b="38709" d="332" s="2023-07-10T16:55:30.959Z" />
                </HttpListEntry>
                <HttpListEntry
                    actualurl="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/A48/init.mp4"
                    interval="608" range="" responsecode="200" trequest="2023-07-10T16:55:30.717Z"
                    tresponse="2023-07-10T16:55:30.727Z" type="InitializationSegment"
                    url="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/A48/init.mp4">
                    <Trace b="651" d="608" s="2023-07-10T16:55:30.727Z" />
                </HttpListEntry>
                <HttpListEntry
                    actualurl="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/A48/844504049.m4s"
                    interval="343" range="" responsecode="200" trequest="2023-07-10T16:55:31.303Z"
                    tresponse="2023-07-10T16:55:31.314Z" type="MediaSegment"
                    url="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/A48/844504049.m4s">
                    <Trace b="13449" d="343" s="2023-07-10T16:55:31.314Z" />
                </HttpListEntry>
                <HttpListEntry
                    actualurl="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/V300/844504050.m4s"
                    interval="358" range="" responsecode="200" trequest="2023-07-10T16:55:31.335Z"
                    tresponse="2023-07-10T16:55:31.345Z" type="MediaSegment"
                    url="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/V300/844504050.m4s">
                    <Trace b="39941" d="358" s="2023-07-10T16:55:31.345Z" />
                </HttpListEntry>
                <HttpListEntry
                    actualurl="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/A48/844504050.m4s"
                    interval="289" range="" responsecode="200" trequest="2023-07-10T16:55:31.748Z"
                    tresponse="2023-07-10T16:55:31.756Z" type="MediaSegment"
                    url="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/A48/844504050.m4s">
                    <Trace b="13094" d="289" s="2023-07-10T16:55:31.756Z" />
                </HttpListEntry>
                <HttpListEntry
                    actualurl="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/V300/844504051.m4s"
                    interval="301" range="" responsecode="200" trequest="2023-07-10T16:55:31.773Z"
                    tresponse="2023-07-10T16:55:31.780Z" type="MediaSegment"
                    url="https://livesim.dashif.org/livesim/sts_1689008128/sid_60623bfd/testpic_2s/V300/844504051.m4s">
                    <Trace b="41458" d="301" s="2023-07-10T16:55:31.780Z" />
                </HttpListEntry>
            </HttpList>
        </QoeMetric>
        <QoeMetric>
            <RepSwitchList>
                <RepSwitchEvent mt="P0Y0DT0H4M28S" t="2023-07-10T16:55:30.922Z" to="V300" />
                <RepSwitchEvent mt="P0Y0DT0H4M28S" t="2023-07-10T16:55:31.363Z" to="A48" />
            </RepSwitchList>
        </QoeMetric>
        <sv:delimiter>0</sv:delimiter>
    </QoeReport>
</ReceptionReport>

I checked against the provided xsd with an online validator. @rjb1000 Can I please ask you to run the above XML in your tool as well

rjb1000 · 2023-07-10T17:23:54Z

I checked against the provided xsd with an online validator. @rjb1000 Can I please ask you to run the above XML in your tool as well

Validates successfully :-)

The only slight oddity is the lack of a UTF-8 encoding declaration in the first line. (That might be an option in the library you are using to generate the XML.)

And the question about whether <HttpList> is needed at all in a QoE metrics report for DASH. (I am not sure, but this seems to overlap with 5GMS consumption reporting to some extent.)

dsilhavy · 2023-07-14T14:48:48Z

@rjb1000 I forgot this one today in the call. I am generating the <?xml version="1.0"?> manually. I will change it to <?xml version="1.0" encoding="utf-8"?> and check if it can be generated by the library instead of appending it manually.

dsilhavy · 2023-07-20T06:00:25Z

Apparently the library is not generating this by itself so I changed the string to "<?xml version="1.0" encoding="utf-8"?>

dsilhavy · 2024-04-04T14:29:23Z

Implemented in #67

dsilhavy added the enhancement New feature or request label Jun 1, 2023

dsilhavy added this to the Version 1.1.0 milestone Jun 1, 2023

dsilhavy self-assigned this Jun 1, 2023

rjb1000 changed the title ~~Metric Reporting - Investigate and derive metrics via PlaybackStatsListener~~ QoE Metrics Reporting - Investigate and derive metrics via PlaybackStatsListener Jun 1, 2023

dsilhavy mentioned this issue Jul 10, 2023

QoE Metrics Reporting - Support for HttpList metric #35

Closed

dsilhavy modified the milestones: Version 1.1.0, Version 1.2.0 Dec 6, 2023

dsilhavy closed this as completed Apr 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QoE Metrics Reporting - Investigate and derive metrics via PlaybackStatsListener #27

QoE Metrics Reporting - Investigate and derive metrics via PlaybackStatsListener #27

dsilhavy commented Jun 1, 2023 •

edited

Loading

rjb1000 commented Jun 1, 2023

dsilhavy commented Jun 19, 2023 •

edited

Loading

dsilhavy commented Jun 19, 2023 •

edited

Loading

rjb1000 commented Jun 19, 2023 •

edited

Loading

dsilhavy commented Jun 19, 2023

dsilhavy commented Jun 19, 2023

dsilhavy commented Jul 10, 2023

rjb1000 commented Jul 10, 2023

dsilhavy commented Jul 14, 2023

dsilhavy commented Jul 20, 2023

dsilhavy commented Apr 4, 2024

QoE Metrics Reporting - Investigate and derive metrics via PlaybackStatsListener #27

QoE Metrics Reporting - Investigate and derive metrics via PlaybackStatsListener #27

Comments

dsilhavy commented Jun 1, 2023 • edited Loading

rjb1000 commented Jun 1, 2023

dsilhavy commented Jun 19, 2023 • edited Loading

dsilhavy commented Jun 19, 2023 • edited Loading

rjb1000 commented Jun 19, 2023 • edited Loading

dsilhavy commented Jun 19, 2023

dsilhavy commented Jun 19, 2023

dsilhavy commented Jul 10, 2023

rjb1000 commented Jul 10, 2023

dsilhavy commented Jul 14, 2023

dsilhavy commented Jul 20, 2023

dsilhavy commented Apr 4, 2024

dsilhavy commented Jun 1, 2023 •

edited

Loading

dsilhavy commented Jun 19, 2023 •

edited

Loading

dsilhavy commented Jun 19, 2023 •

edited

Loading

rjb1000 commented Jun 19, 2023 •

edited

Loading