As an example, a user is watching Netflix on his device, and the time-series data for Throughput could be (45 Mbps, 45 Mbps, 1 Mbps, 0 Mbps, 0 Mbps, 5 Mbps, 45 Mbps, 45 Mbps), and RTT (30 ms, 35 ms, 35 ms, 18 ms, 12 ms, 12 ms, 18 ms, 35 ms, 35 ms), and Loss (0.5%, 1%, 1%, 0.5%, 0.3%, 0.5%, 0.7%, 0.5%).

The user subjectively scores this experience as 4 out of 5. As an example, threshold that would have served well for this single user would be:

- Throughput during 5 or more of 8 samples >50 Mbps means a score of 5
- Throughput during 3 or more of 8 samples >30 Mbps means a score of 4
- Throughput during 2 or more of 8 samples <20 Mbps means a score of 3

Similar thresholds for RTT, Loss and Jitter may be determined in a similar manner. Embodiment so the system and method provide for logical rules for how to combine the metrics in a decision tree, or another manner using machine learning. In will be understood that a variety of machine learning models may be used.

As shown in FIG. 6, the system, via the logic module collects input data that has been received and stored in the lab setting, at 210. The Logic module or active logic module is configured to extract a feature set as noted herein, at 220. The Model module uses the feature set to train the model using reinforcement learning and/or supervised learning, at 230. The model module may now provide the model as an output to determine MOS on a live computer network, at 240.

In some cases, an application level buffering factor can also used. At the instance when user sees QoE as bad in Netflix, it is likely that the issue started earlier and has taken a period of time prior to being noticed by the user. This will give a period of time for which the data transfer may not have been good, but video might have continued to play at a reasonable level due to buffering. For this purpose, calculating and collecting a network QoE is also useful. The network QoE may have had a reduction prior to the user noticing bad QoE.

The processing is done by collecting the above parameters over a window along with timestamps. For example, for Netflix, the statistics can be collected in, for example, 1 second 5 seconds, 10 seconds or the like intervals. This is used to correlate the statistics with the timestamps for which that user has provided QoE. When using reinforcement learning, this allows to provide a credit or debit based on QoE score increase/decrease.

Embodiments of the system and method detailed herein are intended to provide for improved understanding of Key Performance Indicators and their relations with Application Content categories. Further, the system and method are intended to collect labelled data to use for training machine learning models and AppQoE bin definitions and methods. By collecting Pcaps, the system and method are able to provide a detailed analysis of the network traffic, classification of the network traffic and may use the Pcaps to evaluate new metrics to be added to the system in the future.

In building the model, the system and method are intended to aggregate the App QoE scores. The system may collect granular data, for example, App QoE scores may be published to a table along with a set of dimensions or parameters, for example, time, subscriber, application, application category, application content category, location, device, service plan and the like. The data is then intended to be aggregated by the system to determine a feature set. In some cases, the aggregation may be over time, for example a 5 minute aggregation, hourly aggregation, daily aggregation or the like. In other cases, the aggregation may be over a reduced dimension set, where like dimensions are aggregated into a more general dimension set.

Various methods may be used when aggregating the collected data. In one example a weighted average may be used, such as:

weighted
⁢

average

=

∑

n
=
1

N

⁢

score
n

·

weight
n

∑

n
=
1

N

⁢

weight
n

Where N is the number of records to aggregate, score_nis the score of the n'th record and weight_nis the weight of the n'th record.

In some cases, the App QoE score may be calculated prior to being stored in or exported to the Internal Data system. Each record that can be scored may have the score populated in the associated record. In order to aggregate scores from multiple records into a single representation, each App QoE score may be assigned a weight factor that is used when aggregating the App QoE scores. The weight factor will be set per record and by based on, for example the activity and/or duration of the record, the application category, the content category and the like. In some cases, the weights associated with the record holding each calculated score would be in the range 0-1, but as multiple records are aggregated the weight of the aggregated records could exceed that range. It will be understood that other ranges could be used.

In an example, a greater weight will be assigned for a longer duration. It has been determined the longer the duration of the experience, the more it contributes to the aggregated App QoE score. In a specific example: Subscriber X has been watching both Netflix and YouTube within the same 5 minute interval. The Netflix session lasted for 4 minutes and the YouTube session only for 1 minute. There would be two records for subscriber X in IDS. One for the Netflix traffic and one for the YouTube traffic. Let's now assume that the Netflix experience was scored as ‘5’ (A) and the YouTube experience was scored as ‘1’ (E). Since each record also has a weight that is relative to the duration/activity of the session this would look something like this table:

Application

Content
App QoE
App QoE

Subscriber
Application
Category
score
score weight