Troubleshooting
The trouble shooting page helps to investigate stream or user specific metrics across the streaming experience from stream ingest to the playout at the enduser. With this feature you have a complete transparent view at your streams, which helps you to find potential problems and be more capable of solving them faster.
To begin, please sign in to the Analytics Dashboard using your nanoStream Cloud/Bintu account credentials.
If you have not created an account yet, you can sign up or reach out to our dedicated sales team via the contact form or by sending an email to sales(at)nanocosmos.de.
Access
Access to the Trouble Shooting page is available for every nanoStream Cloud customer.
Displaying Stream Playout information will require a business upgrade to sent your H5Live Player metrics to our Analytics backend. Please contact sales(at)nanocosmos.de for business details.
Time Picker
The time picker declares the time period in which you will query for available metric results. Use to " Set to Now" button the to automatically update to the current time.
Date and time values are interpreted as UTC, an offset to your local timezone needs to be considered.
Screenshot: Time Picker
Playouts
After selecting a specific time period you are asked to enter an IP address or an user ID to access playout statistics about all streams that this user retrieved. You will then be able to select one out of the 5 newest playouts based on their start date. To view more than these 5 results you need to adjust the time range accordingly. We also implemented support for currently running playouts.
Only organizations with player metrics option enabled will be able to see the items in this data group.
Screenshot: Playout Time Picker
(A) From (UTC Time)
the start of the time range to search in.
(B) To (UTC Time)
the end of the time range to search in.
(C) By
indicates the time interval granularity. It can be 30 seconds or 1 minute.
(D) Playout
tab which selects to view playout statistics. Playout statistics for all streams in the selected time period are retrieved.
(E) End to End
tab which selects to view both playout and ingest statistics in the same page. Playout and ingest statistics for all streams in the selected time period are retrieved.
(F) Ingest
tab which selects to view ingest statistics. Ingest statistics for all streams in the selected time period are retrieved.
(G) Copy URL to share
will copy the current page setup to clipboard. Users can send this URL to nanocosmos support for further troubleshooting help.
(H) IP address/user ID
specifies either the IP address used in client playouts or user IDs to search for related streams.
To make this work with user IDs, you have to assign IDs to your users beforehand and also transmit that to the player API.
(I) Select a stream
shows all streams that has a playout related to the IP address or user ID set earlier.
Stream Playout Information
If there are metrics available for the selected IP/ user ID within the stated time period, a list of streams viewed by this user is shown. Select the desired stream name to see a list of all playouts for this stream. After you selected a playout more detailed information are shown. Define the time interval with the slider to zoom into streams with a longer duration or to focus on specific parts of the stream. At this point you can synchronize with the corresponding ingest data of the playout on the right side.
Screenshot: Playout Stream Information
(A) Playouts
of selected stream name.
(B)
Start
indicates the start of playout.End
indicates the end of playout.Duration
indicates playback duration.
(C) Groups of meta-data information are shown related to the stream.
The meta-data includes:
User ID
- user ID.IP
- client IP address used for playout.Event ID
- event ID.Custom field 1
- custom field1.Playout ID
- internally generated unique ID.H5Live player version
- version of nanoplayer used.Device
- type of device used.Referrer
- webpage that hosts the nanoPlayer.Country
- location of user.City
- location of user.Resolution width
- width of video stream in pixels.Resolution height
- height of video stream in pixels.OS
- OS used.OS version
- OS version.Browser
- web browser used.Browser version
- browser version.
(D) This tab shows which group of meta-data information is being shown at the moment and how many groups are available.
(E) (F) Click to switch between meta-data information.
(G) This time slider can be used to define the time interval, allowing to zoom into streams with a longer duration or to focus on specific parts of the stream.
(H)
Synchronize ingest
gives the ability to synchronize the currently selected playout with the corresponding ingest data group. In this selection, the playout data group is shown on the left side while the ingest data group is shown on the right side. This is same as selecting the End to End data group.
If an user reloads the page of a stream, it will be accounted as a new playout.
If you set the visible time range to under 5 minutes, you switch from data aggregation mode into raw data mode. Therefore you will get exact datapoints at the given time.
Screenshot: Troubleshooting Raw Data Mode
Visualizations
This area shows charts for various player metrics.
Screenshot: Playout Media Information
Bitrate
[kBits/s]:
The retrieved bitrate on the client side at a specific time. Purple lines symbolize events which are also listed in the Events table.
Latency
[s]:
The latency from ingest to playout. Purple lines symbolize events which are also listed in the Events table.
Buffering Totaltime Ratio
[%]:
The ratio between buffer and playback duration.
A ratio of 0% means no buffering and a ratio of 100% means no playback was possible.
Events
:
All events that occurred in the selected playout.
Ingests
Publishes
The Ingest/Publishes tab selects the ingest data group with all items described in detail next. By either synchronizing the ingest stream from a corresponding playout or entering a valid stream name/tag manually, you will get access to ingest specific data, depending on the used protocol WebRTC or RTMP. You can see which protocol was used in the meta-information section after selecting a publish. Currently running ingests are also supported in this overview and are indicated by (ongoing)
status within the End
column.
Screenshot: Publish Table Pre- & Post- Publish Selection
(A) Publish search input field
to enter stream name or stream tag
(B) Publish table header
Start
indicates the ingest start time.End
indicates the ingest end time.Duration
indicates ingest duration.Has drops
indicates if the ingest was interrupted.
(C) General stream details
show meta-data information related to the stream
Protocol
- streaming protocol used.Data center
- specific data center used for this ingest.Server
- specific server in the data center used.IP
- client IP from which the stream was ingested.Country
- location of client IP.City
- location of client IP.Resolution width
- width of video stream in pixels.Resolution height
- height of video stream in pixels.webcaster ID
- ID of corresponding webcaster instance.
(D) These dot indicators show which group of meta-data information is displayed at the moment and how many groups are available.
(E) Click to switch between meta-data information.
(F) Ingest tabs
- Choose between alerts & publishes to search for
(G) Fine interval slider
can be used to define the time interval, allowing to zoom into streams with a longer duration or to focus on specific parts of the stream.
Stream Time Ratio
The stream time ratio (STR) represents the relation between the elapsed real time and the stream time within a window of 15 seconds. A constant ratio value of 1 would be ideal - but a small fluctuation is expected. A big fluctuation or deviation from the value 1 indicates issues on the ingest side, for instance caused by a limited bandwidth or high CPU load. The higher the deviation from the value 1, the higher the probability for an increased latency on the playback/viewer side.
Visualizations
Define the time interval with the slider to zoom into streams with a longer duration or to focus on specific parts of the stream. The results you get for the ingest stream name are also sorted by newest first as for the playouts. Selecting a time period auto-refreshes the statistics below.
RTMP Ingests
Ingests using RTMP only provide metrics about video and audio bitrates.
Screenshot: Troubleshoot Ingest RTMP
Video bitrate
[kBits/s]: Video bitrate of the ingested stream.Audio bitrate
[kBits/s]: Audio bitrate of the ingested stream.
WebRTC Ingests
Ingests using Web Real Time Communication are providing additional insights to framerate, encoding and package delivery.
To have WebRTC metric data available for the Troubleshooting page, they need to be enabled.
Screenshot: Troubleshoot Ingest Webcaster
Video bitrate
[kBits/s]: Video bitrate of the ingested stream.Audio bitrate
[kBits/s]: Audio bitrate of the ingested stream.Framerate
[fps]: Framerate of the ingested stream.Average frame encode time
[ms]: The average time needed to encode one frame of the stream.
Note that a long encode time is an indicator for high encoder workload, which can lead to ingest problems.
Count of packet loss detections
: Amount of WebRTC notifications about lost data packages.
Note, a high count of lost packages at a time can lead to issues with the video representation on the side of the end user.
Other Samples
Video only ingest workflow
The figure below shows the graphical representation. In this scenario, only video bitrate data can be observed, while audio bitrate is not available. Also, the detected video bitrate is about 1 Mbps and this can be checked against settings on ingest encoder.
Screenshot: Video Only
Audio only ingest workflow
The figure below shows the graphical representation. In this scenario, only audio bitrate data can be observed, while video bitrate is not available. Similarly, the detected audio bitrate is averaging above 25Kbps and this can be checked against settings on ingest encoder.
Screenshot: Audio Only Ingest Workflow
Audio/Video ingest workflow in good condition
The figure below shows the graphical representation. As can be observed, the average video bitrate is about 1.2Mbps while audio bitrate is about 90Kbps. The video and audio bitrates do not deviate too much from the average over time and will not cause overflow or underflow of buffers.
Screenshot Audio/Video Ingest Workflow in good condition
Audio/Video ingest workflow with network issues
The figure below shows the graphical representation. As can be observed, the average video bitrate is about 2Mbps while audio bitrate is about 150Kbps. Towards the end of the graph as highlighted by the red box, the video and audio bitrates fluctuate by more than 50% of the average. This signifies some issues with the ingest from the encoder to nanocosmos CDN. Some causes of these could be ISP related network degradation or encoding anomalies. If this issue is not rectified quickly, then it will cause overflow or underflow of buffers resulting in buffering, connection issues, etc.
Screenshot: Audio/Video Ingest workflow with network issues
Alerts
Via Ingests/Alerts tab you can examine alerted streams and make potential issues visible, identify their causes and get helpful recommendations to solve them. After entering a stream name you able to search for any alert occurences in the past 4 weeks. In the following table you can select any alert to get the basic stream details right away.
Screenshot: Alert Table Pre- & Post- Alert Selection
(A) Alert search input field
to enter stream name
(B) Alert table header
Alert ID
- Each alert gets a unique alert IDSeverity
- Defines the impact level of the underlying issueStatus
- The current alert statusAlert type
- The individual alert typeAlert codes
- Defines the exact alert kindStarted at
- The point in time when the alert was emittedLast seen at
- The last minute when the alert was activeDuration
- The current alert duration from emitting to last seen occurrence
(C) General stream details
Ingest IP
- The IPv4 address that ingested to the stream nameServer
- The server that got the ingested dataProvider
- The corresponding regional ISPCountry
- The country that was ingested fromCity
- The city that was ingested fromVideo Width
- The video resolution widthVideo Height
- The video resolution height
(D) These dot indicators show which group of meta-data information is displayed at the moment and how many groups are available.
(E) Click to switch between meta-data information.
(F) Ingest tabs
- Choose between alerts & publishes to search for
(G) Alerts per page settings
(H) Page navigation
Alerts of Type - Stream Time Ratio Issue
If a performance alert is selected in the alert table, which is less than 2 weeks old, the corresponding performance widget is shown below the alert table, providing crucial insights of the stream time ratio (STR) metric. Depending on how good/bad the values are this widget will show a suboptimal or poor performance. There is an example of suboptimal performance on the left and poor performance on the right below.
Screenshot: Alert Widget STR
(A) Fine Interval Slider
- Is used to fine tune the selected time interval within the alert duration (max 2 hours of displayed data at once)
(B) CSV data export button
- By clicking you can automatically download a .csv
file, containing the currently displayed data
(C) Suboptimal STR Occurences
- Shows the number of data points that are between the orange and red horizontal lines (representing the thresholds)
(D) Poor STR Occurences
- Shows the number of data points that are below the red horizontal line (representing the threshold)
(E) Quick Info
- Short conclusion of the underlying issue
(F) Detection Start
- The beginning point in time that counted for alert emitting later
(G) Emitted Alert
- The point in time when the selected alert has been raised initially
(H) Recovered/Stream Ended
- The point in time when the stream performance recovered from the issue by itself or when the stream ended (currently active alerts do not have this indicator)
(I) Alert Details Panel
- Containing explanantions and further details of this alert and also provides a recommended action to potentially solve the issue
(J) Link to Documentation
- References more knowledge about alerts and how to deal with them
Alerts of Type - Continuous Restarts
In case a Continuous Restarts alert is selected in the alert table, this widget will be shown. It shows the analyzed information that lead to this alert notification.
Screenshot: Alert Widget CR
(A) CSV data export button
- By clicking you can automatically download a .csv
file, containing the currently displayed data
(B) Quick Info
- Short conclusion of the underlying issue
(C) Restart Count
- The counted restarts within the alert's duration
(D) Analyzed Events
- The amount of events that were analyzed within the alert's duration
(E) Min Distance
- The minimum distance between 2 restarts in seconds
(F) Max Distance
- The maximum distance between 2 restarts in seconds
(G) Average Distance
- The average distance between 2 restarts in seconds
(H) Alert Details Panel
- Containing explanantions and further details of this alert and also provides a recommended action to potentially solve the issue
(I) Link to Documentation
- References more knowledge about alerts and how to deal with them
Alerts of Type - Duplicated Ingests
Duplicated Ingests alert details are shown within the duplicated ingests widget when such an alert type is selected above. Depending on the determined reason this widget will provide information about different locations, IPv4 addresses, or infrastructures. In the following example multiple ingest connections were found for different locations at the same time for the chosen stream.
Screenshot: Alert Widget DI
(A) Quick Info
- Short conclusion of the underlying issue
(B) Locations
- List of found locations
(C) Alert Details Panel
- Containing explanantions and further details of this alert and also provides a recommended action to potentially solve the issue
(D) Link to Documentation
- References more knowledge about alerts and how to deal with them