Skip to content

Commit

Permalink
Merge pull request #6997 from jddocs/rc-v1.331.0
Browse files Browse the repository at this point in the history
[Release Candidate] v1.331.0
  • Loading branch information
jddocs authored Jun 11, 2024
2 parents f8cd28e + 8cff8e8 commit c94cd94
Show file tree
Hide file tree
Showing 217 changed files with 446 additions and 4,458 deletions.
1 change: 1 addition & 0 deletions ci/vale/dictionary.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1013,6 +1013,7 @@ hulu
husch
hvc0
hybla
Hydrolix
hykes
hypercorn
hyperefficient
Expand Down
4 changes: 4 additions & 0 deletions ci/vale/styles/Linode/SubstitutionWarning.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,7 @@ swap:
info: information
repo: repository
whitelist(?:ed|ing)?: allowlist
"[Ll]ogin to": log in to
"[Ll]og into": log in to
"[Ll]ogged into": logged in to
"[Ll]ogging into": logging in to
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
---
slug: live-transcoding-ugc-streaming-akamai-cloud-computing
title: "Live Transcoding for UGC Streaming on Akamai Cloud Computing"
description: "This guide outlines design requirements for a live streaming transcoding workflow for a UGC live streaming platform on Akamai Cloud Computing."
authors: ["Linode"]
contributors: ["Linode"]
tags: ["video transcoding", "live transcoding", "ugc live streaming platform"]
published: 2024-06-11
keywords: ['live streaming','live streaming transcoding']
license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)'
---

Live streaming is a key feature of many popular internet services, including social networking, video conferencing, gaming, and sports broadcasting. These services rely on live transcoding of video streams to efficiently distribute content in formats that are suited to the network and device constraints where they are viewed. Video transcoding is a compute-intensive process, so maximizing the number of video streams that can be transcoded on available hardware is a primary consideration.

This transcoding efficiency can vary between the compute offerings of different infrastructure providers, and evaluations of transcoding performance should be performed when selecting a cloud infrastructure platform. Many live streaming services are also latency-sensitive, and the geographic location of the transcoding service affects the latency of the stream. Choosing a location closer to the stream’s viewers reduces latency, so being able to run the service in compute regions that are near your audience is important.

This guide covers a live transcoding architecture for a live streaming platform. This architecture has been implemented and proven by a profiled Akamai customer that operates a popular live streaming platform for user generated content (UGC) with a global audience. This customer previously utilized on-premise transcoding services, but they experienced constraints as their traffic grew in new geographies. To support the growth, a new cloud-based live transcoding service was set up alongside their existing on-premise solution. The cloud live transcoding service offered competitive transcoding efficiency, was installed in geographically-optimal regions, and significantly reduced their total egress fees.

## Live Streaming Transcoding Workflow

1. The live streaming platform user uploads a video stream to the platform’s live origin service.

1. The live origin service directs the video stream to a live transcoding service.

1. The live transcoding service transcodes the stream into desired video formats.

1. A content delivery network accepts the transcoded video and distributes it to platform audiences.

## Overcoming Challenges

### Cost sensitivity

*Reduce costs with transcoding efficiency, scalable infrastructure, and by eliminating egress fees.*

Because video transcoding is a compute-intensive process, compute resources are a primary source of infrastructure cost for live streaming services. It’s important to select compute hardware that is performant for the software that is run by the live transcoding service. It’s also important to test example transcoding workflows on competing cloud infrastructure platforms and measure the transcoding efficiency of each. For example, this can be done by selecting cost-comparable compute instances between platforms and measuring the number of parallel streams that each can transcode on their respective instances. In testing with Akamai’s compute offerings, the live streaming platform described by this reference architecture saw a 33% transcoding efficiency advantage over another hyperscaler that was tested.

Live streaming traffic often flows in unpredictable bursts, and the practice of reserving compute instances in advance doesn’t provide cost advantages for these types of transcoding workloads. Instead, a scaling mechanism for the compute instances that make up a live transcoding service can be used to accommodate traffic bursts.

After a video stream is transcoded by the live transcoding service, it needs to be distributed to a content delivery network. This can also be a significant source of cost when there are egress fees between the live transcoding service platform and the content delivery network. By selecting Akamai compute offerings for the live transcoding service and using Akamai’s CDN, the egress fees for that traffic can be reduced by 100%.

### Latency sensitivity

*Minimize latency with cloud infrastructure close to your customers.*

Low latency is critical for live streaming services. For video conferencing, low latency helps emulate real-time conversation. For sports broadcasting, it can convey important events as they happen. For UGC platforms, low latency can help drive user engagement as users interact with and respond to their audiences.

To enable low latency, live transcoding services should be located near their audiences. By working with a cloud infrastructure platform that offers a wide selection of regions in different geographies, you can ensure the proximity of your live transcoding service as your business expands into new areas. Akamai’s global footprint of compute regions supports expansion into new audiences.

## Live Streaming Transcoding Design Diagram

This solution creates a live video transcoding service on the Akamai cloud computing platform, while retaining an existing on-premise live origin service and on-premise live transcoding service. A load balancer at the on-premise live origin directs traffic between the on-premise transcoding service and the new cloud transcoding service according to the audience for the video stream. The cloud transcoding service is composed of multiple compute instances and block storage volumes working in parallel to handle the transcoding load. Transcoded video streams are distributed by the Akamai CDN to audiences.

1. The platform ingests video streams from US-based platform users. These streams are ingested into an on-premise live origin service.

1. The live origin service directs streams to a live transcoding service. The streams are directed according to the region of the audience for the stream. For US audiences, the streams are sent to an on-premise live transcoding service. For North American, non-US audiences, the streams are directed to the cloud live transcoding service.

1. The cloud live transcoding service ingests video streams from the live origin.

1. Origin video streams are transcoded by compute instances in the transcoding cluster into desired output formats. Block storage volumes attached to each compute instance store temporary files created during the transcoding process. Live streaming traffic flows can sometimes increase in unpredictable bursts, so a scaling mechanism for the number of available compute instances can be set up.

1. Transcoded video streams are uploaded to object storage. This object storage acts as the content origin for the live streaming delivery network.

1. A content delivery network distributes video streams from the object storage content origin to North American, non-US audiences.

![Live Streaming Transcoding Design Diagram](live-streaming-transcoding-design-diagram.jpg "A design diagram for a live streaming transcoding service that ingests streams from publishers and outputs transcoded streams to a CDN")

### Systems and Components

- **On-premise live origin**
- Accepts video streams from platform users and directs them via an on-premise load balancer to a transcoding service. These streams are directed according to the audience’s geographic location:
- Geographic location 1: The platform’s on-premise transcoding service.
- Geographic location 2: The cloud live transcoding service.
- **Cloud live transcoding service**
- **Live transcoding cluster**
- **Live transcoding compute instance:** Accepts a video stream and transcodes it into a desired format for distribution.
- **Block storage:** A block storage volume is attached to each transcoding instance for the temporary storage of video files that are being processed.
- **Transcoding output storage/distribution origin:** The live transcoding instances upload transcoded video to object storage. This object storage location serves as the content origin for the live streaming delivery network.
- **Distribution:** A content delivery network retrieves transcoded video from the object storage distribution origin and delivers it to audiences.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
slug: observability-with-datastream-and-trafficpeak
title: "Large Data Observability With DataStream and Hydrolix TrafficPeak"
description: "This guide reviews Akamai's managed observability solution, Hydrolix TrafficPeak, including product features, how TrafficPeak overcomes observability challenges, and a proven implementation architecture."
authors: ["John Dutton"]
contributors: ["John Dutton"]
published: 2024-06-11
keywords: ['observability','datastream','trafficpeak','hydrolix','logging','data logging','visualization']
license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)'
external_resources:
- '[Official Hydrolix Documentation](https://docs.hydrolix.io/docs/welcome)'
- '[Official TrafficPeak Site](https://hydrolix.io/partner-program/trafficpeak/)'
---

## Overview

Observability workflows are critical to gaining meaningful insight to your application’s health, customer traffic, and overall performance. However, there are challenges that come along with achieving true observability, including large volumes of traffic data, data retention, time to implementation, and the cost of each.

Hydrolix TrafficPeak is a ready-to-use, quickly deployable observability solution built for Akamai Cloud. TrafficPeak works with DataStream to ingest, index, compress, store, and search high-volume, real-time log data at up to 75% less cost than other observability platforms. TrafficPeak customers are provided with a Grafana login and customized dashboards where they can visualize, search, and set up alerting for their data.

This guide looks at the TrafficPeak observability solution and reviews a tested, proven observability architecture built for a high-traffic delivery platform. This solution combines Akamai’s edge-based DataStream log streaming service, SIEM integration, and TrafficPeak built on Linode cloud infrastructure to support large-scale traffic, logging, and data retention.

## TrafficPeak On Akamai Cloud

### What Is TrafficPeak?

TrafficPeak is a fully managed observability solution that works with DataStream log streaming and Akamai Cloud Computing. TrafficPeak is managed and hosted by Hydrolix, and uses Linode Compute Instances alongside Linode Object Storage for data processing and storage. With TrafficPeak, customers are provided with access to a Grafana interface with preconfigured, customizable dashboards for data visualization and monitoring.

### Who Is TrafficPeak For?

TrafficPeak is for Akamai customers that need an all-in-one, cost-effective, turnkey observability solution for large, petabyte-scale volumes of data.

For more detailed information on features and pricing, see: [Observability on Akamai cloud computing: TrafficPeak](https://hydrolix.io/partner-program/trafficpeak/)

## Overcoming Challenges

### Cost Reduction & Visibility

*Reduce observability costs by up to 75% while standardizing data visualization.*

Where other observability solutions are built for application-specific observability, TrafficPeak is built to achieve CDN-level observability supporting extremely large amounts of data. TrafficPeak is designed to work with DataStream to ingest logs from the edge and uses Compute Instances with Object Storage to achieve an efficient, decoupled, and stateless architecture. By keeping traffic and CDN logs on the same platform, TrafficPeak can reduce your observability costs by up to 75%.

With large amounts of data, the need for standardizing data reporting becomes even more important. TrafficPeak’s Grafana dashboard supports as many users you may need, providing a centralized data source. This helps streamline organizational operations and allows developers that work with the data the ability to all get it from the same place.

### Large Data Volumes & Data Retention

*Address data volume and retention challenges with an observability solution on the same platform as your edge delivery.*

There are numerous cost and infrastructure-based challenges that are associated with large volumes of data, including: data acquisition, data storage and retention, data searching and efficiency, and more. Since TrafficPeak is built for Akamai, you can address these challenges by combining your edge and observability solutions on a single platform.

With TrafficPeak, logs are sent directly from Akamai edge to Linode Compute using DataStream, eliminating the need for data traffic to travel outside the Akamai platform. TrafficPeak separates log processing from log storage by utilizing Compute Instances and Object Storage in tandem, uses up to 25x data compression, and offers searchable, accessible hot data retention for 15+ months.

### Complex Data Types

*Achieve observability for complex types of data with visual monitoring and data reporting.*

Complex data (for example, media delivery and gaming data) can have additional challenges: extreme sensitivity to latency, high data volumes, audience insights, application-specific data types, data compliance and security, and more. TrafficPeak’s visual monitoring and data reporting allows you to track audience size, unique viewership, SIEM data, and other audience-specific data. TrafficPeak is also monitored by Hydrolix (so you don’t need to worry about scaling tasks), includes configurable alerting, and supports CMCD when using Akamai’s [Adaptive Media Delivery](https://www.akamai.com/products/adaptive-media-delivery).

### Implementation

It can be both time consuming and costly when implementing an observability solution to support large amounts of data. TrafficPeak addresses this by being ready-to-use out of the box for Akamai customers and by working with your existing application infrastructure.

## TrafficPeak Workflow Diagram

Below is a high-level diagram and walkthrough of a DataStream and TrafficPeak architecture implemented by a high-traffic delivery platform customer on Akamai.

1. The client makes a request for an asset, an API call, or other HTTPs request. That request is routed to the nearest Akamai edge region.

1. If a cached response exists for the request, it is returned by the Akamai edge server. Otherwise, the request is sent to the origin for a response.

1. Delivery/security logs of this process are generated and sent to TrafficPeak on Akamai Cloud infrastructure (Linode Compute Instances for log processing and Linode Object Storage for log storage).

1. TrafficPeak collects the information, providing extended data retention (+15 months), facilitating historical data trends and reporting, and enhancing overall visibility.

1. A Grafana dashboard supporting hundreds of users provides a unified view for DataStream and SIEM functions.

![DataStream With TrafficPeak Diagram](DataStream-With-TrafficPeak-Diagram.png)

For an in-depth overview of Hydrolix’s architecture, see: [The Hydrolix Data Platform](https://docs.hydrolix.io/docs/platform-overview)

### Systems and Components

- **DataStream:** Akamai’s edge-native log streaming service.

- **Hydrolix TrafficPeak:** Akamai’s managed observability solution that runs on Akamai Cloud Computing platform. Comprised of Compute Instances, Object Storage, and a Grafana dashboard.

- **Edge Server:** The edge infrastructure that receives, processes, and serves client requests. In this workflow, edge server activity is logged and sent to TrafficPeak for observability purposes.

- **Data Analysis:** Grafana dashboard; a web-based analytics and visualization platform preconfigured for monitoring log activity processed by TrafficPeak. Configured and made accessible to TrafficPeak customers.

- **VMs:** Compute Instances used to run TrafficPeak’s log ingest and processing software. Managed by Hydrolix.

- **Object Storage:** S3 compatible object storage used to store log data from TrafficPeak. Managed by Hydrolix.
Original file line number Diff line number Diff line change
Expand Up @@ -490,7 +490,7 @@ Notice: Finished catalog run in 0.02 seconds

### Edit SSH Settings

Although a new limited user has successfully been added to the Puppet master, it is still possible to login to the system as root. To properly secure your system, root access should be disabled.
Although a new limited user has successfully been added to the Puppet master, it is still possible to log in to the system as root. To properly secure your system, root access should be disabled.

{{< note >}}
Because you are now logged in to the Puppet master as a limited user, you will need to execute commands and edit files with the user's sudo privileges.
Expand Down
2 changes: 1 addition & 1 deletion docs/products/compute/compute-instances/faqs/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ aliases: ['/beginners-guide/','/platform/linode-beginners-guide/','/platform/bil

If you're relatively new to Linux system administration, or just new to our platform, this guide will help address some of the most common questions we receive. If you've just created your first Linode account, please first refer to our [Creating a Compute Instance](/docs/products/compute/compute-instances/guides/create/) guide and return here once your Compute Instance has been deployed.

## How do I log into my Compute Instance?
## How do I log in to my Compute Instance?

All Compute Instances can be accessed through [Lish](/docs/products/compute/compute-instances/guides/lish/) and [SSH](/docs/guides/connect-to-server-over-ssh/) (if properly configured). Both methods provide you with command line access to your system. You can learn more about connecting to your Linode for the first time in the [connecting to your Linode with SSH](/docs/products/compute/compute-instances/guides/set-up-and-secure/#connect-to-the-instance) section of our [Setting Up and Securing a Compute Instance](/docs/products/compute/compute-instances/guides/set-up-and-secure/) guide.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,7 @@ If you would like to mount or unmount additional disks on your system, repeat th

### Change Root

*Changing root* is the process of changing your working root directory. When you change root (abbreviated as *chroot*) to your root disk, you are able to run commands as though you are logged into that system.
*Changing root* is the process of changing your working root directory. When you change root (abbreviated as *chroot*) to your root disk, you are able to run commands as though you are logged in to that system.

Chroot allows you to change user passwords, remove/install packages, and do other system maintenance and recovery tasks in your Compute Instance's normal Linux environment.
Expand Down
Loading

0 comments on commit c94cd94

Please sign in to comment.