AWS installation: help in understanding architecture


#1

Hi

I’m planning to install the CE with the following setup

(i) platform packages on AWS EC2 Ubuntu 16.04 Xenial server. .
(ii) S3 storage for transcoded videos
(iii) Cloudfront CDN for distribution

Having gone through the package installation instructions I have the following questions regarding how the entire solution is going to work:

(i) Please correct any errors in my understanding - the content curators will be connecting to the Kaltura server running on the EC2 instance to upload videos. The Kaltura core packages will transcode them and store them in the S3 bucket via the CDN. End-users of the video will also connect to the Kaltura server on the EC2 instance, which gets the transcoded content from S3 via the CDN and delivers it to the end user using the video player in the applications package. My problem with this understanding is - why have the CDN at all if it only going to be an intermediary between the EC2 instance and S3? If I get rid of the CDN, then my EC2 server is going to be badly hit by the number of users!!

(ii) Alternative to (i) - the content curators will be connecting to the Kaltura server running on the EC2 instance to upload videos. The Kaltura core packages will transcode them and store them in the S3 bucket via the CDN. The end-users will get the transcoded content from S3 via the CDN. But in this case, how will end users get the benefit of the Kaltura video player in the applications layer on the EC2 instance?

Once those questions are cleared, I have the following questions regarding the implementation details of installing the platform. I understand that some of these questions are more about AWS than about Kaltura. Please bear with me regarding that.

(i) How does the Kaltura server on the EC2 instance connect with S3 to store the transcoded content? Is it simply through the opening of the FTP/TCP port on the EC2 instance and granting the EC2 instance access to the S3 bucket through a IAM account?

(ii) How does the Kaltura server know how to connect to the CDN? Is it through CDN hostname parameter specified during the installation of the Kaltura server?

(iii) Are the Nginx and RTMP modules mandatory to install if my use case does not include Live Streaming but only video on demand?

(iv) Should selinux be in permissive mode only? Will the Kaltura server not work if selinux is simply disabled?

(v) Is the Mail server mandatory to install for the kaltura server to work? We don’t need any email notifications in our use case.

(vi) If I manage to complete the installation, then what are the end points/url that are exposed? How do I start getting access to the AMC and start uploading videos?

Thanks


#2

Hi @vinod_kumar1,

Here is how it works: upon hitting “play”, the player will make an API request called playManifest to the Kaltura server [an EC2 instance in your case]. This, by default, will return an m3u8 manifest, for example:

{"entryId":"0_nhvgpnvi","duration":33,"baseUrl":"","flavors":[{"url":"https:\/\/test.kaltura.org:8443\/hls\/p\/102\/sp\/10200\/serveFlavor\/entryId\/0_nhvgpnvi\/v\/12\/flavorId\/0_6l6ee5y1\/name\/a.mp4\/index.m3u8","ext":"mp4","bitrate":851,"width":960,"height":540,"audioLanguage":null,"audioLanguageName":null,"audioLabel":null,"audioCodec":null,"defaultAudio":false,"frameRate":25},{"url":"https:\/\/test.kaltura.org:8443\/hls\/p\/102\/sp\/10200\/serveFlavor\/entryId\/0_nhvgpnvi\/v\/12\/flavorId\/0_rqill4mq\/name\/a.mp4\/index.m3u8","ext":"mp4","bitrate":1337,"width":1024,"height":576,"audioLanguage":null,"audioLanguageName":null,"audioLabel":null,"audioCodec":null,"defaultAudio":false,"frameRate":25},{"url":"https:\/\/test.kaltura.org:8443\/hls\/p\/102\/sp\/10200\/serveFlavor\/entryId\/0_nhvgpnvi\/v\/12\/flavorId\/0_t4t0hjvy\/name\/a.mp4\/index.m3u8","ext":"mp4","bitrate":1668,"width":1280,"height":720,"audioLanguage":null,"audioLanguageName":null,"audioLabel":null,"audioCodec":null,"defaultAudio":false,"frameRate":25}]}

As you can see, the manifest specifies the different flavours [metadata and the URI for each flavour].
When working against a CDN endpoint, the URI will lead to that [an S3 bucket in your case].
So, while the API request to fetch the HLS manifest will be made against your Kaltura EC2 instance, the content will be served from a CDN edge.

Two important notes:

  • When using the Kaltura platform for production purposes [as opposed to testing], you should never have a single Kaltura instance but rather, set up a clustered ENV
  • HLS is the default ABR [Adaptive BitRate] protocol we use but we support others, includeing HDS and DASH

I’m not sure I understand the question… the Kaltura player code is embedded on the HTML page served to the user. It mostly contains JS code though there’s a bit of PHP in the backend and of course, HTML and CSS as well. As mentioned in the paragraph above, the player makes a playmanifest API request and that returns the URIs for the relevant media resources hosted on the CDN edges. Thus, the actual media content will be served from the nearest CDN edge.

S3 does not work over FTP. The S3 API client is used. For a short introduction to S3’s RESTful API, see:
https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAPI.html

You may also find this doc beneficial:

Yes, the Nginx VOD module, as its name implies, also requires Nginx. We provide our own Nginx package called kaltura-nginx that already includes all the needed modules, configuration templates and postinst scripts.

It certainly would but it’s always best to set SELinux to permissive mode instead. When operating in permissive mode, you still get the auditing capabilities, which can be useful, without the enforcement of policy [which can and in Kaltura’s case WILL hinder critical operations]

If you absolutely do not require email notifications of any kind then you don’t have to deploy an MTA. However, that would mean you’d have to manually set the passwd for all your users, either via the admin web I/Fs or using the API.

(vi) If I manage to complete the installation, then what are the end points/url that are exposed? How do I start getting access to the AMC and start uploading videos?

The hostname used for the Kaltura endpoint [service URL] must be resolvable from both inside the server and from any machine you intend to perform playback from or access the management web I/Fs. of course, whatever port you choose to set up the Kaltura Apache vhost on must be accessible, as well as the port you choose to use for Nginx.


#3

Jess, thank you for your response. I will think through it can get back to you if I have any further questions.

Cheers


#4

Jess, I understood how end users will get to consume the videos which are stored in s3 via the CDN. Thank you for your detailed answer on that. I have to explore further about the need for the mail server,. Your response on that point has given me something to think about. And thank you so much for the link about the platform with Ec2 and S3. That is very helpful.

Cheers


#5

A couple of more questions -

(i) Is this link the client libraries for the hosted edition?
https://developer.kaltura.com/api-docs/Client_Libraries

(ii) For the Community Edition, are we supposed to generate it using the .generate.sh command after installation of the platform on the EC2 instance?

or is my understanding of client libraries completely messed up?

Cheers


#6

Hi @vinod_kumar1,

The SDKs available from https://developer.kaltura.com/api-docs/Client_Libraries are meant to be used against our own hosted ENV [i.e: www.kaltura.com or as it is sometimes knows - the Kaltura SaaS]. While when running Kaltura CE of the same version, they will most probably work correctly, you should instead generate clients by invoking the following from one of your Kaltura API servers:

# php /opt/kaltura/app/generator/generate.php <client name>

Where is one of the clients listed in /opt/kaltura/clients-generator/config/generator.all.ini.
That would ensure they are compatible with the Kaltura Server version you have deployed.

The post install scripts use the same method to generate the mandatory PHP clients, which are used by our various admin I/F as well as by the Kaltura Batch daemon.

Cheers,