Nginx Live Stream Configuration

@jess

Ok I find the source of the problem with this folder : Secure Tmp Systemd
(https://support.plesk.com/hc/en-us/articles/115000063849-Directories-like-tmp-systemd-private-overflow-and-cause-server-crash-due-to-lack-of-disk-space)

grep -R PrivateTmp /etc/systemd/
/etc/systemd/system/multi-user.target.wants/ntpd.service:PrivateTmp=true
/etc/systemd/system/multi-user.target.wants/kaltura-nginx.service:PrivateTmp=true
/etc/systemd/system/multi-user.target.wants/dovecot.service:PrivateTmp=true
/etc/systemd/system/multi-user.target.wants/mariadb.service:PrivateTmp=true
/etc/systemd/system/multi-user.target.wants/httpd.service:PrivateTmp=true

I will try this week-end to de-activate this function and I’ll come back here to close this topic :slight_smile:

Thanks a lot and have a nice week-end.
GreG

Hi @ZgreG,

This is what PrivateTmp does:

    If true sets up a new file system
    namespace for the executed processes and mounts a private /tmp
    directory inside it, that is not shared by processes outside of
    the namespace. This is useful to secure access to temporary files
    of the process, but makes sharing between processes via /tmp
    impossible. Defaults to false. 

I never tried to use it myself but it should still work. However, you can always change the directory in nginx.conf to something else, just to eliminate the option that PrivateTmp is the cause of the issue. Remember to reload Nginx if you do so.

At any rate, if you do see the m3u8 and TS files, try to play and entry while you have your browser’s dev tools open and check for failing requests under the “Network” tab and for error under the “Console” tab and paste them here.

Hello @jess

I fixed the problem with the PrivateTmp service.
Now I see the .m3u8 file and all .TS files in the /var/tmp/hlsme folder.

When I try to play the stream in my Web browser with HTML-5 player nothing happens and I don’t have any error message in the console :

kWidget: Kaltura HTML5 Version: 2.57 ( iframe )   mwEmbedLoader.php:50:636
GET http://vodyou.fr/lib/css/shortlink.css [HTTP/1.1 200 OK 0 ms]
GET http://vodyou.fr/html5/html5lib/v2.57/load.php [HTTP/1.1 200 OK 0 ms]
GET http://vodyou.fr/p/101/sp/10100/thumbnail/entry_id/0_bgkkb253/version/0/width/560/height/315 [HTTP/1.1 200 OK 0 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 30 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 5032 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 21 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 30 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 30 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 25 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 25 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 35 ms]
GET XHR http://vodyou.fr/p/101/sp/10100/playManifest/entryId/0_bgkkb253/format/applehttp/protocol/http/uiConfId/23448169/a.m3u8 [HTTP/1.1 200 OK 43 ms]
GET XHR http://vodyou.fr:1935/hlsme/test.m3u8 [12 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 5032 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 5033 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 5033 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 5032 ms]
GET XHR http://vodyou.fr/api_v3/index.php [HTTP/1.1 200 OK 5035 ms]

In KMC the entry is :

HLS stream URL: http://vodyou.fr:1935/hlsme/test.m3u8
applehttp stream URL: http://vodyou.fr:1935/hlsme/test.m3u8

I didn’t have any error in /opt/kaltura/log/kaltura nginx error.log.
Only informations lines when I start the stream.

2017/06/19 13:09:03 [info] 17057#17057: *206 connect: app='kLive' args='' flashver='FMLE/3.0 (compatible; Lavf57.56' swf_url='' tc_url='rtmp://5.135.188.49:1935/kLive' page_url='' acodecs=0 vcodecs=0 object_encoding=0, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:03 [info] 17057#17057: *206 createStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:03 [info] 17057#17057: *206 publish: name='test' args='' type=live silent=0, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:04 [info] 17057#17057: *207 client connected '5.135.188.49'
2017/06/19 13:09:05 [info] 17050#17050: *204 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:05 [info] 17050#17050: *204 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:05 [info] 17057#17057: *205 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:05 [info] 17057#17057: *205 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:09 [info] 17057#17057: *207 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:09 [info] 17057#17057: *207 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:10 [info] 17049#17049: *208 client connected '5.135.188.49'
2017/06/19 13:09:10 [info] 17056#17056: *209 client connected '5.135.188.49'
2017/06/19 13:09:14 [info] 17056#17056: *210 client connected '5.135.188.49'
2017/06/19 13:09:15 [info] 17049#17049: *208 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:15 [info] 17049#17049: *208 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:15 [info] 17056#17056: *209 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:15 [info] 17056#17056: *209 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:16 [info] 17057#17057: *211 client connected '82.235.134.184'
2017/06/19 13:09:16 [info] 17050#17050: *212 client connected '5.135.188.49'
2017/06/19 13:09:19 [info] 17056#17056: *210 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:19 [info] 17056#17056: *210 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:20 [info] 17057#17057: *213 client connected '5.135.188.49'
2017/06/19 13:09:21 [info] 17050#17050: *212 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:21 [info] 17050#17050: *212 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:24 [info] 17057#17057: *214 client connected '5.135.188.49'
2017/06/19 13:09:24 [info] 17049#17049: *215 client connected '5.135.188.49'
2017/06/19 13:09:25 [info] 17057#17057: *213 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:25 [info] 17057#17057: *213 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:26 [info] 17057#17057: *211 disconnect, client: 82.235.134.184, server: 0.0.0.0:1935
2017/06/19 13:09:26 [info] 17057#17057: *211 deleteStream, client: 82.235.134.184, server: 0.0.0.0:1935
2017/06/19 13:09:29 [info] 17057#17057: *214 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:29 [info] 17057#17057: *214 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:29 [info] 17049#17049: *215 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:29 [info] 17049#17049: *215 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:30 [info] 17057#17057: *216 client connected '5.135.188.49'
2017/06/19 13:09:34 [info] 17050#17050: *217 client connected '5.135.188.49'
2017/06/19 13:09:34 [info] 17057#17057: *218 client connected '5.135.188.49'
2017/06/19 13:09:35 [info] 17057#17057: *216 disconnect, client: 5.135.188.49, server: 0.0.0.0:1935
2017/06/19 13:09:35 [info] 17057#17057: *216 deleteStream, client: 5.135.188.49, server: 0.0.0.0:1935

The file /opt/kaltura/log/kaltura_nginx_access.log is empty.

If you have an idea it would be with pleasure :wink:
GreG

Hi @ZgreG,

In KMC, it should not be:

http://vodyou.fr:1935/hlsme/test.m3u8

It should be:

http://vodyou.fr:88/hlsme/test.m3u8

Port 1935 is for streaming [RTMP], not for playback.

@jess

Everything is working :slight_smile:
This topic can be closed.

Thank a lot for your help.
GreG

Most welcome and glad to hear it:)

@jess

I am having this same issue and I have disable the ‘PrivateTMP’ feature. However, let me note that this functionally is enable by default each time a new process or service is created with yum. I have not figured out how to disable it from doing that on installation of new services. This is how I disabled it on my system.

First, edit the file with your favorite editor vi, nano or whatever you prefer.

# nano /usr/lib/systemd/system/kaltura-nginx.service

Find this line in the file

PrivateTmp=true

then change it to this

PrivateTmp=false

then run these commands

# systemctl daemon-reload
# systemctl restart kaltura-nginx

The location of this file might differ from OS to OS but my current OS is CentOS 7.1.3 and even after doing this change I can see the files in the temp directories:

/var/tmp/hlsme
/var/tmp/dashme

However, my files are still not streaming. I have tried to stream using OBS from my desktop and using my server CLI but I am not able to playback the stream from Kaltura HTML5 player or my VLC player from my desktop. I getting the following error when using VLC:

Your input can’t be opened
VLC is unable to open the MRL ‘http://cdn.mydomain.com:88/hlsme/stream.m3u8’. Check the log for details.

Now in my kaltura_nginx_errors.log I am seeing both the OBS stream and the CLI based stream starting and stopping.

2017/07/27 01:28:01 [info] 19011#19011: *1 exec: starting managed child ‘on’, client: MY_DESKTOP_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:01 [notice] 19011#19011: signal 17 (SIGCHLD) received
2017/07/27 01:28:01 [notice] 19011#19011: unknown process 11248 exited with code 1
2017/07/27 01:28:01 [info] 19011#19011: *1 exec: child 11248 exited; ignoring, client: MY_DESKTOP_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:01 [info] 19011#19011: *1 exec: terminating child 11248, client: MY_DESKTOP_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:01 [info] 19010#19010: *2 exec: starting managed child ‘on’, client: MY_SERVER_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:01 [notice] 19010#19010: signal 17 (SIGCHLD) received
2017/07/27 01:28:01 [notice] 19010#19010: unknown process 11249 exited with code 1
2017/07/27 01:28:01 [info] 19010#19010: *2 exec: child 11249 exited; ignoring, client: MY_SERVER_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:01 [info] 19010#19010: *2 exec: terminating child 11249, client: MY_SERVER_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:06 [info] 19011#19011: *1 exec: starting managed child ‘on’, client: MY_DESKTOP_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:06 [notice] 19011#19011: signal 17 (SIGCHLD) received
2017/07/27 01:28:06 [notice] 19011#19011: unknown process 11254 exited with code 1
2017/07/27 01:28:06 [info] 19011#19011: *1 exec: child 11254 exited; ignoring, client: MY_DESKTOP_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:06 [info] 19011#19011: *1 exec: terminating child 11254, client: MY_DESKTOP_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:06 [info] 19010#19010: *2 exec: starting managed child ‘on’, client: MY_SERVER_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:06 [notice] 19010#19010: signal 17 (SIGCHLD) received
2017/07/27 01:28:06 [notice] 19010#19010: unknown process 11255 exited with code 1
2017/07/27 01:28:06 [info] 19010#19010: *2 exec: child 11255 exited; ignoring, client: MY_SERVER_IP, server: MY_KALTURA_SERVER_IP:1935
2017/07/27 01:28:06 [info] 19010#19010: *2 exec: terminating child 11255, client: MY_SERVER_IP, server: MY_KALTURA_SERVER_IP:1935

You will note from my log that I set the IP used for Kaltura and NGINX to a specific IP and not the default host IP address. This helps to avoid conflicts with other services such as Red5, Wowza or simular that usually generate port conflicts and require use of non-standard ports.

I check the premissions for the tmp folders used for hlsme and dashme and they are:

# ls -al /var/tmp/
total 44
drwxrwxrwt. 11 root    root 4096 Jul 26 22:23 .
drwxr-xr-x. 21 root    root 4096 Jul 21 13:31 ..
drwx------.  2 kaltura root 4096 Jul 27 01:41 dashme
drwx------.  2 kaltura root 4096 Jul 27 01:41 hlsme

However, the files in the folder are owned and in the ‘kaltura’ usergroup as well as own by the same group.

This stream is from ffmpeg using CLI

-rw-r--r--.  1 kaltura kaltura   27072 Jul 27 01:41 frontdoor-3225.ts
-rw-r--r--.  1 kaltura kaltura   27072 Jul 27 01:41 frontdoor-3226.ts
-rw-r--r--.  1 kaltura kaltura    9588 Jul 27 01:41 frontdoor-3227.ts
-rw-r--r--.  1 kaltura kaltura     571 Jul 27 01:41 frontdoor.m3u8

This stream is from OBS using my desktop

-rw-r--r--.  1 kaltura kaltura 2874708 Jul 27 01:41 stream-965.ts
-rw-r--r--.  1 kaltura kaltura 2860044 Jul 27 01:41 stream-966.ts
-rw-r--r--.  1 kaltura kaltura 1492156 Jul 27 01:41 stream-967.ts
-rw-r--r--.  1 kaltura kaltura     510 Jul 27 01:41 stream.m3u8

As you can see the files are being created. Now, the big difference between trying to playback from my compute and Kaltura CE is that my computer basically returns an error saying no stream is available at the URL. Kaltura HTML5 player actual keeps trying to connect and play the stream but just spins non-stop and when I stop the stream completely it basically saying no broadcast is available. So this implies it can detect the stream but is not playing it.

I check the stream from Kaltura HTML5 player using the internal generated page too and view the page with my web developer tool to search for errors. There was only one error which was mixed content error saying that the URL ‘http://cdn.mydomain.com:88/hlsme/stream.m3u8’ was blocked.

Now, at this point I tried changing the url to ‘https://cdn.mydomain.com:8443/hlsme/stream.m3u8’ and got nothing in VLC and Kaltura HTML5 player display the “No broadcast is available” message. Since Kaltura HTML5 player is detecting data or stream is being recieved and VLC appears to be rejected or timesout I am wondering if this is an issue with permissions or something related to the proper configuration of the NGINX so it properly access the assigned tmp folder for the hls and dash files.

Now, I also check for SSL errors but there is no actually NGINX ssl error log for Kaltura even configured so I am not able to troubleshoot the same way to resolve issues related to the content not loading via the SSL port on NGINX.

Hi @hiphopservers,

Seems your current issue is simply that playback is attempted over HTTP whereas the page in which the player is embedded is over SSL. That would make the browser block it due to “mixed content”.

When you say:

Can you please describe what it is that you did?

Is Nginx currently listening on port 8443 with SSL?
To verify:

# netstat -plnt|grep nginx

Should return something like:
tcp        0      0 0.0.0.0:1935                0.0.0.0:*                   LISTEN      23726/nginx         
tcp        0      0 0.0.0.0:88                  0.0.0.0:*                   LISTEN      23726/nginx         
tcp        0      0 0.0.0.0:8443                0.0.0.0:*                   LISTEN      23726/nginx
# cat /etc/nginx/conf.d/ssl.conf

Should return something similar to the below, only with your own FQDN hostname and cert files:
server {
    listen     8443   ssl;
    server_name test.kaltura.org;

    ssl_certificate      /etc/pki/tls/certs/kaltura.org.crt;
    ssl_certificate_key  /etc/pki/tls/private/kaltura.org.key;

    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  5m;

    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers   on;

    include /etc/nginx/conf.d/kaltura.conf;
}

Assuming these are OK, what’s the output for:
mysql> select * from delivery_profile where id in (1001,1002,1003)\G
?

A much more detailed explanation about the Nginx VOD module, delivery profiles and Apache configuration can be found here:

@jess

That would explain the issue with the Kaltura HTML5 player but not why the stream is not loading in my desktop VLC player since there were not NGINX playback restrictions in the configuration file to limit what IPs or clients could access the direct link.

I entered this link into both my desktop VLC player and got an immediate error that the connection was not available. Unless the HTTP URL which dies in VLC after the attempt to connect to the stream times out. Similar with Kaltura HTML5 player which immediately displays a message saying “No Broadcast is Available” unlike the HTTP version which just spins.

# netstat -plnt|grep nginx
tcp        0      0 KALTURA_SERVER_IP:1935      0.0.0.0:*               LISTEN      12204/nginx: master
tcp        0      0 KALTURA_SERVER_IP:88        0.0.0.0:*               LISTEN      12204/nginx: master
tcp        0      0 KALTURA_SERVER_IP:8443      0.0.0.0:*               LISTEN      12204/nginx: master

As you can see Kaltura-NGINX is listening on the caption ports. However as I mention the non-SSL ports just timeout or never actually stream. So it is detecting data being sent but nothing is actually showing up. The SSL connection is just straight refused.

# cat /etc/nginx/conf.d/ssl.conf

The SSL cert configured here is using my Wildcard Cert we discuss previously. You will notice I have bind the process to a specific IP on my system in the configuration file for all ports.

# HTTPS server
#
server {
    listen     KALUTRA_SERVER_IP:8443   ssl;
    server_name cdn.mydomain.com;

    ssl_certificate      /etc/ssl/certs/server.crt;
    ssl_certificate_key  /etc/ssl/certs/server.key;

    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  5m;

    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers   on;

    include /etc/nginx/conf.d/kaltura.conf;
}

My results from this qurey is as follows:

*************************** 1. row ***************************
             id: 1001
     partner_id: 0
     created_at: 2016-03-18 13:34:15
     updated_at: 2016-03-18 13:34:15
           name: Kaltura HLS segmentation
           type: 61
    system_name: Kaltura HLS segmentation
    description: Kaltura HLS segmentation
            url: https://cdn.mydomain.com:88/hls
      host_name: cdn.mydomain.com
     recognizer: NULL
      tokenizer: NULL
         status: 0
media_protocols: NULL
  streamer_type: applehttp
     is_default: 1
      parent_id: 0
    custom_data: NULL
       priority: 0
*************************** 2. row ***************************
             id: 1002
     partner_id: 0
     created_at: 2016-03-18 13:34:15
     updated_at: 2016-03-18 13:34:15
           name: Kaltura HDS segmentation
           type: 63
    system_name: Kaltura HDS segmentation
    description: Kaltura HDS segmentation
            url: https://cdn.mydomain.com:88/hds
      host_name: cdn.mydomain.com
     recognizer: NULL
      tokenizer: NULL
         status: 0
media_protocols: NULL
  streamer_type: hdnetworkmanifest
     is_default: 1
      parent_id: 0
    custom_data: NULL
       priority: 0
*************************** 3. row ***************************
             id: 1003
     partner_id: 0
     created_at: 2016-03-18 13:34:15
     updated_at: 2016-03-18 13:34:15
           name: Kaltura DASH segmentation
           type: 68
    system_name: Kaltura DASH segmentation
    description: Kaltura DASH segmentation
            url: https://cdn.mydomain.com:88/dash
      host_name: cdn.mydomain.com
     recognizer: NULL
      tokenizer: NULL
         status: 0
media_protocols: NULL
  streamer_type: mpegdash
     is_default: 1
      parent_id: 0
    custom_data: NULL
       priority: 0
3 rows in set (0.00 sec)

Hi @hiphopservers,

In the delivery_profile table, the base URL should be ‘https://cdn.mydomain.com:8443’ not ‘https://cdn.mydomain.com:88’.
Update the delivery_profile.url columns and playback should work. If it doesn’t, please provide a sample URL where the player is embedded and I’ll take a look.

The resolution to this issue is related to the SSL certificate defined in NGINX. My specific SSL certificate is a Wildcard and requires a CA Root Certificate and since NGINX does not have the functionally to define a CA certificate path the same way as Apache you must set the cert with a bundle-crt or a pem certificate as shown in this example.

I have figured out the solution with a basic Bing search. Which return comments that when using an SSL Cert with NGINX you should assign the CA crt-bundle or set a pem as the cert as shown.

Edit the following file to reflect your correct information.

# /etc/nginx/conf.d/ssl.conf

# HTTPS server
#
server {
listen     KALUTRA_SERVER_IP:8443   ssl;
server_name cdn.mydomain.com;

ssl_certificate    /etc/ssl/your_domain_name.pem; (or bundle.crt)
ssl_certificate_key    /etc/ssl/your_domain_name.key;

ssl_session_cache shared:SSL:1m;
ssl_session_timeout  5m;

ssl_ciphers  HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers   on;

include /etc/nginx/conf.d/kaltura.conf;

}
You have to define the https URL in Kaltura with the correct SSL port when this is completed as previously described.