Nginx Setup Help


#1

After updating to 13.20 (from a much older version using progressive download), I’ve run into some trouble with the nginx configuration. My test server doesn’t use HTTPs, so it went well there, but something appears to be wrong on my production server.

When I first ran the config script, I had both the VOD_PACKAGER_PORT and VOD_PACKAGER_SSL_PORT set to 88. I did try re-running the config with the SSL PORT set to 8443.

I did find the delivery profile table (taken from the instructions) and manually switched the url to use 8443 instead of 88. Ports 88 and 8443 are both open.

My nginx error log looks like this:
2018/06/06 14:59:14 [error] 13520#13520: *1 SSL_do_handshake() failed (SSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol) while SSL handshaking to upstream, client: [ip removed], server: servername, request: “GET /hls/p/100/sp/10000/serveFlavor/entryId/0_ss5ctjeb/v/2/flavorId/0_3lpnwekh/name/a.mp4/index.m3u8 HTTP/1.1”, subrequest: “/kalapi_proxy/hls/p/100/sp/10000/serveFlavor/entryId/0_ss5ctjeb/v/2/flavorId/0_3lpnwekh/name/a.mp4”, upstream: “https://127.0.0.1:80/hls/p/100/sp/10000/serveFlavor/entryId/0_ss5ctjeb/v/2/flavorId/0_3lpnwekh/name/a.mp4?pathOnly=1”, host: “servername:8443”, referrer: “https://servername/index.php/extwidget/preview/partner_id/100/uiconf_id/23448445/entry_id/0_ss5ctjeb/embed/auto?&flashvars[streamerType]=auto

2018/06/06 14:59:14 [error] 13520#13520: *1 open() “/etc/nginx/html/50x.html” failed (2: No such file or directory), client: [ip removed], server: servername, request: “GET /hls/p/100/sp/10000/serveFlavor/entryId/0_ss5ctjeb/v/2/flavorId/0_3lpnwekh/name/a.mp4/index.m3u8 HTTP/1.1”, host: “servername:8443”, referrer: “https://servername/index.php/extwidget/preview/partner_id/100/uiconf_id/23448445/entry_id/0_ss5ctjeb/embed/auto?&flashvars[streamerType]=auto

In the browser when I try to play, I get an error that references the Access-Control-Allow-Origin header:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://servername:8443/hls/p/100/sp/10000/serveFlavor/entryId/0_um16sz7y/v/12/flavorId/0_w9x78t15/name/a.mp4/index.m3u8. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

(I’ve blocked out the requesting IP and server IP, but they do show up correctly)

I found mention of adding this to kaltura.conf in another thread, but it seemed to be the non-SSL config. I wasn’t sure if I needed to add it (or where to add it) for an SSL config. I’m happy to provide any of the config files. In the other threads I’ve found, they look to be the same (as well as the delivery profiles table).

Thanks for your help!


#2

Hi @craig.jamieson,

I believe your problem starts here:

When you try:

$ curl $NGINX_HOSTNAME:$NGINX_PORT

Does it work? I suspect it would fail…

The file ssl_certificate points to should contain both the cert and the CA, is it possible that’s not your case and that it’s missing the CA?


#3

Hi @jess!

I did have the wrong cert setup: it was the one I was using for apache. I can’t re-use the same cert for nginx, since it does require cert -> intermediate -> ca. I grabbed the pem version of it that I had made for monit and tried that instead. Accessing via curl now returns a 404 error (previously without intermediate -> ca it showed a cert error).

The error in the nginx logs does persist when I try to play a video. Is there possibly something wrong with my Access-Control-Allow-Origin part of the header response (the browser error)? I did check the hls section in kaltura.conf for nginx and it seems to have the directive to add it, but perhaps it needs to go somewhere else.


#4

Hi @craig.jamieson,

I suggest you start by following this thread and verifying everything is as it should be:

If, afterwards, you still have issues/questions, please post your full config [masking sensitive data] as well as the records from the delivery_profile table and I’ll gladly assist you further.

Also, when making the playmanifest request [made when you hit “play”], please check /opt/kaltura/log/kaltura_prod.log and verify the correct delivery profile was chosen. It may not be what you think it is:)


#5

Hi @jess, here’s a summary of the differences:

in kaltura.ssl.conf, my virtualhost directive is _ default _:443 (without the spaces), rather than the servername. I got an SSL error if I use the servername directly - perhaps that is related?

in nginx.conf: there’s one additional line in below listen 88 -> include /etc/nginx/conf.d/live.conf
and I have an RTMP configuration directive at the bottom (listen 1935) which isn’t present in the one in the thread
[for reference, I don’t actually use the live streaming option, so I think these could be removed]

kaltura.conf: There are two extra directives at the bottom, one for /dashme and one for /hlsme, otherwise the files are the same.

ssl.conf: these files are the same

delivery profiles match for 1001, 1002, 1003 (there are some default ones: 1 - 5, 301 - 304)

When I check the kaltura_prod.log there are a number of lines, one does return with Delivery ID for parterId [100] and streamer type [applehttp] is 1001, so I think it’s selecting the correct delivery profile, since that would be the HLS one.

kaltura.conf:

	# static files (crossdomain.xml, robots.txt etc.) + fallback to api
	location / {
		root   /etc/nginx/static;
		try_files $uri @api_fallback;
	}

	# nginx status page
	location /nginx_status {
		stub_status on;
		access_log off;
	}

	# vod status page
	location /vod_status {
		vod_status;
		access_log off;
	}
	
	# internal location for vod subrequests
	location /kalapi_proxy/ {
		internal;
		proxy_pass https://kalapi/;
		proxy_set_header Host $http_host;
	}
	
	# serve flavor progressive (clipFrom/To are not supported with 'vod none' so they are proxied)
	location ~ ^/p/\d+/(sp/\d+/)?serveFlavor/((?!clipFrom)(?!clipTo).)*$ {
		vod none;

		add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
		expires 100d;
	}
	
	# serve flavor HLS
	location ~ ^/hls/p/\d+/(sp/\d+/)?serveFlavor/ {
		vod hls;
		vod_bootstrap_segment_durations 2000;
		vod_bootstrap_segment_durations 2000;
		vod_bootstrap_segment_durations 2000;
		vod_bootstrap_segment_durations 4000;

		add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
		add_header Access-Control-Allow-Headers "*";
		add_header Access-Control-Expose-Headers "Server,range,Content-Length,Content-Range";
		add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS";
		add_header Access-Control-Allow-Origin "*";
		expires 100d;
	}
	
	# serve flavor DASH
	location ~ ^/dash/p/\d+/(sp/\d+/)?serveFlavor/ {
		vod dash;
		vod_segment_duration 4000;
		vod_bootstrap_segment_durations 3500;
		vod_align_segments_to_key_frames on;
		vod_dash_manifest_format segmenttemplate;
		
		add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
		add_header Access-Control-Allow-Headers "origin,range,accept-encoding,referer";
		add_header Access-Control-Expose-Headers "Server,range,Content-Length,Content-Range";
		add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS";
		add_header Access-Control-Allow-Origin "*";
		expires 100d;
	}
	
	# serve flavor HDS
	location ~ ^/hds/p/\d+/(sp/\d+/)?serveFlavor/ {
		vod hds;
		vod_segment_duration 6000;
		vod_align_segments_to_key_frames on;
		vod_segment_count_policy last_rounded;
		
		add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
		add_header Access-Control-Allow-Origin "*";
		expires 100d;
	}

	# serve flavor MSS
	location ~ ^/mss/p/\d+/(sp/\d+/)?serveFlavor/ {
		vod mss;
		vod_segment_duration 4000;
		vod_manifest_segment_durations_mode accurate;
		
		add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
		expires 100d;
	}	
	
	# all unidentified requests fallback to api (inc. playManifest)
	location @api_fallback {
		proxy_pass https://kalapi;
		proxy_set_header Host $http_host;
	}
	
	#error_page  404			  /404.html;

	# redirect server error pages to the static page /50x.html
	#
	error_page   500 502 503 504  /50x.html;
	location = /50x.html {
		root   html;
	}

            location /dashme {
                    open_file_cache off;
                    root /var/tmp;
                    add_header Cache-Control no-cache;
                    # To avoid issues with cross-domain HTTP requests (e.g. during development)
                    add_header Access-Control-Allow-Origin *;
            }
            location /hlsme {
                    open_file_cache off;
                    types {
                            application/vnd.apple.mpegurl m3u8;
                    }
                    root /var/tmp;
                    add_header Cache-Control no-cache; # Prevent caching of HLS fragments
                    add_header Access-Control-Allow-Origin *; # Allow web player to access our playlist
            }

nginx.conf

user  kaltura;
worker_processes  auto;

error_log  /opt/kaltura/log/kaltura_nginx_errors.log;

pid		/var/run/nginx.pid;

events {
	worker_connections  1024;
	multi_accept on;
	use epoll;
}

http {
	upstream kalapi {
		server server.ca;
	}
		
	include	   mime.types;
	default_type  application/octet-stream;

	log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
		'$status $bytes_sent $request_time "$http_referer" "$http_user_agent" "-" - '
		'"$sent_http_x_kaltura" "$http_host" $pid $sent_http_x_kaltura_session - '
		'$request_length "$sent_http_content_range" "$http_x_forwarded_for" '
		'"$http_x_forwarded_server" "$http_x_forwarded_host" "$sent_http_cache_control" '
		'$connection ';

	access_log /opt/kaltura/log/kaltura_nginx_access.log main;

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;

	keepalive_timeout 60;
	keepalive_requests 1000;
	client_header_timeout 20;
	client_body_timeout 20;
	reset_timedout_connection on;
	send_timeout 20;

	gzip  on;
	gzip_types application/vnd.apple.mpegurl video/f4m application/dash+xml text/xml;
	# common vod settings
	vod_mode mapped;
	vod_upstream_location /kalapi_proxy;
	vod_upstream_extra_args "pathOnly=1";

	# shared memory zones
	vod_metadata_cache metadata_cache 512m;
	vod_mapping_cache mapping_cache 64m;
	vod_response_cache response_cache 64m;
	vod_performance_counters perf_counters;

	# common file caching / aio
	open_file_cache max=1000 inactive=5m;
	open_file_cache_valid 2m;
	open_file_cache_min_uses 1;
	open_file_cache_errors on;
	aio on;

	server {
		listen 88;
		server_name server.ca;
		include /etc/nginx/conf.d/live.conf;
		include /etc/nginx/conf.d/kaltura.conf;

	}
	include /etc/nginx/conf.d/ssl.conf;
}
# RTMP configuration
rtmp {
    server {
        listen 1935; # Listen on standard RTMP port
        chunk_size 4000;

        # This application is to accept incoming stream
        application kLive {
                live on; # Allows live input from above
                dash on;
                dash_path /var/tmp/dashme;

                hls on; # Enable HTTP Live Streaming
                hls_cleanup on;
                hls_sync 100ms;
                hls_fragment 2s;

                hls_path /var/tmp/hlsme/;


        }
    }
}

ssl.conf

# HTTPS server
#
server {
    listen     8443   ssl;
    server_name server.ca;

    ssl_certificate      /etc/pki/tls/certs/nginx/server.ca.crt;
    ssl_certificate_key  /etc/pki/tls/private/server.ca.key;

    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  5m;

    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers   on;

    include /etc/nginx/conf.d/kaltura.conf;
}

zzzkaltura.ssl.conf

	<IfModule !ssl_module>
	LoadModule ssl_module modules/mod_ssl.so
</IfModule>


SSLPassPhraseDialog  builtin
SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout  300
SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin
<IfVersion < 2.4>
        SSLMutex default
</IfVersion>
<IfVersion >= 2.4>
        Mutex sysvsem default
</IfVersion>
SSLCryptoDevice builtin

SSLCertificateFile /etc/pki/tls/certs/server.ca.crt
SSLCertificateKeyFile /etc/pki/tls/private/server.ca.key
SSLCACertificateFile /etc/pki/tls/certs/intermediate.crt

<VirtualHost _default_:443>
	SSLEngine on
	SSLProtocol all -SSLv2

	SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
	
	ErrorLog "/opt/kaltura/log/kaltura_apache_errors_ssl.log"
	CustomLog /opt/kaltura/log/kaltura_apache_access_ssl.log vhost_kalt
	
	Include "/opt/kaltura/app/configurations/apache/conf.d/enabled.*.conf"
</VirtualHost>

delivery_profile:

*************************** 1. row ***************************
id: 1001
partner_id: 0
created_at: 2016-04-29 11:46:18
updated_at: 2016-04-29 11:46:18
name: Kaltura HLS segmentation
type: 61
system_name: Kaltura HLS segmentation
description: Kaltura HLS segmentation
url: https://server.ca:8443/hls
host_name: server.ca
recognizer: NULL
tokenizer: NULL
status: 0
media_protocols: NULL
streamer_type: applehttp
is_default: 1
parent_id: 0
custom_data: NULL
priority: 0
*************************** 2. row ***************************
id: 1002
partner_id: 0
created_at: 2016-04-29 11:46:18
updated_at: 2016-04-29 11:46:18
name: Kaltura HDS segmentation
type: 63
system_name: Kaltura HDS segmentation
description: Kaltura HDS segmentation
url: https://server.ca:8443/hds
host_name: server.ca
recognizer: NULL
tokenizer: NULL
status: 0
media_protocols: NULL
streamer_type: hdnetworkmanifest
is_default: 1
parent_id: 0
custom_data: NULL
priority: 0
*************************** 3. row ***************************
id: 1003
partner_id: 0
created_at: 2016-04-29 11:46:18
updated_at: 2016-04-29 11:46:18
name: Kaltura DASH segmentation
type: 68
system_name: Kaltura DASH segmentation
description: Kaltura DASH segmentation
url: https://server.ca:8443/dash
host_name: server.ca
recognizer: NULL
tokenizer: NULL
status: 0
media_protocols: NULL
streamer_type: mpegdash
is_default: 1
parent_id: 0
custom_data: NULL
priority: 0
3 rows in set (0.00 sec)


#6

Hi @craig.jamieson,

Are you saying that:
$ curl https://server.ca/api_v3
does not return correctly?

Obviously, server.ca is just a place holder but when making a request over HTTPs using the actual hostname, this should work… if it doesn’t, it would certainly explain it.

What’s the response you’re getting when making the request?
Also, when hitting play, capture the playmanifest request the player makes and make it manually using curl -v from the shell and have a look at the response.


#7

curl https://server/api_v3 returns this:

<?xml version="1.0" encoding="utf-8"?><xml><result><error><code>SERVICE_NOT_SPECIFIED</code><message>Service name was not specified, please specify one</message><objectType>KalturaAPIException</objectType><args></args></error></result><executionTime>0.024806976318359</executionTime></xml>

When I hit play, I can see in the browser console that my sample video is requesting: https://server.ca:8443/hls/p/100/sp/10000/serveFlavor/entryId/0_um16sz7y/v/12/flavorId/0_w9x78t15/name/a.mp4/index.m3u8

Is that what you meant? or did you want a log from the server? The browser console also gives the note about the Access-Control-Allow-Origin header missing.

Hitting that with curl -v gives:

 About to connect() to server.ca port 8443 (#0)
   Trying XXX.XXX.XXX.XXX...
 Connected to server.ca (XXX.XXX.XXX.XXX) port 8443 (#0)
 Initializing NSS with certpath: sql:/etc/pki/nssdb
   CAfile: /etc/pki/tls/certs/ca-bundle.crt
 CApath: none
 SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 Server certificate:
       subject: CN=server.ca,O=,L=,ST=,C=CA
       start date:  GMT
       expire date:  GMT
       common name: server.ca
       issuer: CN=GlobalSign Organization Validation CA - SHA256 - G2,O=GlobalSign nv-sa,C=BE
 GET /hls/p/100/sp/10000/serveFlavor/entryId/0_um16sz7y/v/12/flavorId/0_w9x78t15/name/a.mp4/index.m3u8 HTTP/1.1
 User-Agent: curl/7.29.0
 Host: server.ca:8443
 Accept: */*

 HTTP/1.1 404 Not Found
 Server: nginx/1.13.12
 Date: Thu, 07 Jun 2018 20:24:43 GMT
 Content-Type: text/html
 Content-Length: 170
 Connection: keep-alive

html>
head><title>404 Not Found</title></head>
body bgcolor="white">
center><h1>404 Not Found</h1></center>
hr><center>nginx/1.13.12</center>
/body>
/html>
 Connection #0 to host server.ca left intact

I did, however, discover why the server.ca directive in my virtual host wasn’t working. A long, long time ago, you used to have to add an entry to the hosts file on the server that looked like: 127.0.0.1 fqdn, which I must have added out of habit. Now that I’ve removed it, the virtual host is fine with the fqdn listed.

If you’d like more info for further testing (like the servername to try and an actual embed), please let me know and I can always email that to you.


#8

Good news @jess! I figured out what my issue was.

I noticed on the error log in the server that the upstream field was saying: upstream: “https://server_ip:80/hls/p/100/sp/10000/serveFlavor/entryId/0_ss5ctjeb/v/2/flavorId/0_3lpnwekh/name/a.mp4?pathOnly=1”

And figured that the port 80 was probably incorrect. I ran just the nginx config (kaltura-nginx-config.sh) that you mentioned in one of your other posts. The first thing it prompts you for is “Kaltura API host and port (without the protocol)” and mine was defaulting to fqdn:80. When I changed it to fqdn:443, now the videos play as they should. I used an old answers file when I did the upgrade, so I suspect I didn’t setup nginx correctly a couple of years ago (when nginx wasn’t needed).

So my updates go smoothly in the future, do you happen to know which line in the answers file references “Kaltura API host and port (without the protocol)”? Or does this get stored in a config file somewhere outside of the answers file?

Digging a little bit more, I noticed that the information seems to only be used in nginx.conf. In the “upstream kalapi” section, if just the hostname is listed, then it seems to use port 80, whereas now it’s listed as hostname:443 and that seems to work.


#9

Hi @craig.jamieson,

Considering I’m the one who wrote this code, it would be mighty curious if I didn’t know:)
The var in question is WWW_HOST.
For a full listing of all the needed vars see:

If you want to know which var is used for any of the prompts, you can always look inside the shell script, in this case:

        echo -e "${CYAN}Kaltura API host and port (without the protocol) [${YELLOW}`hostname`:80${CYAN}]:${NORMAL} "
        read -e WWW_HOST
        if [ -z "$WWW_HOST" ];then
                WWW_HOST="`hostname`:80"
        fi

Cheers,


#10

I guess you do know a thing or two about it, @jess. :slight_smile:

I’m not sure how I’ve missed this in the past - the answer template is fantastic! I had been looking for something similar the other day prior to the update. Will bookmark for later use.

Thanks for your help narrowing things down to find my error. It’s great to be up and running on the new version.


#11

Welcome, @craig.jamieson:)