Updating to 15.3 - nginx error

I’m having some trouble updating my kaltura server (attempted on test server, it’s a bit simpler: http only) to 15.3. I’m currently on 13.20 I believe, one of the versions before kmcng was added. Everything seems to go well with the update, but my html5 videos won’t play afterwards (using the old flash method works). Here’s the error message:

==> /opt/kaltura/log/kaltura_nginx_errors.log <==
2019/07/24 08:08:09 [error] 1512#1512: *1 media_set_parse_json: failed to parse json -1: expected digit got 0x0 while sending to client, client: XXX.XXX.XXX.XXX, server: server.ca, request: “GET /dash/p/100/sp/10000/serveFlavor/entryId/0_rdqphwxq/v/2/flavorId/0_24orke06/forceproxy/true/name/a.mp4/manifest.mpd HTTP/1.1”, host: “server.ca:88”, referrer: “http://server.ca/html5/html5lib/v2.77.1/mwEmbedFrame.php/p/100/uiconf_id/23448412/entry_id/0_rdqphwxq?wid=_100&iframeembed=true&flashvars[kAnalony.plugin]=false&flashvars[dash.plugin]=true&flashvars[closedCaptions.plugin]=true&flashvars[closedCaptions.hideWhenEmpty]=true&flashvars[disableAlerts]=true&entry_id=0_rdqphwxq&hash=1

2019/07/24 08:08:09 [error] 1512#1512: *1 open() “/etc/nginx/html/50x.html” failed (2: No such file or directory), client: XXX.XXX.XXX.XXX, server: server.ca, request: “GET /dash/p/100/sp/10000/serveFlavor/entryId/0_rdqphwxq/v/2/flavorId/0_24orke06/forceproxy/true/name/a.mp4/manifest.mpd HTTP/1.1”, host: “server.ca:88”, referrer: “http://dmz6.nurs.ualberta.ca/html5/html5lib/v2.77.1/mwEmbedFrame.php/p/100/uiconf_id/23448412/entry_id/0_rdqphwxq?wid=_100&iframeembed=true&flashvars[kAnalony.plugin]=false&flashvars[dash.plugin]=true&flashvars[closedCaptions.plugin]=true&flashvars[closedCaptions.hideWhenEmpty]=true&flashvars[disableAlerts]=true&entry_id=0_rdqphwxq&hash=1

I spent a couple of days trying to track it down, but haven’t had any success, so I thought I’d ask here. I thought perhaps it was due to the “vod_max_mapping_response_size” setting, which I found while searching, but upping the size didn’t seem to change anything. I also tried looking at the network requests in the browser, but I couldn’t narrow down what it might be.

I’m wondering if it’s perhaps a permissions issue. Many years ago when I did the install, I believe it was run as root, so in the past my update issues have often been related to permissions issues. I tried combing through the various shell files that do the configuration, but I haven’t been able to tell for certain which folders need to have the apache user as the owner.

Thanks for your help. The html5 kmc is really great, I’m looking forward to using it a bit more.

Hello @craig.jamieson,

Start by reviewing my responses here Install kaltura without nginx.
If after following that, you cannot locate the root cause, please provide the output for all the commands specified in that thread and we’ll take it from there.

Thanks,

Hi @jess! Thanks for agreeing to help.

Here’s a file by file comparison:

nginx.conf ->

user  kaltura;
worker_processes  auto;

error_log  /opt/kaltura/log/kaltura_nginx_errors.log;

pid             /var/run/nginx.pid;

events {
        worker_connections  1024;
        multi_accept on;
        use epoll;
}

http {
        upstream kalapi {
                server server.ca;
                add_header Access-Control-Allow-Origin server.ca
        }

        include    mime.types;
        default_type  application/octet-stream;

        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                '$status $bytes_sent $request_time "$http_referer" "$http_user_agent" "-" - '
                '"$sent_http_x_kaltura" "$http_host" $pid $sent_http_x_kaltura_session - '
                '$request_length "$sent_http_content_range" "$http_x_forwarded_for" '
                '"$http_x_forwarded_server" "$http_x_forwarded_host" "$sent_http_cache_control" '
                '$connection ';

        access_log /opt/kaltura/log/kaltura_nginx_access.log main;

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;

        keepalive_timeout 60;
        keepalive_requests 1000;
        client_header_timeout 20;
        client_body_timeout 20;
        reset_timedout_connection on;
        send_timeout 20;

        gzip  on;
        gzip_types application/vnd.apple.mpegurl video/f4m application/dash+xml text/xml;
        # common vod settings
        vod_mode mapped;
        vod_upstream_location /kalapi_proxy;
        vod_upstream_extra_args "pathOnly=1";

        # shared memory zones
        vod_metadata_cache metadata_cache 512m;
        vod_mapping_cache mapping_cache 64m;
        vod_response_cache response_cache 64m;
        vod_performance_counters perf_counters;

        # common file caching / aio
        open_file_cache max=1000 inactive=5m;
        open_file_cache_valid 2m;
        open_file_cache_min_uses 1;
        open_file_cache_errors on;
        aio on;

        vod_max_mapping_response_size 10m;

        server {
                listen 88;
                server_name server.ca;
                include /etc/nginx/conf.d/live.conf;
                include /etc/nginx/conf.d/kaltura.conf;

        }
        include /etc/nginx/conf.d/ssl.conf;
}
# RTMP configuration
rtmp {
    server {
        listen 1935; # Listen on standard RTMP port
        chunk_size 4000;

        # This application is to accept incoming stream
        application kLive {
                live on; # Allows live input from above

                # ADAPTIVE BITRATE SUPPORT
                # Once the stream is alive, transcode for adaptive streaming
                # This single ffmpeg command takes the input and transforms
                # the source into 4 different streams with different bitrate
                # and quality. The scaling done here respects the aspect
                # ratio of the input.

                # This is commented by default as it is CPU and RAM intensive
                # If you opt to enable it, ensure you have sufficient resources.

                #exec /opt/kaltura/bin/ffmpeg -i rtmp://dmz6.nurs.ualberta.ca:1935/$app/$name -async 1 -vsync -1
                #        -c:v libx264 -b:v 256k -b:a 32k -vf "scale=480:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://server.ca:1935/kLive/$name_low
                #        -c:v libx264 -b:v 768k -b:a 96k -vf "scale=720:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://server.ca:1935/kLive/$name_mid
                #        -c:v libx264 -b:v 1024k -b:a 128k -vf "scale=960:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://server.ca:1935/kLive/$name_high
                #        -c:v libx264 -b:v 1920k -b:a 128k -vf "scale=1280:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://server.ca:1935/kLive/$name_hd720
                #        -c copy -f flv rtmp://server.ca:1935/kLive/$name_src;

                dash on;
                dash_path /var/tmp/dashme;

                hls on; # Enable HTTP Live Streaming
                # Instruct clients to adjust resolution according to bandwidth
                hls_variant _low BANDWIDTH=288000; # Low bitrate, sub-SD resolution
                hls_variant _mid BANDWIDTH=448000; # Medium bitrate, SD resolution
                hls_variant _high BANDWIDTH=1152000; # High bitrate, higher-than-SD resolution
                hls_variant _hd720 BANDWIDTH=2048000; # High bitrate, HD 720p resolution
                hls_variant _src BANDWIDTH=4096000; # Source bitrate, source resolution

                hls_cleanup on;
                hls_sync 100ms;
                hls_fragment 2s;
                hls_playlist_length 300m;
                hls_path /var/tmp/hlsme/;

                # Enable recording archived files of each stream
                record all;
                # This does not need to be publicly accessible since we'll convert and publish the files later
                record_path /var/tmp/rec;
                record_suffix _%Y-%m-%d_%H-%M-%S.flv;
                record_lock on;

                # Define the two scripts that will run when recording starts and when it finishes
                exec_publish /opt/kaltura/bin/create_live_entry.php $args;
                exec_record_done /opt/kaltura/bin/upload_live_recording.php $args $path;


        }
    }
}

I added the add_header directive (which isn’t part of the default) as mentioned in the other thread to try and help. I have a “vod_max_mapping_response_size 10m;” directive since I read it might help with my specific error. I have a bunch of extra directives in the rtmp part for live streaming, but I don’t suspect those would mess things up (happy to test though if you’d like).

kaltura.conf ->

        # static files (crossdomain.xml, robots.txt etc.) + fallback to api
        location / {
                root   /etc/nginx/static;
                try_files $uri @api_fallback;
        }

        # nginx status page
        location /nginx_status {
                stub_status on;
                access_log off;
        }

        # vod status page
        location /vod_status {
                vod_status;
                access_log off;
        }

        # internal location for vod subrequests
        location /kalapi_proxy/ {
                internal;
                proxy_pass http://kalapi/;
                proxy_set_header Host $http_host;
        }

        # serve flavor progressive (clipFrom/To are not supported with 'vod none' so they are proxied)
        location ~ ^/p/\d+/(sp/\d+/)?serveFlavor/((?!clipFrom)(?!clipTo).)*$ {
                vod none;

                add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
                expires 100d;
        }

        # serve flavor HLS
        location ~ ^/hls/p/\d+/(sp/\d+/)?serveFlavor/ {
                vod hls;
                vod_bootstrap_segment_durations 2000;
                vod_bootstrap_segment_durations 2000;
                vod_bootstrap_segment_durations 2000;
                vod_bootstrap_segment_durations 4000;

                add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
                add_header Access-Control-Allow-Headers "*";
                add_header Access-Control-Expose-Headers "Server,range,Content-Length,Content-Range";
                add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS";
                add_header Access-Control-Allow-Origin "*";
                expires 100d;
        }

        # serve flavor DASH
        location ~ ^/dash/p/\d+/(sp/\d+/)?serveFlavor/ {
                vod dash;
                vod_segment_duration 4000;
                vod_bootstrap_segment_durations 3500;
                vod_align_segments_to_key_frames on;
                vod_dash_manifest_format segmenttemplate;

                add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
                add_header Access-Control-Allow-Headers "origin,range,accept-encoding,referer";
                add_header Access-Control-Expose-Headers "Server,range,Content-Length,Content-Range";
                add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS";
                add_header Access-Control-Allow-Origin "*";
                expires 100d;
        }

        # serve flavor HDS
        location ~ ^/hds/p/\d+/(sp/\d+/)?serveFlavor/ {
                vod hds;
                vod_segment_duration 6000;
                vod_align_segments_to_key_frames on;
                vod_segment_count_policy last_rounded;

                add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
                add_header Access-Control-Allow-Origin "*";
                expires 100d;
        }

        # serve flavor MSS
        location ~ ^/mss/p/\d+/(sp/\d+/)?serveFlavor/ {
                vod mss;
                vod_segment_duration 4000;
                vod_manifest_segment_durations_mode accurate;

                add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
                expires 100d;
        }

        # all unidentified requests fallback to api (inc. playManifest)
        location @api_fallback {
                proxy_pass http://kalapi;
                proxy_set_header Host $http_host;
        }

        #error_page  404                          /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
                root   html;
        }

        location /dashme {
                open_file_cache off;
                root /var/tmp;
                add_header Cache-Control no-cache;
                # To avoid issues with cross-domain HTTP requests (e.g. during development)
                add_header Access-Control-Allow-Origin *;
        }
        location /hlsme {
                open_file_cache off;
                types {
                        application/vnd.apple.mpegurl m3u8;
                }
                root /var/tmp;
                add_header Cache-Control no-cache; # Prevent caching of HLS fragments
                add_header Access-Control-Allow-Origin *; # Allow web player to access our playlist
        }

I have http://kalapi in a couple of places, since it’s http only as a test server. I think the two sections at the end are new (location /dashme, location /hlsme) relative to the old thread.

ssl.conf (probably not needed, since it’s http only, but I figured I would post)

server {
    listen     8443   ssl;
    server_name server.ca;

    ssl_certificate      /etc/ssl/certs/device.crt;
    ssl_certificate_key  /etc/pki/tls/private/device.key;

    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  5m;

    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers   on;

    include /etc/nginx/conf.d/kaltura.conf;
}

This appears to match the one in the other thread.

delivery_profiles ->

+------+------------+---------------------+---------------------+------------------------------------------------+------+------------------------------------------------+------------------------------------------------+--------------------------------------+-----------------------+------------+-----------+--------+-----------------+-------------------+------------+-----------+------------------------------------+----------+
| id   | partner_id | created_at          | updated_at          | name                                           | type | system_name                                    | description                                    | url                                  | host_name             | recognizer | tokenizer | status | media_protocols | streamer_type     | is_default | parent_id | custom_data                        | priority |
+------+------------+---------------------+---------------------+------------------------------------------------+------+------------------------------------------------+------------------------------------------------+--------------------------------------+-----------------------+------------+-----------+--------+-----------------+-------------------+------------+-----------+------------------------------------+----------+
| 1001 |          0 | 2016-04-28 09:11:04 | 2016-04-28 09:11:04 | Kaltura HLS segmentation                       |   61 | Kaltura HLS segmentation                       | Kaltura HLS segmentation                       | http://server.ca:88/hls  | server.ca | NULL       | NULL      |      0 | NULL            | applehttp         |          1 |         0 | NULL                               |        0 |
| 1002 |          0 | 2016-04-28 09:11:04 | 2016-04-28 09:11:04 | Kaltura HDS segmentation                       |   63 | Kaltura HDS segmentation                       | Kaltura HDS segmentation                       | http://server.ca:88/hds  | server.ca | NULL       | NULL      |      0 | NULL            | hdnetworkmanifest |          1 |         0 | NULL                               |        0 |
| 1003 |          0 | 2016-04-28 09:11:04 | 2016-04-28 09:11:04 | Kaltura DASH segmentation                      |   68 | Kaltura DASH segmentation                      | Kaltura DASH segmentation                      | http://server.ca:88/dash | server.ca | NULL       | NULL      |      0 | NULL            | mpegdash          |          1 |         0 | NULL                               |        0 |
+------+------------+---------------------+---------------------+------------------------------------------------+------+------------------------------------------------+------------------------------------------------+--------------------------------------+-----------------------+------------+-----------+--------+-----------------+-------------------+------------+-----------+------------------------------------+----------+

Those values all appear to be correct I think.

I did add: Header set Access-Control-Allow-Origin “*” in the kaltura.conf for apache as you mentioned in the other thread, but that didn’t seem to fix it either. I double checked to make sure port 88 is open (80, 88, 443, 1935, 1936, 2812) are all open.

I think that’s all that I could find in the other thread. If I missed anything, I’m happy to gather + post.

I think I’ve fixed this and wanted to report back in case anyone finds this thread in the future. On my test server I add a hosts entry that points to 127.0.0.1 for the servername (so everything works), but when configuring I set the IP range to a smaller block. localhost falls outside that block, which is why my requests were generating errors.

I’m going to run through the update again from the start and ensure that is correct.

@jess : when I run kaltura-config-all.sh, I get an error about the elasticsearch (added in 14.7?) daemon not being found after it gets installed. Is this something I should worry about?

Update went through OK, so that problem is solved. I thought perhaps elastic search was simply a port not being opened, but it seems there’s an error when I try to start it:

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2019-07-31 15:57:58 MDT; 8s ago
     Docs: http://www.elastic.co
  Process: 27123 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 27123 (code=exited, status=1/FAILURE)

Jul 31 15:57:57 server.ca elasticsearch[27123]: 2019-07-31 15:57:57,656 main ERROR Null object returned for RollingFile in Appenders.
Jul 31 15:57:57 server.ca elasticsearch[27123]: 2019-07-31 15:57:57,656 main ERROR Null object returned for RollingFile in Appenders.
Jul 31 15:57:57 server.ca elasticsearch[27123]: 2019-07-31 15:57:57,656 main ERROR Null object returned for RollingFile in Appenders.
Jul 31 15:57:57 server.ca elasticsearch[27123]: 2019-07-31 15:57:57,656 main ERROR Unable to locate appender "rolling" for logger config "root"
Jul 31 15:57:57 server.ca elasticsearch[27123]: 2019-07-31 15:57:57,657 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexi...wlog.index"
Jul 31 15:57:57 server.ca elasticsearch[27123]: 2019-07-31 15:57:57,657 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Jul 31 15:57:57 server.ca elasticsearch[27123]: 2019-07-31 15:57:57,657 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Jul 31 15:57:58 server.ca systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 31 15:57:58 server.ca systemd[1]: Unit elasticsearch.service entered failed state.
Jul 31 15:57:58 server.ca systemd[1]: elasticsearch.service failed.

Any idea what might be causing that?

Hi @craig.jamieson,

Start by taking a look at the logs under /opt/kaltura/log/elasticsearch.
Please note that ES is not mandatory and the base platform can function without it at present time but it does provide far better searching capabilities and will eventually be utilised in KMC as well.

Cheers,

Hi @jess,

Thanks for the words of encouragement. I have made some progress by checking out the logs. I didn’t have the correct permissions set on the log or data folders, but I can’t recall which one, since I reset them both. log - /opt/kaltura/log/elasticsearch data - /opt/kaltura/var/lib/elasticsearch/tmp

Update: Upon more investigation, I’ve found that /opt/kaltura/var/lib/elasticsearch/tmp gets its permissions set properly, but elasticsearch won’t run until I set elasticsearch as the owner on /opt/kaltura/var/lib/elasticsearch as well. Permissions did get properly applied to the /opt/kaltura/var/lib/elasticsearch/tmp folder. The log folder (/opt/kaltura/log/elasticsearch) seems to get set to kaltura:apache as the other files/folders in the log folder.

After fixing that I got the kaltura-es-config.sh script to complete, when it does the command - curl -f “http://127.0.0.1:9200/_cluster/health?wait_for_status=green&timeout=10s” it does return success after a couple of 404/405 errors. I do see an entry in the log file where it changes its status from yellow to green.

I did run into a small issue with it not being set to automatically run. I think it runs the chkconfig part of the if statement rather than the systemctl portion. I fixed that manually thereafter.

How do I know that it’s running correctly afterwards? If I reboot and try running the curl command again, I only ever get a 408 error, so I’m worried that it’s not running correctly. I do see a status change from red to yellow, but never a new one suggesting it’s moved to green.

I’m also curious about the kaltura-elastic-populate service that seems to get started at the end. I didn’t notice an entry to get it to run automatically, so I was curious if it’s supposed to run all the time.

Hi @craig.jamieson,

The kaltura-elastic-populate is the parallel of kaltura-populate for Sphinx. In a clustered ENV, it needs to run on the ES nodes in order to sync the data from the operational MySQL DB to the ES indices (same as kaltura-populate does for Sphinx).

In order to sync previously created objects, you need to run the scripts under /opt/kaltura/app/deployment/base/scripts/elastic/.

Cheers,