Hi @stanrate,
To debug, run
# kaltlog
from the server’s shell and look at the errors outputted to STDOUT and, in parallel, launch your browser’s dev tools and look for errors under the “Network” and “Console” tabs.
It would also help if you provide a sample URL where the player is embedded so I can take a look.
Hi,
I decided to try a fresh install of 12.15 on RHEL 6.8
It was a clean install and the “PHP error” that occurred during the 11.2 to 12.15 upgrade did not occur during the install.
The only sanity check error was:
[DWH cycle] [FAILED, RC: 255] - [31.777014468]
I logged into KMC and went into content and noticed there were five sample videos and six players.
I chose the “normal web quality” video and the HTML5 player.
When play was pressed, google Chrome network showed index.m3u8 with a status of (failed). After a moment, a second index.m3u8 with a status of (failed).
I let the spinning continue for a minute before pressing pause.
Same thing happened in firefox.
A network tcp trace showed the PC (running Chrome and Firefox) had exchanged HTTPS (port 443) packets and HTTP (port 88) packets with the RHEL server.
Since the web browsers logged failures for index.m3u8 packets, I changed the two “access_log” lines in /etc/nginx/conf.d/kaltura.conf from “off” to “on” and re-ran the tests, first test using Chrome and second using Firefox.
access_log had the following four entries (I changed my PC’s IP address to x.x.x.x):
x.x.x.x - - [26/May/2017:11:38:54 +1000] “\x16\x03\x01\x00\xC1\x01\x00\x00\xBD\x03\x03u\xE9\x08>;\x85qU\xDC\xBE\x13\x22c\xAD\x07\xBB\xF09\x93\xD8\x94\xB2U\xDC\xE6\x87\xA8\xA0\x93V\xCEf\x00\x00\x22\xC0+\xC0/\xC0,\xC00\xCC\xA9\xCC\xA8\xCC\x14\xCC\x13\xC0\x09\xC0\x13\xC0” 400 325 0.002 “-” “-” “-” - “-” “-” 23455 - - 0 “-” “-” “-” “-” “-” 25
x.x.x.x - - [26/May/2017:11:38:55 +1000] “\x16\x03\x01\x00\xC1\x01\x00\x00\xBD\x03\x03\xFC\x96\x118\xEAb\xFD\xCF\xE8{\xBEO1\xDE\xAAV\xD6\xC7\xAD\x90\xB8#I” 400 325 0.003 “-” “-” “-” - “-” “-” 23455 - - 0 “-” “-” “-” “-” “-” 26
x.x.x.x - - [26/May/2017:11:40:33 +1000] “\x16\x03\x01\x00\xBE\x01\x00\x00\xBA\x03\x03\x0F\x050\x9D\x0C\xB9,\x050\x83\x9E\xF5\xCD\xE8B\xDE\x1Aqg\xC9\xBD\x14\xED\x06\xA2N\x80^\xC7!\xAD\xE1\x00\x00\x16\xC0+\xC0/\xC0” 400 325 0.002 “-” “-” “-” - “-” “-” 23455 - - 0 “-” “-” “-” “-” “-” 27
x.x.x.x - - [26/May/2017:11:40:34 +1000] “\x16\x03\x01\x00\xBE\x01\x00\x00\xBA\x03\x03r\xC3O” 400 325 0.002 “-” “-” “-” - “-” “-” 23455 - - 0 “-” “-” “-” “-” “-” 28
These entries contain http 400 which explains why the web browsers reported failures.
I updated RHEL with the latest RPMs and rebooted and removed and installed Kaltura 12.15 again (this time under RHEL 6.9).
Got a clean install and only one sanity check error for DWH. Re-ran the Chrome and Firefox tests and got the same results and the access_log had the same http 400 entries.
kaltlog only shows errors for analytics which I understand is normal. Here is an example:
ERR: exception ‘KalturaAPIException’ with message ‘Action “trackevent” does not exists for service “analytics”’ in /data/kaltura/app/api_v3/lib/KalturaDispatcher.php:57
What should be done to get a Kaltura supplied sample video to play using the html5 player?
Any help would be much appreciated. Thanks.
Hi @stanrate,
Like I said before, would be much easier if you provided a sample URL.
If you cannot, then at least provide the full request that’s failing and the HTTP RC returned [at a certain point you mentioned HTTP 400 but I’m not sure on which request]. You can mask the actual domain if you want.
Much more important than the access log, is the error log, what do you see there? Also, what’s the value for internal_ip_range in /opt/kaltura/app/configurations/local.ini?
Lastly, please describe your topology further: are you behind a load balancer or is this an all in one instance?
From "A network tcp trace showed the PC (running Chrome and Firefox) had exchanged HTTPS (port 443) packets and HTTP (port 88) packets with the RHEL server."
I gather the Kaltura API endpoint is over SSL? if so, note that the Nginx endpoint [can be a few nodes behind an LB, of course] should also be configured with SSL, otherwise, the browser will block the playback on account of “mixed content”.
Please see this post for a full explanation on how things should work between the Apache [running the Kaltura Server] and Nginx [with the VOD module, responsible for serving the manifest and fragments]:
Hi Jess,
I changed the answer file and reconfigured Kaltura so that httpd and nginx both use http. The html5 player now works. Thus, there are no problems removing Kaltura 11.2 RPMs and installing Kaltura 12.15 on our RHEL6 host.
I now need to go back and try to find out why the upgrade from 11.2 to 12.15 using yum update ‘kaltura’ reports a PHP failure. Here is a bit of the log showing the failure:
Updating : kaltura-base-2.15.0-17.noarch 13/37
Starting ntpd:
Regenarating client libs… this will take up to 2 minutes to complete.
httpd is stopped
INFO: Downloading ready-made schema from: /data/kaltura/web/content/clientlibs/KalturaClient.xml
INFO: Generating from api version: 3.3.0, generated at: 28-05-2017
INFO: Now generating: testme using TestmeGenerator
INFO: Generate client library [testme]
INFO: Copy sources from [/data/kaltura/clients-generator/sources/testme]
INFO: Saving client library to [/data/kaltura/web/content/clientlibs/testme]
INFO: testme generated successfully
INFO: Now generating: testmeDoc using TestmeDocGenerator
INFO: Delete old files [/data/kaltura/web/content/clientlibs/testmeDoc, /data/kaltura/app/api_v3/web/testmeDoc]
INFO: Generate client library [testmeDoc]
INFO: Copy sources from [/data/kaltura/clients-generator/sources/testmeDoc]
INFO: Saving client library to [/data/kaltura/web/content/clientlibs/testmeDoc]
INFO: testmeDoc generated successfully
INFO: Now generating: batchClient using Php5ClientGenerator
WARN: Service [schedule_scheduleevent] not found
WARN: Service [schedule_scheduleresource] not found
WARN: Service [multicenters_filesyncimportbatch] not found
WARN: Service [playready_playreadydrm] not found
WARN: Service [drm_drmprofile] not found
WARN: Missing type [KalturaHuluDistributionField]
WARN: Missing type [KalturaYouTubeDistributionField]
WARN: Missing type [KalturaYouTubeDistributionFeedSpecVersion]
WARN: Missing type [KalturaYahooDistributionField]
WARN: Missing type [KalturaYahooDistributionProcessFeedActionStatus]
WARN: Missing type [KalturaVerizonVcastDistributionField]
WARN: Missing type [KalturaUverseClickToOrderDistributionField]
WARN: Missing type [KalturaUverseDistributionField]
WARN: Missing type [KalturaUnicornDistributionField]
WARN: Missing type [KalturaQuickPlayDistributionField]
WARN: Missing type [KalturaMsnDistributionField]
WARN: Missing type [KalturaMetroPcsDistributionField]
WARN: Missing type [KalturaFreewheelGenericDistributionField]
WARN: Missing type [KalturaFacebookDistributionField]
WARN: Missing type [KalturaDailymotionDistributionField]
WARN: Missing type [KalturaCrossKalturaDistributionField]
WARN: Missing type [KalturaAttUverseDistributionField]
INFO: Delete old files [/data/kaltura/web/content/clientlibs/batchClient, /data/kaltura/app/batch/client]
INFO: Generate client library [batchClient]
INFO: Copy sources from [/data/kaltura/clients-generator/sources/php5]
INFO: Copy tests from [/data/kaltura/clients-generator/sources/php5]
INFO: Saving client library to [/data/kaltura/web/content/clientlibs/batchClient]
INFO: batchClient generated successfully
INFO: Now generating: php5ZendClientAdminConsole using PhpZendClientGenerator
INFO: Delete old files [/data/kaltura/web/content/clientlibs/php5ZendClientAdminConsole, /data/kaltura/app/admin_console/lib/Kaltura/Client]
INFO: Generate client library [php5ZendClientAdminConsole]
INFO: Copy sources from [/data/kaltura/clients-generator/sources/zend]
INFO: Copy tests from [/data/kaltura/clients-generator/sources/zend]
INFO: Saving client library to [/data/kaltura/web/content/clientlibs/php5]
INFO: Trying to package
DEBUG: Changing dir to [/data/kaltura/web/content/clientlibs/php5]
INFO: Executing: tar -czf “…/php5_28-05-2017.tar.gz” …/php5
tar: Removing leading `…/’ from member names
INFO: Package created successfully: …/php5_28-05-2017.tar.gz
DEBUG: Restoring dir to [/data/kaltura/clients-generator]
INFO: php5 generated successfully
INFO: Now generating: php5ZendVarConsole using PhpZendClientGenerator
INFO: Delete old files [/data/kaltura/web/content/clientlibs/php5ZendVarConsole, /data/kaltura/app/var_console/lib/Kaltura/Client]
INFO: Generate client library [php5ZendVarConsole]
INFO: Copy sources from [/data/kaltura/clients-generator/sources/zend]
INFO: Copy tests from [/data/kaltura/clients-generator/sources/zend]
INFO: Saving client library to [/data/kaltura/web/content/clientlibs/php5ZendVarConsole]
INFO: php5ZendVarConsole generated successfully
INFO: Now generating: php5ZendHostedPages using PhpZendClientGenerator
INFO: Delete old files [/data/kaltura/web/content/clientlibs/php5ZendHostedPages]
INFO: Generate client library [php5ZendHostedPages]
INFO: Copy sources from [/data/kaltura/clients-generator/sources/zend]
INFO: Copy tests from [/data/kaltura/clients-generator/sources/zend]
INFO: Saving client library to [/data/kaltura/web/content/clientlibs/php5ZendHostedPages]
INFO: php5ZendHostedPages generated successfully
INFO: Now generating: testsClient using Php5ClientGenerator
PHP Fatal error: Uncaught exception ‘Exception’ with message ‘Missing type [KalturaInternalToolsSession]’ in /data/kaltura/clients-generator/lib/ClientGeneratorFromXml.php:268
Stack trace:
#0 /data/kaltura/clients-generator/lib/ClientGeneratorFromXml.php(320): ClientGeneratorFromXml->loadTypesRecursive(‘KalturaInternal…’)
#1 /data/kaltura/clients-generator/lib/ClientGeneratorFromXml.php(217): ClientGeneratorFromXml->loadActionTypes(Object(DOMElement))
#2 /data/kaltura/clients-generator/lib/ClientGeneratorFromXml.php(82): ClientGeneratorFromXml->loadExcludeList()
#3 /data/kaltura/clients-generator/lib/Php5ClientGenerator.php(6): ClientGeneratorFromXml->__construct(’/tmp/kaltura.ge…’, ‘php5’, Object(Zend_Config))
#4 [internal function]: Php5ClientGenerator->__construct(’/tmp/kaltura.ge…’, Object(Zend_Config))
#5 /data/kaltura/clients-generator/exec.php(247): ReflectionClass->newInstance(’/tmp/kaltura.ge…’, Object(Zend_Config))
#6 /data/kaltura/app/generator/generate.php(8): require_once(’/data/kaltura/c…’)
#7 {main}
in /data/kaltura/clients-generator/lib/ClientGeneratorFromXml.php on line 268
Starting monit: Starting Monit 5.21.0 daemon with http interface at [*]:2812
[ OK ]
httpd dead but subsys locked
Starting httpd:
Updating : kaltura-batch-12.15.0-1.noarch
I searched to see if anyone else had reported a similar PHP failure but could not find any hits. I have tried upgrading from 11.2 to 12.8, 12.10, 12.14 and 12.15 but the same PHP failure messages are logged every time and html5 playback in google chrome does not work. The Kaltura host is on a internal network. I suspect the PHP failure may be messing up the upgrade and may be the cause of html5 playback not working. What can be done to identify the reason why the PHP failure occurs and how can it be fixed?
Thanks.
Hi @stanrate,
During an upgrade, the following code should run as part of the kaltura-base %postinst phase:
php /opt/kaltura/app/deployment/updates/update.php -i -d >> /opt/kaltura/log/kalt_up.log 2>&1
php /opt/kaltura/app/deployment/updates/update.php -i -s >> /opt/kaltura/log/kalt_up.log 2>&1
service httpd stop
service kaltura-monit stop
find /opt/kaltura/app/cache/ -type f -exec rm {} \;
rm -f /opt/kaltura/app/base-config-generator.lock
php /opt/kaltura/app/generator/generate.php
service httpd start
service kaltura-monit start
php /opt/kaltura/app/deployment/base/scripts/installPlugins.php
I’m not sure what happened in your case but from your output, it seems it failed to run properly.
I suggest you run each of these commands manually, making sure they all exit with RC 0.
If one of these fails, please attach the full output [STDOUT and STDERR] and the RC and I’ll gladly help you debug further.
Hi Jess,
I tried to upgrade Kaltura again from 11.2 to 12.15 by running a yum command for each RPM that had to be updated or installed. Here are the commands I ran:
yum clean all
yum -y install php-pecl-memcache
yum -y update libzen
yum -y update kaltura-html5-studio
yum -y update kaltura-postinst
yum -y install kaltura-libmediainfo
yum -y install kaltura-mediainfo
yum -y update kaltura-sphinx
yum -y install php-process
yum -y install x265-libs
yum -y update kaltura-ffmpeg
yum -y update kaltura-nginx
yum -y update kaltura-monit
Instead of running “yum -y update kaltura-base”, I ran the following commands so that kaltura-base RPM would be updated but the post scripts would not run:
yum -y update kaltura-base --downloadonly --downloaddir=/tmp
rpm -Uv --nopost /tmp/kaltura-base-12.15.0-17.noarch.rpm
rpm -qp --scripts /tmp/kaltura-base-12.15.0-17.noarch.rpm
I then put the post scripts into post.sh and then ran “sh -vx post.sh”. Here is the contents of post.sh
if [ -r "/opt/kaltura/app/configurations/local.ini" -a -r "/opt/kaltura/app/configurations/base.ini" ];then
sed -i "s@^\(kaltura_version\).*@\1 = 12.15.0@g" /opt/kaltura/app/configurations/local.ini
echo "Regenarating client libs.. this will take up to 2 minutes to complete."
if service httpd status;then
service httpd stop
fi
# this is read by kaltura-sphinx-schema-update.sh to determine rather or not to run
touch /opt/kaltura/app/configurations/sphinx_schema_update
find /opt/kaltura/app/cache/ -type f -exec rm {} \;
rm -f /opt/kaltura/app/base-config-generator.lock
php /opt/kaltura/app/generator/generate.php
php /opt/kaltura/app/deployment/base/scripts/installPlugins.php
php /opt/kaltura/app/deployment/base/scripts/populateSphinxMetadata.php
find /opt/kaltura/app/cache/ /opt/kaltura/log /opt/kaltura/var/run -type d -exec chmod 775 {} \;
find /opt/kaltura/log -type f -exec chmod 664 {} \;
chown -R kaltura.apache /opt/kaltura/app/cache/ /opt/kaltura/log /opt/kaltura/var/run
chmod 775 /opt/kaltura/web/content
service kaltura-monit start
if rpm -q httpd >> /dev/null;then
if ! service httpd status;then
service httpd start
fi
fi
# we now need CREATE and DROP priv for 'kaltura' on kaltura.*
if [ -r /etc/kaltura.d/system.ini ];then
. /etc/kaltura.d/system.ini
# disbale Default_Akamai_HLS_direct since we want the nginx vod-module profile to be used [ID 1001, system_name: Kaltura HLS segmentation]
echo "update delivery_profile set is_default=0 where id=1 and system_name='Default_Akamai_HLS_direct';"|mysql -h$DB1_HOST -u $DB1_USER -p$DB1_PASS -P$DB1_PORT $DB1_NAME
echo "GRANT INSERT,UPDATE,DELETE,SELECT,ALTER,DROP,CREATE ON kaltura.* TO '$DB_USER'@'%';FLUSH PRIVILEGES;"|mysql -h$DB1_HOST -u $SUPER_USER -p$SUPER_USER_PASSWD -P$DB1_PORT
fi
php /opt/kaltura/app/deployment/updates/update.php -i -d >> /opt/kaltura/log/kalt_up.log 2>&1
php /opt/kaltura/app/deployment/updates/update.php -i -s >> /opt/kaltura/log/kalt_up.log 2>&1
php /opt/kaltura/app/deployment/base/scripts/installPlugins.php >> /opt/kaltura/log/kalt_up.log 2>&1
fi
post.sh ran file and a PHP error did not occur!
I then reverted the RHEL6.8 VM back to its original state using a VM snapshot and ran the above commands again but this time allowed yum to run the post script. The post script failed with the PHP error as it did when I had run “yum update ‘kaltura’”.
I then reverted the VM and re-ran the commands but used “rpm -U -vv /tmp/kaltura-base-12.15.0-17.noarch.rpm”. The post script failed.
I then reverted the VM and re-ran the first set of commands where I manually ran the post commands and they ran fine.
This does not make sense. When the kaltura-base post script commands are run by either yum update or by rpm -U, a PHP failure is logged but when rpm is used to install kaltura-base without running the post script commands and then I run them manually, a PHP failure does not occur.
I also discovered that when “yum update ‘kaltura’” is run and the PHP failure occurs, if yum reinstall kaltura-base is run, the PHP failure does not occur.
Since I wanted to see whether a clean upgrade would fix the problem, I ran the first series of commands where the kaltura-base post script was run by hand and then I ran “yum update ‘kaltura’” so that yum would install the rest of the RPMs in the correct sequence.
After looking at the console output, the only failure I could see was the following:
Cleanup : kaltura-nginx-1.8.0-9.x86_64 35/37
warning: /etc/nginx/nginx.conf saved as /etc/nginx/nginx.conf.rpmsave
Binary upgrade failed, please check nginx’s error.log
Cleanup : libzen-0.4.33-1.el6.x86_64
I ran the following:
nginx: [emerg] open() “/etc/nginx/nginx.conf” failed (2: No such file or directory)
nginx: configuration file /etc/nginx/nginx.conf test failed
Perhaps the “Binary upgrade failed” message is expected because kaltura-config-all.sh has not yet been run?
I then ran kaltura-config-all.sh with an answer file. The only errors were:
Some warning messages were logged - see below:
Running base config…
kaltura-base-12.15.0-17.noarch
Welcome to Kaltura Server 12.15.0 post install setup.
Checking MySQL version…
Ver 5.1.73 found compatible
Generating client libs…
This can take a few minutes to complete, see log at /opt/kaltura/log/generate.php.log.
Configuration of 12.15.0 finished successfully!
Running FrontEnd config…
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
kaltura-front-12.15.0-1.noarch
Enabling Apache config - apps.conf
Enabling Apache config - var.conf
Enabling Apache config - admin.conf
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
Stopping memcached: [ OK ]
Starting memcached: [ OK ]
Shutting down monit: [ OK ]
Starting monit: Starting Monit 5.21.0 daemon with http interface at [*]:2812
[ OK ]
Running Sphinx config…
kaltura-sphinx-2.2.1-20.x86_64
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
searchd (pid 16091 16090) is running…
Stopping searchd: Sphinx 2.2.1-id64-dev (r4097)
Copyright © 2001-2013, Andrew Aksyonoff
Copyright © 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file ‘/opt/kaltura/app/configurations/sphinx/kaltura.conf’…
stop: successfully sent SIGTERM to pid 16091
Backing up files to /opt/kaltura/sphinx.bck.1496117730. Once the upgrade is done and tested, please remove this directory to save space
Starting searchd: Sphinx 2.2.1-id64-dev (r4097)
Copyright © 2001-2013, Andrew Aksyonoff
Copyright © 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file ‘/opt/kaltura/app/configurations/sphinx/kaltura.conf’…
listening on all interfaces, port=9312
WARNING: index ‘kaltura_base’: no fields configured (use rt_field directive) - NOT SERVING
WARNING: index ‘kaltura_base_gt_in_charset’: no fields configured (use rt_field directive) - NOT SERVING
precaching index 'kaltura_entry’
precaching index 'kaltura_category’
precaching index 'kaltura_kuser’
precaching index 'kaltura_category_kuser’
precaching index 'kaltura_cue_point’
precaching index 'kaltura_entry_distribution’
precaching index 'kaltura_caption_item’
precaching index 'kaltura_tag’
precaching index 'kaltura_metadata’
precaching index 'kaltura_schedule_event’
precached 10 indexes in 0.025 sec
Stopping searchd: Sphinx 2.2.1-id64-dev (r4097)
Copyright © 2001-2013, Andrew Aksyonoff
Copyright © 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file ‘/opt/kaltura/app/configurations/sphinx/kaltura.conf’…
stop: successfully sent SIGTERM to pid 21917
Starting searchd: Sphinx 2.2.1-id64-dev (r4097)
Copyright © 2001-2013, Andrew Aksyonoff
Copyright © 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file ‘/opt/kaltura/app/configurations/sphinx/kaltura.conf’…
listening on all interfaces, port=9312
WARNING: index ‘kaltura_base’: no fields configured (use rt_field directive) - NOT SERVING
WARNING: index ‘kaltura_base_gt_in_charset’: no fields configured (use rt_field directive) - NOT SERVING
precaching index 'kaltura_entry’
precaching index 'kaltura_category’
precaching index 'kaltura_kuser’
precaching index 'kaltura_category_kuser’
precaching index 'kaltura_cue_point’
precaching index 'kaltura_entry_distribution’
precaching index 'kaltura_caption_item’
precaching index 'kaltura_tag’
precaching index 'kaltura_metadata’
precaching index 'kaltura_schedule_event’
precached 10 indexes in 0.024 sec
Running Batch config…
kaltura-batch-12.15.0-1.noarch
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
Reloading httpd:
Stopping memcached: [ OK ]
Starting memcached: [ OK ]
Starting monit: Starting Monit 5.21.0 daemon with http interface at [*]:2812
[ OK ]
kaltura-nginx-1.12.0-3.x86_64
Reloading nginx: [ OK ]
Running DWH config…
kaltura-dwh-12.14.0-1.noarch
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
The Kaltura DWH DB seems to already be installed.
DB creation will be skipped.
DWH configured.
====================================================================================================================
Setup completed successfully!
To access your Kaltura tools visit:
http://klomeo.austrac.gov.au
To verify the integrity of your deployment and that all components are fully configured and installed,
you can run the sanity tests using the following command:
/opt/kaltura/bin/kaltura-sanity.sh
Thank you for running Kaltura! To keep Kaltura viable, stable and tested, please join the community and help by contributing sanity tests that verify overall platform stability: http://bit.ly/kaltura-ci , and by contributing to the project roadmap by solving simple tasks and challenges: http://bit.ly/kaltura-tasks.
After that, the sanity check was run. The upload content failed to digest - see below:
[Space on /] [PASSED, RC: 0] - [.005106469]
[Space on /opt/kaltura/web] [PASSED, RC: 0] - [.004078938]
INFO: Downloading ready-made schema from: /data/kaltura/web/content/clientlibs/KalturaClient.xml
INFO: Generating from api version: 3.3.0, generated at: 30-05-2017
INFO: Now generating: php5full using Php5ClientGenerator
INFO: Delete old files [/data/kaltura/web/content/clientlibs/php5full, /data/kaltura/app/clients/php5]
INFO: Generate client library [php5full]
INFO: Copy sources from [/data/kaltura/clients-generator/sources/php5]
INFO: Copy tests from [/data/kaltura/clients-generator/sources/php5]
INFO: Saving client library to [/data/kaltura/web/content/clientlibs/php5full]
INFO: Trying to package
DEBUG: Changing dir to [/data/kaltura/web/content/clientlibs/php5full]
INFO: Executing: tar -czf “…/php5full_30-05-2017.tar.gz” …/php5full
tar: Removing leading `…/’ from member names
INFO: Package created successfully: …/php5full_30-05-2017.tar.gz
DEBUG: Restoring dir to [/data/kaltura/clients-generator]
INFO: php5full generated successfully
[Check httpd daemon status] [PASSED, RC: 0] - [.037684941]
Napping 1 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 2 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 3 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 4 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 5 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 6 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 7 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 8 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 9 seconds to allow the daemon httpd to finish its init after monit restarted it…
Napping 10 seconds to allow the daemon httpd to finish its init after monit restarted it…
[Check httpd daemon is started by Monit] [PASSED, RC: 0] - [10.445763639]
[check daemon httpd init status] [PASSED, RC: 0] - [.034518892]
[Check kaltura-sphinx daemon status] [PASSED, RC: 0] - [.043540208]
[Check kaltura-sphinx daemon is started by Monit] [PASSED, RC: 0] - [10.563596529]
[check daemon kaltura-sphinx init status] [PASSED, RC: 0] - [.004749978]
[Check kaltura-batch daemon status] [PASSED, RC: 0] - [.024983703]
Napping 1 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
Napping 2 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
Napping 3 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
Napping 4 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
Napping 5 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
Napping 6 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
Napping 7 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
Napping 8 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
Napping 9 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
Napping 10 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it…
[Check kaltura-batch daemon is started by Monit] [PASSED, RC: 0] - [10.328074876]
[check daemon kaltura-batch init status] [PASSED, RC: 0] - [.004509558]
[Check kaltura-monit daemon status] [PASSED, RC: 0] - [.020262914]
[check daemon kaltura-monit init status] [PASSED, RC: 0] - [.004018090]
[Check memcached daemon status] [PASSED, RC: 0] - [.025015370]
Napping 1 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 2 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 3 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 4 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 5 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 6 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 7 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 8 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 9 seconds to allow the daemon memcached to finish its init after monit restarted it…
Napping 10 seconds to allow the daemon memcached to finish its init after monit restarted it…
[Check memcached daemon is started by Monit] [PASSED, RC: 0] - [10.349627946]
[check daemon memcached init status] [PASSED, RC: 0] - [.004566884]
[kaltura-html5lib ver in KMC config.ini] [PASSED, RC: 0] - [.060610742]
[kaltura-kdp3 ver in KDP3 config.ini] [PASSED, RC: 0] - [.031845157]
[kaltura-kmc ver in KMC config.ini] [PASSED, RC: 0] - [.037226683]
[Get KMC SWFs] [PASSED, RC: 0] - [1.377584603]
[check_start_page] [PASSED, RC: 0] - [.038044794]
[check_testme_page] [PASSED, RC: 0] - [.009773557]
[check_kmc_index_page] [PASSED, RC: 0] - [.037369337]
[check_admin_console_index_page] [PASSED, RC: 0] - [.668384513]
[check_studio_index_page] [PASSED, RC: 0] - [.018343897]
[check_clipapp_index_page] [PASSED, RC: 0] - [.568013981]
[Create Partner] [PASSED, RC: 0] - [1.750939502]
[Local dropfolder creation] [PASSED, RC: 0] - [.557556230]
[Create flavor param] [PASSED, RC: 0] - [.490043604]
[Delete flavor param] [PASSED, RC: 0] - [.222607854]
[HTTP notification: ID: 29, URL: http://localhost/1.php] [PASSED, RC: 0] - [.490043604]
[Mail notification: ID: 30, Subject: Your video is ready to be played!, Mail body: Hello world:)] [PASSED, RC: 0] - [.490043604]
[Upload content kaltura_logo_animated_blue.flv] [PASSED, RC: 0] - [.862898361]
Napping 10 seconds to allow entry 0_8gdao93s to digest…
Napping 10 seconds to allow entry 0_8gdao93s to digest…
Napping 10 seconds to allow entry 0_8gdao93s to digest…
Napping 10 seconds to allow entry 0_8gdao93s to digest…
Napping 10 seconds to allow entry 0_8gdao93s to digest…
Napping 10 seconds to allow entry 0_8gdao93s to digest…
Napping 10 seconds to allow entry 0_8gdao93s to digest…
Napping 10 seconds to allow entry 0_8gdao93s to digest…
Napping 10 seconds to allow entry 0_8gdao93s to digest…
[kaltura_logo_animated_blue.flv - 0_8gdao93s status] [FAILED, RC: 1] - [-.513004702]
Sounds like there are more problems to investigate. What should we do next?
Thanks
I ran kaltlog and saw following messages:
There were wrong permissions. I found and followed these instructions:
================================================================
drwxrwxr-x 7 kaltura apache 4096 Aug 3 10:00 /opt/kaltura/app/cache
so, directory is 775 and both the kaltura and apache user should be able to write to it.
Start by removing all cached files [be sure NOT to remove directories] by using:
find /opt/kaltura/app/cache/ -type f -exec rm {} ;
then, set permissions correctly with:
Then remove the lock files:
This fixed the permissions but when I ran the sanity check, it unfixed the permissions and they were incorrect again - see below:
drwxrwxr-x. 14 apache kaltura 4096 May 30 16:11 /opt/kaltura/app/cache/
I fixed the permissions again and while the sanity check was running, I constantly checked permissions and found the sanity check made the permissions incorrect. The following log shows the sanity check run. Note that the permissions went wrong about the time that the sanity check logged the “php5full generated successfully” message.
[Space on /] [PASSED, RC: 0] - [.004580021]
[Space on /opt/kaltura/web] [PASSED, RC: 0] - [.003992865]
INFO: Downloading ready-made schema from: /data/kaltura/web/content/clientlibs/KalturaClient.xml
INFO: Generating from api version: 3.3.0, generated at: 30-05-2017
INFO: Now generating: php5full using Php5ClientGenerator
INFO: Delete old files [/data/kaltura/web/content/clientlibs/php5full, /data/kaltura/app/clients/php5]
INFO: Generate client library [php5full]
INFO: Copy sources from [/data/kaltura/clients-generator/sources/php5]
INFO: Copy tests from [/data/kaltura/clients-generator/sources/php5]
INFO: Saving client library to [/data/kaltura/web/content/clientlibs/php5full]
INFO: Trying to package
DEBUG: Changing dir to [/data/kaltura/web/content/clientlibs/php5full]
INFO: Executing: tar -czf “…/php5full_30-05-2017.tar.gz” …/php5full
tar: Removing leading `…/’ from member names
INFO: Package created successfully: …/php5full_30-05-2017.tar.gz
DEBUG: Restoring dir to [/data/kaltura/clients-generator]
INFO: php5full generated successfully
[Check httpd daemon status] [PASSED, RC: 0] - [.053135284]
What should I do to prevent the sanity check from mucking up permissions?
Thanks.
I ran the sanity check again and found the ownership changed when the following message was output:
INFO: php5full generated successfully
before the message was output, it was:
drwxrwxr-x. 14 kaltura apache 4096 May 30 16:47 /opt/kaltura/app/cache/
when the message was output, it changed to:
drwxrwxr-x. 14 apache kaltura 4096 May 30 16:47 /opt/kaltura/app/cache/
Can you tell me what causes the sanity check script to muck up the ownership?
Thanks
Hi @stanrate,
drwxrwxr-x. 14 kaltura apache 4096 May 30 16:47 /opt/kaltura/app/cache/
and
drwxrwxr-x. 14 apache kaltura 4096 May 30 16:47 /opt/kaltura/app/cache/
are essentially the same, in both cases, the apache user and the kaltura user have write permissions on /opt/kaltura/app/cache so both will work.
‘apache’ is the user running Apache and as a result, all the Kaltura Server code running under the Kaltura Apache VHost, the ‘kaltura’ user is used for several things [running Sphinx, Nginx and, most important in this context, the batch daemon]. So long as both have write access to the cache dir and its sub dirs, things should work.
If you’re still having problems when trying to ingest an entry, please start by looking for errors in /opt/kaltura/log/kaltura_batch.log and /opt/kaltura/log/batch/*
Additionally, got to Admin Console->Batch Process Control and input the entry ID. This will display a drill down of all batch jobs related to this entry, their status and, if applicable, the reason of failure.
Hi Jess,
After the following two commands were run, the sanity script was able to ingest the test video and I was able to upload a video via the KMC:
chmod -R 775 /opt/kaltura/app/cache
chown -R kaltura.apache /opt/kaltura/app/cache
The next step was to try to play a video in the KMC. Using the html5 player, I tried to play a 3 second video. It played fine, no problems. I then tried to play a longer video but spinning started and continued until the video was paused. During this time, kaltlog only showed the following messages every 10 seconds which I understand is ok because analytics does not work in the community edition.
#3 {main}
2017-05-31 11:48:53 [0.000900] [192.168.7.53] [9561039] [5] [API] [KalturaFrontController->getExceptionObject] ERR: exception ‘KalturaAPIException’ with message ‘Action “trackevent” does not exists for service “analytics”’ in /data/kaltura/app/api_v3/lib/KalturaDispatcher.php:57
Stack trace:
#0 /data/kaltura/app/api_v3/lib/KalturaFrontController.php(113): KalturaDispatcher->dispatch(‘analytics’, ‘trackEvent’, Array)
Before pressing “play”, I had started the following trace so I could see whether the player was getting VOD packets from nginx but tcpdump did not show any packets.
tcpdump -i any port 88
I had previously run tcpdump for port 80 and could see a lot of packets between my PC running google chrome and httpd port 80.
In google chrome “network”, I saw the following when the 3 second video was playing:
When the longer video was playing, network showed:
http://host.domain/p/101/sp/10100/playManifest/entryId/0_trkhgcsh/flavorIds/0_u1s0wplu,0_552bzbrq,0_7ncck1wp/format/applehttp/protocol/http/a.m3u8?referrer=aHR0cDovL2tsb21lby5hdXN0cmFjLmdvdi5hdQ==&ks=ZTYyNmZhNzEyOGNkMzE0M2RlMDRkMWQyNTNhODE0YWY5MWY2NTNjMnwxMDE7MTAxOzE0OTYyNzc4NzA7MjsxNDk2MTkxNDcwLjEzMDI7aXRfd2ViX29wc0BhdXN0cmFjLmdvdi5hdTsqLGRpc2FibGVlbnRpdGxlbWVudDs7&playSessionId=6b027139-7b09-684b-c6a7-db4f4ee1ba74&clientTag=html5:v2.55.2&preferredBitrate=464&uiConfId=23448188
http://host.domain/i/p/101/sp/10100/serveFlavor/entryId/0_trkhgcsh/v/2/flavorId/0_u1s0wplu/index_0_av.m3u8
The main difference between the 3 second video and the longer video was that instead of a.mp4, a.m3u8 was present in the network trace. The same trace entries were present when the videos were played using firefox.
kaltlog only shows messages for analytics every 10 seconds during the spinning when the longer video won’t play. Where do we go to from here?
Thanks.
Hi Jess,
Here is a screen shot showing network traffic. It was taken after about a minute of spinning.
Thanks.
Hi @stanrate,
Very short entries are served directly over HTTP, without the use of adaptive bitrate protocols. This is because for very short entries, there’s no added value to using HLS, HDS or DASH.
The duration threshold is controlled by the short_entries_max_duration directive in /opt/kaltura/app/configurations/base.ini.
This would explain why very short videos play correctly whereas longer ones do not.
By default, when the playback method is HLS, HDS or DASH, HLS being the default for entries longer than the value for short_entries_max_duration is set to, the Nginx VOD module is used to repackage the MP4 file as HLS and serve the manifest and segments.
Looking at your HTTP requests during playback, it looks as though the manifest and fragments are correctly delivered [requests return with HTTP 200] so to further debug this, I’m going to have to ask for a sample URL.
Also, do you see any errors in the “Console” tab when opening your browser’s dev tools util?
Hi Jess,
Thanks for all your help so far.
Unfortunately, the url can only be accessed within our organization. I had a closer look at what happens when we press play and noticed that google chrome console does indeed show an exception. We are running Version 53.0.2785.116 m of google chrome. This is what chrome console shows when play is pressed using KDP3 Dark skin player (I changed the actual hostname and domain to “host.domain”.
Uncaught (in promise) DOMException: The play() request was interrupted by a new load request.
Here is a screen shot:
When I try to play a 44 minute video using KDP3 Dark skin player, chrome seems to download 857MB and then fail. See following snapshot:
When I try to play a large video in firefox, after about 20 seconds or so, firefox reports an allocation size overflow - see below screen shot:
I reproduced the spinning problem in firefox just in case the problem might be caused by a chrome bug. The spinning occurs when I use our player which is a copy of the html5 player with minor changes as well as when I use the standard KDP3 Dark skin player.
Thanks.
Hi @stanrate,
First, let’s make sure you’re using an HTML5 player and not the legacy Flash one. Go to KMC->Studio and create a new player, then, in the preview and embed view, select the new player and try to play.
If it still does not work, please run:
mysql> select conf_file_path from ui_conf where id=$PLAYER_ID;
That should output something like:
conf_file_path: /content/generatedUiConf/23/454/ui_conf_23454793__2.xml
this is a path relative to /opt/kaltura/web
the file I’m really interested in will be of the same name except it’ll have config in it, so for example:
/opt/kaltura/web/content/generatedUiConf/23/454/ui_conf_23454793__2.config.xml
Paste the contents here.
Hi Jess
I created new player and tried it but still got spinning and the DOMexception.
Here is the info:
mysql> select conf_file_path from ui_conf where id=23449072;
±---------------------------------------------------------+
| conf_file_path |
±---------------------------------------------------------+
| /content/generatedUiConf/23/449/ui_conf_23449072__12.xml |
±---------------------------------------------------------+
1 row in set (0.00 sec)
ls -l /data/kaltura/web/content/generatedUiConf/23/449/ui_conf_23449072*
-rw-r-----. 1 apache kaltura 1747 Jun 1 10:58 /data/kaltura/web/content/generatedUiConf/23/449/ui_conf_23449072__12.config.xml
-rw-r-----. 1 apache kaltura 13523 Jun 1 10:56 /data/kaltura/web/content/generatedUiConf/23/449/ui_conf_23449072__12.xml
-rw-r-----. 1 apache kaltura 376 Jun 1 10:56 /data/kaltura/web/content/generatedUiConf/23/449/ui_conf_23449072__2.config.xml
{
“plugins”: {
“topBarContainer”: {
“plugin”: true
},
“controlBarContainer”: {
“plugin”: true
},
“scrubber”: {
“plugin”: true
},
“largePlayBtn”: {
“plugin”: true
},
“playHead”: {
“plugin”: true
},
“playPauseBtn”: {
“plugin”: true
},
“volumeControl”: {
“showSlider”: true,
“layout”: “horizontal”,
“pinVolumeBar”: false,
“accessibleControls”: false,
“accessibleVolumeChange”: 0.1,
“plugin”: true
},
“durationLabel”: {
“plugin”: true
},
“currentTimeLabel”: {
“plugin”: true
},
“keyboardShortcuts”: {
“volumePercentChange”: “0.1”,
“shortSeekTime”: “5”,
“longSeekTime”: “10”,
“volumeUpKey”: “38”,
“volumeDownKey”: “40”,
“togglePlaybackKey”: “32”,
“shortSeekBackKey”: “37”,
“shortSeekForwardKey”: “39”,
“openFullscreenKey”: “70”,
“closeFullscreenkey”: “27”,
“gotoBeginingKey”: “36”,
“gotoEndKey”: “35”,
“longSeekForwardKey”: “ctrl+39”,
“longSeekBackKey”: “ctrl+37”,
“percentageSeekKeys”: “49,50,51,52,53,54,55,56,57”,
“plugin”: true
},
“liveCore”: {
“plugin”: true
},
“liveStatus”: {
“plugin”: true
},
“liveBackBtn”: {
“plugin”: true
},
“fullScreenBtn”: {
“plugin”: true
},
“logo”: {
“href”: “http://www.kaltura.com”,
“title”: “Kaltura”,
“plugin”: true
},
“playersJsReceiver”: {
“plugin”: true
}
},
“uiVars”: [
{
“key”: “autoPlay”,
“value”: false,
“overrideFlashvar”: false
},
{
“key”: “autoMute”,
“value”: false,
“overrideFlashvar”: false
},
{
“key”: “enableTooltips”,
“value”: true,
“overrideFlashvar”: false
},
{
“key”: “adsOnReplay”,
“value”: true,
“overrideFlashvar”: false
}
],
“layout”: {
“skin”: “kdark”
}
Thanks.
Here is contents of the other config.xml in case it helps:
{“plugins”:{“topBarContainer”:[],“controlBarContainer”:[],“scrubber”:[],“largePlayBtn”:[],“playHead”:[],“playPauseBtn”:[],“volumeControl”:[],“durationLabel”:[],“currentTimeLabel”:[],“keyboardShortcuts”:[],“liveCore”:[],“liveStatus”:[],“liveBackBtn”:[],“fullScreenBtn”:[],“logo”:[],“playersJsReceiver”:[]},“uiVars”:{“autoPlay”:false,“autoMute”:false},“layout”:{“skin”:“kdark”}}
Since I and a colleague have spent a lot of time trying everything we can think of to get an upgrade from Kaltura 11.2 to several 12.x releases working without success on the RHEL6 host, I decided to try a fresh installation of the latest Kaltura on the latest CentOS7. A VM was created and CentOS 7 installed followed by all the latest patches. I then installed the latest Kaltura RPMs (Kaltura 12.16) and ran configure and sanity check. No problems. I tried to play one of the installation videos on the installed html5 player but it would not play via “View a standalone page with this player” until a new publisher had been created. After that I was able to use “View a standalone page with this player” to get a url that could be pasted into chrome or firefox to play the video. A new video was uploaded and it also played fine. A lot of testing was done in chrome and firefox and no problems were experienced. This morning I came into work and found the filesystem had become full because lots of logs had been written to the api log file. After adding several GB to the filesystem I rebooted so I could check that everything came up ok. I then ran the sanity check which completed successfully and tried to play a video. Spinning started and continued and chrome console showed the DOMexception that I saw after an 11.2 to 12.x upgrades. I don’t understand why videos would play yesterday and not today. Perhaps the out-of-space filesystem problem has triggered a problem that now prevents playback. kaltlog does not show any errors. This is the same as after an upgrade where kaltlog did not show any errors when the DOMexception was shown by chrome console. I showed my colleague that playback was no longer working and took a snapshot of the VM, reverted the VM to its state prior to kaltura installation and then installed 12.16 again. After the config and sanity check, I created a new publisher and was able to play videos. It’s probably only a matter of time before something triggers a problem that will again cause DOMexception to be shown and spinning to continue indefinitely. When playback was working, I had used tcpdump to see that TCP packets were being exchanged between chrome running on my PC and nginx on port 88 (as well as the usual TCP packets that are exchanged between the PC and httpd on TCP port 80). After the problem happened, tcpdump no longer showed any packets on TCP port 88 exchanged with nginx.
What information can I provide to help debug this issue?
Thanks.
Hi @stanrate,
I’m afraid I can’t help you further without access to the system. Can you please send me a private message so we can coordinate?
Hi Jess,
I replied to your email with additional information. Did you receive it?
I have investigated further using /opt/kaltura/log/kaltura_prod.log to try to understand what might be causing video playback to work in one 12.16 server but no longer in another 12.16 server, and also not in the 11.2 that was upgraded to 12.15
After checking entries in /opt/kaltura/log/kaltura_prod.log on the 3 systems, I found an SQL statement that queries the “delivery_profile” table. On the system that is working, the SQL statement includes “AND delivery_profile.STREAMER_TYPE=‘applehttp’”.
On the other two systems, this is missing.
The statement is very long so I’ll paste everything starting with “FROM” for these 3 systems:
WHERE delivery_profile.PARTNER_ID=‘0’ AND delivery_profile.IS_DEFAULT='1’
AND delivery_profile.STREAMER_TYPE='applehttp’
AND delivery_profile.TYPE NOT IN (‘10135’,‘10136’,‘1013’,‘1002’,‘1001’,‘1003’,‘1005’,‘1006’,‘1007’,‘1008’,‘1009’,‘1010’) AND delivery_profile.STATUS IN (‘0’,‘3’)
ORDER BY (delivery_profile.HOST_NAME is not null) DESC,(delivery_profile.TOKENIZER is not null) DESC,(delivery_profile.PARTNER_ID<>102) ASC,delivery_profile.PRIORITY ASC
WHERE delivery_profile.PARTNER_ID IN (‘0’,‘102’) AND delivery_profile.ID IN (‘1’)
AND delivery_profile.TYPE NOT IN (‘10135’,‘10136’,‘1013’,‘1002’,‘1001’,‘1003’,‘1005’,‘1006’,‘1007’,‘1008’,‘1009’,‘1010’) AND delivery_profile.STATUS IN (‘0’,‘3’)
WHERE delivery_profile.PARTNER_ID IN (‘0’,‘101’) AND delivery_profile.ID IN (‘1’)
AND delivery_profile.TYPE NOT IN (‘10128’,‘10129’,‘1013’,‘1002’,‘1001’,‘1003’,‘1005’,‘1006’,‘1007’,‘1008’,‘1009’,‘1010’) AND delivery_profile.STATUS IN (‘0’,‘3’)
Another log entry contains:
INFO: Delivery ID for partnerId [102] and streamer type [applehttp] is 1
and on another system it contains:
INFO: Delivery ID for partnerId [102] and streamer type [applehttp] is 1001
Thus, I suspect the problem is caused by the selection of an incorrect delivery id but I’m baffled by the differences in the sql statements. What causes the code to choose between running these two types of sql statements that I suspect leads to the choice of an incorrect delivery id? What can we do to fix the problem?
Thanks.
Hi @stanrate,
I did get your email, will review it shortly, thanks.
As for the delivery profiles, see:
This was first pushed into the 12.3.0 branch so if the ENVs in question were upgraded from earlier versions, it will account for what you’re seeing.
Simply execute an update statement like so:
mysql> update delivery_profile set is_default=0 where system_name='Default_Akamai_HLS_direct' and partner_id=0;
That should fix it.
The default profile for HLS is the one with the ID of 1001, for instance, on my ENV:
mysql> select * from delivery_profile where id=1001\G
*************************** 1. row ***************************
id: 1001
partner_id: 0
created_at: 2017-06-05 07:02:56
updated_at: 2017-06-05 07:02:56
name: Kaltura HLS segmentation
type: 61
system_name: Kaltura HLS segmentation
description: Kaltura HLS segmentation
url: http://ce-csi.dev.kaltura.com:88/hls
host_name: ce-csi.dev.kaltura.com
recognizer: NULL
tokenizer: NULL
status: 0
media_protocols: NULL
streamer_type: applehttp
is_default: 1
parent_id: 0
custom_data: NULL
priority: 0
1 row in set (0.00 sec)