Upgrading Kaltura CE from 11.5.0 to 13.0

daniel_mueller_1 · August 9, 2017, 1:25pm

Hello Community,

It’s time to upgrade my Kaltura CE Installation to it’s newest Release 13.0.0.
Last time I did upgrade, I did run in some problems:

After fixing my broken Kaltura I was a bit careful to upgrade again, like “Never touch a running System”
But now I’t time to go over to a new release and before I do I’d like to ask if I can simply folow the upgrade steps that are descibed below or if I have to consider something else before?

github.com

kaltura/platform-install-packages/blob/master/doc/install-kaltura-redhat-based.md#upgrade-kaltura

# Installing Kaltura on a Single Server (RPM)
This guide describes RPM installation of an all-in-one Kaltura server and applies to all major RH based Linux distros including Fedora Core, RHEL, CentOS, etc.
([Note the supported distros and versions](http://kaltura.github.io/platform-install-packages/#supported-distros)).

[Kaltura Inc.](http://corp.kaltura.com) also provides commercial solutions and services including pro-active platform monitoring, applications, SLA, 24/7 support and professional services. If you're looking for a commercially supported video platform  with integrations to commercial encoders, streaming servers, eCDN, DRM and more - Start a [Free Trial of the Kaltura.com Hosted Platform](http://corp.kaltura.com/free-trial) or learn more about [Kaltura' Commercial OnPrem Edition™](http://corp.kaltura.com/Deployment-Options/Kaltura-On-Prem-Edition). For existing RPM based users, Kaltura offers commercial upgrade options.

#### Table of Contents
[Prerequites](pre-requisites.md)

[Pre-Install Steps](install-kaltura-redhat-based.md#pre-install-steps)

[Non-SSL Step-by-step Installation](install-kaltura-redhat-based.md#non-ssl-step-by-step-installation)

[Apache SSL Step-by-step Installation](install-kaltura-redhat-based.md#apache-ssl-step-by-step-installation)

[Nginx SSL Configuration](nginx-ssl-config.md)

[Unattended Installation](install-kaltura-redhat-based.md#unattended-installation)

[Live Streaming with Nginx and the RTMP module](install-kaltura-redhat-based.md#live-streaming-with-nginx-and-the-rtmp-module)

This file has been truncated. show original

What are your experiences with Upgrading Kaltura to a newer release?

Thanks in advance,
Daniel

jess · August 10, 2017, 1:50pm

Hi Daniel,

The upgrade howto you referenced includes all the steps you need to perform when upgrading.
As with any complex server side product, upgrades can get complicated, we do our very best to simplify and test the process continuously but there are always edge cases. My best recommendation to avoid downtime is to first do so on a staging ENV. If you’ll have issues post upgrading, do post them and I’ll gladly help you solve them.
In general, I recommend performing upgrades at small intervals, you don’t have to upgrade every other week, but the longer you wait between upgrades, the bigger the delta and risk.

Thanks,

daniel_mueller_1 · August 10, 2017, 5:11pm

@jess

Hi jess,
Thanks for the suggestion. Today I’ve tried the upgrade and it seems that it worked almost fine.
Previously I did create some Publisher Users and they’re all here if I log in to the admin console. For each Publisher I did also create some subusers (KMC Users) who work together with the same media content pool. It seems that all of the subusers are gone. The list is empty --> See Screenshot .Is there a way to restore them? Where are they stored and is it needed to save them before the upgrade?

Another question I have is if there is a plan to replace the management by “Flash” in the Management Console regarding to a future release.
See https://techcrunch.com/2017/07/25/get-ready-to-say-goodbye-to-flash-in-2020/

Thanks in advance,
Daniel

jess · August 10, 2017, 5:52pm

Hi @daniel_mueller_1,

Do you see these users in the kuser table?

mysql> select * from kaltura.kuser\G

if so, try running:
# php /opt/kaltura/app/deployment/base/scripts/populateSphinxKusers.php
and make sure it finishes w/o errors, if there are errors, please paste them here.

As for KMC, see my reply here: KMCng instead of flash kmc

daniel_mueller_1 · August 11, 2017, 10:17am

@jess
Thank you so much. With a query of that SQL statement, I could see all the users that were missing.
The php command worked fine without any errors and now I can see them again

It seems that I have a similar issue described in the topic here:

If there is a recommendation how to set the permissions on /content/generatedUiConf, I’ll correct them.
Creating a new player in the universal studio worked fine.

If I should find anything else, I’ll post again

Thanks Jess for your precious support

daniel_mueller_1 · August 11, 2017, 11:01am

@jess

I did run the sanity check. Looks mostly fine as passed, but there are three errors:

[kaltura_logo_animated_blue.flv - 0_v7r0kq4p status] [FAILED, RC: 1] - [-.417988825]
Napping 30 seconds to allow mail to be sent out…
[Found an email sending entry for mb-agsstr01.gib.bs@kaltura.com[PID is 115] in /var/log/maillog] [PASSED, RC: 0] - [30.050433933]
Testing analytics, be patient…

DWH cycle] [FAILED, RC: 255] - [22.027554228]`

[Upload content kaltura_logo_animated_green.flv] [PASSED, RC: 0] - [1.328202913]
[Upload bulk using CSV] [PASSED, RC: 0] - [.624513846]
[Upload bulk using XML] [PASSED, RC: 0] - [.590370439]
[Create player] [PASSED, RC: 0] - [.349613337]
[Generate thumb] [FAILED, RC: 255] - [.268686404]
[Delete partner] [PASSED, RC: 0] - [.636003718]

Batch deamon is running.
After tracing the issue with the following command:

grep -A 1 -B 1 --color “ERR:|PHP|trace|CRIT|[error]” /opt/kaltura/log/kaltura_batch.log /opt/kaltura/log/batch/*log

I get the following output:

/opt/kaltura/log/kaltura_batch.log-2017-08-11 12:08:26 [0.000000] [734248497] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1502380172
/opt/kaltura/log/kaltura_batch.log:PHP Warning: file_put_contents(/opt/kaltura/app/cache//batch/config.ini): failed to open stream: Permission denied in /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php on line 185
/opt/kaltura/log/kaltura_batch.log-2017-08-11 12:08:27 [0.142494] [734248497] [2] [BATCH] [KGenericScheduler->loadConfig] INFO: 2.0.2
–
/opt/kaltura/log/kaltura_batch.log-2017-08-11 12:09:09 [0.099489] [734248497] [147] [BATCH] [KGenericScheduler->initSingleWorker] INFO: Initilizing KAsyncTagIndex
/opt/kaltura/log/kaltura_batch.log:2017-08-11 12:09:09 [0.000509] [734248497] [148] [BATCH] [KProcessWrapper->init] DEBUG: Now executing [setsid /usr/bin/php /opt/kaltura/app/batch/…/plugins/tag_search/lib/batch/tag_index/KAsyncTagIndexExe.php eJzFVW1v2jAQ7k+Z8nFSScJ7w5e2jKqsG1RANWlfkBMfYOHYmeOs0Kr/fWfHpl3LNGmaVCEROJ/vnnvunss0aUZJcDPPNkArTtSClNuhFCu2DpK4mTyW5js4+Xjyo4IK5uwBggFLogHae9a+FFDqORREES1VYA6SoGGeJrBxKCHTTIrXPol9dpzPlhWjnQZBy2CQ2vjNOu+ScC7vv0rKViwjJpD1iNHjrHZggsLOo3K2TFZCG1uzj8Z+baREk2BA0GbqMvBysmN5lY92kFUm9ILlYGAhqrgZRZH53U0CAzzHtNMkRuDfEeXSc1SH+i9Q47dIY9sAJFpDXnxiCpmUan9L9MaSh9BCWehwS7iuFAl1XgSDJ8QTP7M6gZ0ea1AWTzCYvOgcl4QCndftcUfuJvhekCRKHp+em2nuXDEOI6WkmmtlruFx3EoCEGsmYFq4wt+JLOSElAXWu0DKriSnYAeuFb3lKqwdl1Vhqnov5s7sGH6W6ULeVpwv5JCgGC1mnMJ6Bo1MVubujpV6BppB6VTUChysw/FYIOafhDuHjn22fZJyhNFnlXCnsRVqEjDqEna6dUK0rRTDSvh+QmpRmB7P4NS25YMmawsBA+t9AT7JzUW5F9mCrMd189CK5ZeZYoX2Y9vFOI1GWPAK56UMMRKuCKKyTchZGqZE4y9jtInC3yOiUhvFpnCZhUd2PHPTFj0WpSYiOzBmSz6zotYC1Jg6Ck5jp3bcVwq05QPNhLajVbuFnzTrZrTT6bRXaUxbKWn1YtK3l0zvs0pxsz5kpX3v+pE/LOv16rNhN7u9dtT2rB6OD1SjBxJcahXFjTVLG2l54BKbyzK4U7bBRkwbrYskDI/441jQ+43nvdl+JQEu1yGeN/DpWaR+xww3ufQz0et1HOrshbXbbXtMGGQkSMqBOo5t4ShPjYQ8SHEYn1GlZAHhJSjOhPNhgump4HuvflO7glxqmGOpoFylhsbzy4vF8Hp5N/ty7k2mwZRDvbgndWc1vsPGz8vjn2VtV4p72R2ZsKejoq/fmccHEocLJXhNUFWgvkm1rXfTn+xo9n/+tkd+AbtJZ+s= ‘[1744517670]’ >> /opt/kaltura/log/batch//tagindex-0-2017-08-11.log 2>> /opt/kaltura/log/batch//tagindex-0-2017-08-11.err.log]
/opt/kaltura/log/kaltura_batch.log-2017-08-11 12:09:10 [1.001702] [734248497] [149] [BATCH] [KGenericScheduler->shouldExecute] DEBUG: Worker [KAsyncTagIndex] id [560] running batches [1] max instances [1]
–
/opt/kaltura/log/batch/clearcuepoints-0-2017-08-11.err.log:PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘couldn’t connect to host’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
/opt/kaltura/log/batch/clearcuepoints-0-2017-08-11.err.log:Stack trace:
/opt/kaltura/log/batch/clearcuepoints-0-2017-08-11.err.log-#0 /opt/kaltura/app/batch/client/KalturaClient.php(4284): KalturaClientBase->doQueue()
–
/opt/kaltura/log/batch/kschedulehelper-0-2017-08-11.log-2017-08-11 12:07:58 [0.000099] [1041408163] [7] [BATCH] [KalturaClientBase->doCurl] NOTICE: post: {“format”:“3”,“ignoreNull”:true,“clientTag”:“batch: agsstr01.gib.bs KScheduleHelper index: 0 sessionId: 1041408163”,“apiVersion”:“3.3.0”,“partnerId”:"-1",“ks”:“YTkxMzMwNWExNjFhNThjMmJmYWNjMDNmMzE4ZTMyODZiZWRiODVkOHwtMTs7MTUwNTAzODA3ODsyOzE1MDI0NDYwNzguODkzNjtiYXRjaFVzZXI7ZGlzYWJsZWVudGl0bGVtZW50Oy0xOw==”,“kalsig”:“06bfd6d6cfc05f00a7b44f879e3c5f64”}
/opt/kaltura/log/batch/kschedulehelper-0-2017-08-11.log:2017-08-11 12:07:58 [0.000531] [1041408163] [8] [BATCH] [KScheduleHelper->run] ERR: exception ‘Exception’ with message ‘System is not yet ready - ping failed’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
/opt/kaltura/log/batch/kschedulehelper-0-2017-08-11.log:Stack trace:
/opt/kaltura/log/batch/kschedulehelper-0-2017-08-11.log-#0 /opt/kaltura/app/batch/batches/KScheduleHelper.class.php(42): KalturaLog::err(‘System is not y…’)

Do you have any suggestions how to resolve that?
Thanks in advance

jess · August 11, 2017, 12:05pm

Hi @daniel_mueller_1,

Permissions of the /opt/kaltura/app/cache seem to be wrong.

# chown -R apache.kaltura /opt/kaltura/app/cache
# find /opt/kaltura/app/cache/ -type f -exec chmod 660 {} \;
# find /opt/kaltura/app/cache/ -type d -exec chmod 775 {} \;

Also, you did not mention whether we’re on RHEL/CentOS or Debian/Ubuntu, though I believe it’s the former… if it’s Debian or Ubuntu, then instead of:

# chown -R apache.kaltura /opt/kaltura/app/cache
it should be:
# chown -R www-data.kaltura /opt/kaltura/app/cache

This should straighten things up.

For the UI conf dir, you can do the same on /opt/kaltura/web/content/generatedUiConf.

daniel_mueller_1 · August 11, 2017, 3:51pm

@jess

Hi Jess, I did set the permissions as described, Thank you so much for your help

If I run the kaltlog command, I can see two other errors and one of them is constantly repeating:

==> /opt/kaltura/log/kaltura_apache_errors.log <==
[Fri Aug 11 14:28:32 2017] [error] [client 127.0.0.1] File does not exist: /opt/kaltura/app/alpha/web/localthumb
[Fri Aug 11 14:34:20 2017] [error] [client 127.0.0.1] File does not exist: /opt/kaltura/app/alpha/web/localthumb

And that’s the error that is constantly repeating:

0 /opt/kaltura/app/plugins/schedule/bulk_upload/BulkUploadSchedulePlugin.php(237): kPluginableEnumsManager::apiToCore(‘BulkUploadObjec…’, ‘scheduleBulkUpl…’)

–
2017-08-11 16:21:52 [0.000545] [127.0.0.1] [377205948] [28] [API] [KalturaStatement->execute] DEBUG: Sql took - 0.00031208992004395 seconds
2017-08-11 16:21:52 [0.000427] [127.0.0.1] [377205948] [29] [API] [kCoreException->__construct] ERR: exception ‘kCoreException’ with message ‘Dynamic enum not found [scheduleBulkUpload.SCHEDULE_EVENT] for type [BulkUploadObjectType]’ in /opt/kaltura/app/alpha/lib/kPluginableEnumsManager.php:167

Any ideas how i can fix them too?

Another issue was send to me by a user. He did work with categories, but as you can see on the screenshot on the left there are no more categories displayed:

If I click on a video I can see that the category is still there:
categorie

If I try to recreate this category, I got the message: “Category allready exist”, but it’s not visible on the left side…

Is there a way to get this categories back?

Thanks in advance

aquileasfx1 · August 11, 2017, 4:20pm

Hello @daniel_mueller_1

you have to run sphinx to see the categories in the left panel

you can do that by running the following:

rm -f /opt/kaltura/log/sphinx/data/*
cd /opt/kaltura/app/deployment/base/scripts/
for i in populateSphinx*;do php $i >/tmp/$.log;done

have a good day

jess · August 11, 2017, 4:40pm

Actually, better use this wrapper /opt/kaltura/bin/kaltura-sphinx-reindex.sh:)
As for the other issue:

See Internal Server Error in KMC after 11.0.4 Upgrade

daniel_mueller_1 · August 11, 2017, 7:45pm

@jess and @aquileasfx1
Thanks, that’s firstclass community work!
After running the kaltura-sphinx-reindex script, all the categories got restored and are available now!

For the other issue I’ve followed the suggestions from the other thread! It seems that the generate.php script went trough.

If I run again the kaltlog I can see some permission errors with the batch process job. A video that was uploaded is still queued and did not go over to the converting process. I’ll post a screenshot (For some reason copy/past from my terminal session don’t work…)

kaltlog

I’m confident that we’ll also resolve this issue.
Thanks in advance

PS: If the content from the screenshot is unreadable try to click on that link:
https://forum.kaltura.org/uploads/default/original/2X/0/05bf1d4101cf5def61b356aa21c1ba25c3fd2e78.JPG

jess · August 14, 2017, 9:56am

Hi @daniel_mueller_1,

According to your output, the permissions on /opt/kaltura/app/cache are still incorrect.
Please correct them in accordance to what I wrote here:

daniel_mueller_1 · August 14, 2017, 10:46am

@jess
Hi Jess, I did recheck and set the permission command over /opt/kaltura/app/cache again as you’ve described.
Now it seems that the permission error is gone, but if I rerun the kaltlog command, I got a bunch of new errors that I havn’t seen before. Im slightly confused

==> /opt/kaltura/log/kaltura_apache_errors.log <==
[Mon Aug 14 09:47:30 2017] [error] [client 127.0.0.1] File does not exist: /opt/kaltura/app/alpha/web/localthumb
[Mon Aug 14 09:47:30 2017] [error] [client 127.0.0.1] File does not exist: /opt/kaltura/app/alpha/web/localthumb
[Mon Aug 14 09:47:30 2017] [error] [client 127.0.0.1] File does not exist: /opt/kaltura/app/alpha/web/localthumb
[Mon Aug 14 10:19:39 2017] [error] [client 127.0.0.1] File does not exist: /opt/kaltura/app/alpha/web/localthumb
[Mon Aug 14 10:19:39 2017] [error] [client 127.0.0.1] File does not exist: /opt/kaltura/app/alpha/web/localthumb
[Mon Aug 14 10:19:39 2017] [error] [client 127.0.0.1] File does not exist: /opt/kaltura/app/alpha/web/localthumb

/opt/kaltura/log/batch/bulkuploadcloser-0-2017-08-13.err.log
PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Empty reply from server’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
Stack trace:
/opt/kaltura/app/batch/client/KalturaClient.php(1364): KalturaClientBase->doQueue()

==> /opt/kaltura/log/batch/extractmedia-0-2017-08-14.err.log <==
PHP Warning: Division by zero in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 281
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Warning: Division by zero in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282

Any ideas what I can do next?
Thanks in advance

jess · August 14, 2017, 11:24am

Hi @daniel_mueller_1,

In order to understand why you’re getting a fatal error in /opt/kaltura/log/batch/bulkuploadcloser-0-2017-08-13.err.log, you need to open that log file and look at the lines leading to the final error but… do you even use dropfolders? if not, then it’s not that critical to debug.

As for the notices in /opt/kaltura/log/batch/extractmedia-0-2017-08-14.err.log, same thing. Need to open the log file and locate the root cause.

However, is the batch daemon now up and running? are you able to transcode new videos successfully? are there any other apparent issues at the moment?

daniel_mueller_1 · August 14, 2017, 1:08pm

@jess
Hi thanks for your suggestions. I’ve never used dropfolders, so I think that I can ignore this error.
Batch deamon is up and running.

I did upload two videos and I was able to transcode them successfully

If I should find any other issues, I’ll post again but for the moment it looks fine for me!

Thanks for your first class support

jess · August 14, 2017, 1:10pm

Hi @daniel_mueller_1,

Most welcome && happy videoing,

daniel_mueller_1 · February 20, 2018, 1:48pm

@jess

Yesterday I did upgrade from Kaltura 13.0.0 to the newest release 13.13.0
This time mostly everything went through fine.

I’ve still have a question.
According to the “kaltlog” output I got the following error:

==> /opt/kaltura/log/batch/extractmedia-0-2018-02-20.err.log <==
PHP Warning: Division by zero in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 281
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Notice: Trying to get property of non-object in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282
PHP Warning: Division by zero in /opt/kaltura/app/infra/media/mediaInfoParser/KMediaFileComplexity.php on line 282

What does this Division by zero error mean?

I’m also interessted to setup kmc-ng. In which release of Kaltura CE is it planed to implent it instead of a Flash based KMC?

jess · March 13, 2018, 3:03pm

Hi @daniel_mueller_1,

Sorry for the delayed response.
The issues you posted above were due to a missing directive in /opt/kaltura/app/configurations/batch/batch.ini.
I’ve just fixed this and it will be included in the next release [13.16.0].
However, you can already manually patch the config file, simply add:

params.ffmpegCmd                                    = /opt/kaltura/bin/ffmpeg

Right after:

209 [KAsyncExtractMedia : JobHandlerWorker]
210 id                                                  = 60
211 friendlyName                                        = Extract Media
212 type                                                = KAsyncExtractMedia
213 scriptPath                                          = batches/ExtractMedia/KAsyncExtractMediaExe.php
214 params.mediaInfoCmd=/opt/kaltura/bin/mediainfo
215 params.ffprobeCmd                                   = /opt/kaltura/bin/ffprobe

in /opt/kaltura/app/configurations/batch/batch.ini

No need to restart the kaltura-batch daemon after doing so.

In regards to KMCng, no ETA yet I’m afraid. That said, you’re welcome to manually deploy it and start testing.
See the repo’s README for instructions and in particular:

Thanks,