Updating 11.13 to 12.0 Help

Community Forums Backend Server Support
1

I’ve attempted to run an update again on my kaltura server, but I’ve run into a few issues. Initially, when running /opt/kaltura/bin/kaltura-db-update.sh, the db update fails. When I look in the file /opt/kaltura/app/deployment/sql_updates, I see:

/opt/kaltura/app/deployment/updates/sql/2014_01_19_category_entry_add_privacy_context.sql
/opt/kaltura/app/deployment/updates/sql/2013_10_22_add_drm_policy_table.sql
/opt/kaltura/app/deployment/updates/sql/2014_03_04_update_drm_device_table.sql
/opt/kaltura/app/deployment/updates/sql/2014_03_20_add_root_id_batch_job_lock.sql
/opt/kaltura/app/deployment/updates/sql/2014_01_22_create_delivery_table.sql

which all seem to be very old updates and have already been completed. Since they were completed, the db-update script fails. This doesn’t seem to be a big deal (the tables appear to be changed already), but I thought it’s worth mentioning.

After performing the next step (/opt/kaltura/bin/kaltura-config-all.sh) with the answer file I used for the original install, the sanity test fails (create partner fails, which prevents dependent tests from running).

Running kaltlog lists these two errors (or similar) over and over:
==> /opt/kaltura/log/batch/dropfoldercontentprocessor-0-2016-08-03.err.log <==
PHP Fatal error: Class ‘KalturaWorkerQueueFilter’ not found in /opt/kaltura/app/batch/batches/KJobHandlerWorker.class.php on line 265

PHP Warning: file_put_contents(/opt/kaltura/app/cache//batch/controls/KAsyncTransformMetadata.0.run): failed to open stream: Permission denied in /opt/kaltura/app/batch/scheduler/KScheduleHelperManager.class.php on line 199

Trying to access the admin console throws this error:
==> /opt/kaltura/log/kaltura_apache_errors_ssl.log <==
[Wed Aug 03 09:12:28.304275 2016] [:error] [pid 20494] [client 129.128.36.159:55331] PHP Fatal error: Class ‘Kaltura_Client_Enum_PermissionName’ not found in /opt/kaltura/app/plugins/admin_console/kaltura_internal_tools/admin/KalturaInternalToolsSystemHelperAction.php on line 26, referer: https://[servername_removed]/start/index.php

I thought perhaps something was wrong with the DB, so I dropped it (/opt/kaltura/bin/kaltura-drop-db.sh) and re-ran the install. However, the same errors persist when running kaltlog.

I’d be very grateful for any help. It seems like there’s a path missing or not getting set. For example, there are a few files that define the KalturaWorkerQueueFilter class, but presumably none are getting included from a missing config somewhere.

2

Hi,

Running /opt/kaltura/bin/kaltura-db-update.sh is no longer needed.
I updated the documentation just now.
Dropping the DB means you lost all the data… unless it is what you intended, I suggest you reimport it.

As for your actual problem, it starts with bad permissions on /opt/kaltura/app/cache.
It should be:
# ls -ald /opt/kaltura/app/cache/
drwxrwxr-x 7 kaltura apache 4096 Aug 3 10:00 /opt/kaltura/app/cache

so, directory is 775 and both the kaltura and apache user should be able to write to it.

Start by removing all cached files [be sure NOT to remove directories] by using:
find /opt/kaltura/app/cache/ -type f -exec rm {} ;
then, set permissions correctly with:
# chmod -R 775 /opt/kaltura/app/cache
# chown -R kaltura.apache /opt/kaltura/app/cache

Then remove the lock files:
# rm /opt/kaltura/app/base-config.lock
# rm /opt/kaltura/app/base-config-generator.lock

and rerun /opt/kaltura/bin/kaltura-config-all.sh and let’s see.

3

Thanks Jess! I nuked the DB as a last resort after reading about it in some other threads. I was working on a test server, so no harm done. The error persisted in the logs both before and after, so I’m optimistic that your solution will fix it (I nuked the DB, but not the /opt/kaltura directory). Thanks too for confirming that the db-update is no longer needed.

I’ll give it a try and report back.

4

Bit of progress, but something is still not quite right. Verified permissions on the cache directory and ran the command to remove files (directories were still there). Removed the lock files and re-ran the config. I’m not longer seeing the errors come up with kaltlog running, but the sanity test fails on the Create Partner step. All other tests up to that point pass.

I noticed while running the sanity test that it downloads php5full, which I don’t recall seeing before (but I could be wrong). The output on the “Space on /opt/kaltura/web” step shows:

INFO: Downloading ready-made schema from: /opt/kaltura/web/content/clientlibs/KalturaClient.xml
INFO: Generating from api version: 3.3.0, generated at: 03-08-2016
INFO: Now generating: php5full using Php5ClientGenerator
INFO: Delete old files [/opt/kaltura/web/content/clientlibs/php5full, /opt/kaltura/app/clients/php5]
INFO: Generate client library [php5full]
INFO: Saving client library to [/opt/kaltura/web/content/clientlibs/php5full]
INFO: Trying to package
DEBUG: Changing dir to [/opt/kaltura/web/content/clientlibs/php5full]
INFO: Executing: tar -czf “…/php5full_03-08-2016.tar.gz” …/php5full
tar: Removing leading `…/’ from member names
INFO: Package created successfully: …/php5full_03-08-2016.tar.gz
DEBUG: Restoring dir to [/opt/kaltura/app/generator]
INFO: php5full generated successfully

I am able to login to the admin console successfully again. So definitely a step in the right direction.

5

Hi,

The output generating the clientlibs is fine.
Need to understand why the partner creation is failing. I suggest you run it with:
# /opt/kaltura/bin/create_partner.php <admin_secret> <service_url>
where admin_secret is what you get out of:
mysql> select admin_secret from partner = -2;

the rest is the new partner email, a passwd for it and the endpoint URL for your CE installation.

Try it and see what the error you get is and we can take it from there.

6

Looks like perhaps the upgrade went well. It’s a cert error. The test server has a fake cert as you would expect. When I did the config-all part, it does output a note detecting a self-signed certificate and turns off verifyssl. Is there somewhere else that I can turn it off (or check) to let it bypass the check?

Here’s the output:

PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘Peer’s certificate has an invalid signature.’ in /opt/kaltura/web/content/clientlibs/php5/KalturaClientBase.php:362
Stack trace:
#0 /opt/kaltura/web/content/clientlibs/php5/KalturaClient.php(6552): KalturaClientBase->doQueue()
#1 /opt/kaltura/bin/create_partner.php(19): KalturaSessionService->start(‘d0b9abd97e6dfe0…’, NULL, 2, -2, NULL, NULL)
#2 {main}
thrown in /opt/kaltura/web/content/clientlibs/php5/KalturaClientBase.php on line 362

7

Hi Jess, sorry to drudge up an old thread, but I still haven’t quite been able to update successfully. I had to abandon this for a little bit (getting ready for new term), but I’ve tried updating to 12.0. I fiddled around for awhile (permissions are good now), but after uploading a test video, it gets stuck in conversion. To make sure it wasn’t anything to do with the fake cert I was using, when I updated, I set it to use http instead of https when running kaltura-config-all.sh for the update.

When I run the sanity test, here’s some output from the content upload part (kaltura_logo_animated_blue.flv). It uploads OK, but it never digests:

2016-09-07 09:05:57 [0.000193] [127.0.0.1] [871097114] [437] [API] [kFlowHelper::generateThumbnailsFromFlavor] ERR: exception ‘Exception’ with message ‘getConversionProfile2ForEntry Error: Cannot find conversion profile for entry_id [0_taoivhyx] OLD conversion_quality [] NEW conversion_profile_2_id [6]’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/batch2/kFlowHelper.php(1487): KalturaLog::err(‘getConversionPr…’)

2016-09-07 09:05:58 [0.000441] [127.0.0.1] [1595588211] [36] [API] [KalturaStatement->execute] DEBUG: Sql took - 0.00036501884460449 seconds
2016-09-07 09:05:58 [0.000148] [127.0.0.1] [1595588211] [37] [API] [kCoreException->__construct] ERR: exception ‘kCoreException’ with message ‘Dynamic enum not found [scheduleBulkUpload.SCHEDULE_EVENT] for type [BulkUploadObjectType]’ in /opt/kaltura/app/alpha/lib/kPluginableEnumsManager.php:167
Stack trace:
#0 /opt/kaltura/app/plugins/schedule/bulk_upload/BulkUploadSchedulePlugin.php(237): kPluginableEnumsManager::apiToCore(‘BulkUploadObjec…’, ‘scheduleBulkUpl…’)

#12 {main}
2016-09-07 09:05:58 [0.000088] [127.0.0.1] [1595588211] [38] [API] [kEventsManager::raiseEvent] ERR: exception ‘kCoreException’ with message ‘Dynamic enum not found [scheduleBulkUpload.SCHEDULE_EVENT] for type [BulkUploadObjectType]’ in /opt/kaltura/app/alpha/lib/kPluginableEnumsManager.php:167
Stack trace:
#0 /opt/kaltura/app/plugins/schedule/bulk_upload/BulkUploadSchedulePlugin.php(237): kPluginableEnumsManager::apiToCore(‘BulkUploadObjec…’, ‘scheduleBulkUpl…’)

2016-09-07 09:05:58 [0.001598] [127.0.0.1] [172077606] [29] [API] [KalturaStatement->execute] DEBUG: Sql took - 0.0015230178833008 seconds
2016-09-07 09:05:58 [0.000151] [127.0.0.1] [172077606] [30] [API] [kCoreException->__construct] ERR: exception ‘kCoreException’ with message ‘Dynamic enum not found [scheduleBulkUpload.SCHEDULE_EVENT] for type [BulkUploadObjectType]’ in /opt/kaltura/app/alpha/lib/kPluginableEnumsManager.php:167
Stack trace:
#0 /opt/kaltura/app/plugins/schedule/bulk_upload/BulkUploadSchedulePlugin.php(237): kPluginableEnumsManager::apiToCore(‘BulkUploadObjec…’, ‘scheduleBulkUpl…’)

#12 {main}

When manually uploading, I see the following in the log:

2016-09-07 09:12:05 [0.000130] [129.128.36.159] [30385258] [51] [PS2] [KExternalErrors::dieError] ERR: exception ‘Exception’ with message ‘exiting on error 6 - requested flavor was not found’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)

        [tmp_name] => /opt/kaltura/web/tmp/phpFWUief
        [error] => 0
        [size] => 37201805

–
[tmp_name] => /opt/kaltura/web/tmp/phpFWUief
[error] => 0
[size] => 37201805

2016-09-07 09:12:06 [0.000325] [129.128.36.159] [1820171709] [144] [API] [KalturaStatement->execute] DEBUG: Sql took - 0.00025296211242676 seconds
2016-09-07 09:12:06 [0.000227] [129.128.36.159] [1820171709] [145] [API] [kCoreException->__construct] ERR: exception ‘kCoreException’ with message ‘Dynamic enum not found [scheduleBulkUpload.SCHEDULE_EVENT] for type [BulkUploadObjectType]’ in /opt/kaltura/app/alpha/lib/kPluginableEnumsManager.php:167
Stack trace:
#0 /opt/kaltura/app/plugins/schedule/bulk_upload/BulkUploadSchedulePlugin.php(237): kPluginableEnumsManager::apiToCore(‘BulkUploadObjec…’, ‘scheduleBulkUpl…’)

#28 {main}
2016-09-07 09:12:06 [0.000088] [129.128.36.159] [1820171709] [146] [API] [kEventsManager::raiseEvent] ERR: exception ‘kCoreException’ with message ‘Dynamic enum not found [scheduleBulkUpload.SCHEDULE_EVENT] for type [BulkUploadObjectType]’ in /opt/kaltura/app/alpha/lib/kPluginableEnumsManager.php:167
Stack trace:
#0 /opt/kaltura/app/plugins/schedule/bulk_upload/BulkUploadSchedulePlugin.php(237): kPluginableEnumsManager::apiToCore(‘BulkUploadObjec…’, ‘scheduleBulkUpl…’)

It looks like it’s going off to the database and not finding something it’s looking for. I can’t really tell whether it’s the missing flavor, conversion profile, or that it’s unable to schedule the conversion event. Perhaps the three altogether are a sign of a table permissions issue.

I did go back and try uploading on the original 11.13 install and everything works well there.

8

Hi @craig.jamieson,

The main problem here is:
exception ‘kCoreException’ with message ‘Dynamic enum not found [scheduleBulkUpload.SCHEDULE_EVENT] for type [BulkUploadObjectType]’ in /opt/kaltura/app/alpha/lib/kPluginableEnumsManager.php:167

Try:
# php /opt/kaltura/app/deployment/updates/update.php -i -d >> /opt/kaltura/log/kalt_up.log 2>&1
# php /opt/kaltura/app/deployment/updates/update.php -i -s >> /opt/kaltura/log/kalt_up.log 2>&1
# php /opt/kaltura/app/deployment/base/scripts/installPlugins.php >> /opt/kaltura/log/kalt_up.log 2>&1

And let’s see how we are afterwards.

9

BTW, any reason why you’re upgrading to 11.19 specifically? the latest stable is now 12.0.0…

10

Thanks @jess! It’s an older thread where I was trying to update to 11.19, but couldn’t quite get it completed before the start of term. Since re-trying, I have been using 12.0. I’ll see if I can update the thread title to the new version.

Made some progress, but not quite there yet. I ran the sanity test afterwards, but it appears that it wipes out the log files, so I can’t attach the contents unfortunately. Much of what it said was that updates had already been run, but it did improve the file uploading. It now works, although the DWH test says it failed.

While on upload test, kaltlog picked up a couple of things:

2016-09-08 08:27:59 [0.000255] [127.0.0.1] [1956918863] [389] [API] [kFlowHelper::generateThumbnailsFromFlavor] ERR: exception ‘Exception’ with message ‘getConversionProfile2ForEntry Error: Cannot find conversion profile for entry_id [0_ibb2jqi0] OLD conversion_quality [] NEW conversion_profile_2_id [6]’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/batch2/kFlowHelper.php(1487): KalturaLog::err(‘getConversionPr…’)

2016-09-08 08:28:09 [0.000109] [127.0.0.1] [433830930] [59] [API] [KalturaObject->generateFromObjectClass] ALERT: exception ‘Exception’ with message ‘getter for property customData was not found on object kConvertJobData’ in /opt/kaltura/app/infra/log/KalturaLog.php:65
Stack trace:
#0 /opt/kaltura/app/api_v3/lib/types/KalturaObject.php(153): KalturaLog::alert(‘getter for prop…’)

2016-09-08 08:29:16 [0.000200] [127.0.0.1] [440597435] [526] [API] [kBatchManager::createFlavorAsset] ERR: exception ‘Exception’ with message ‘Flavor [3] is none-comply’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/batch2/kBatchManager.php(88): KalturaLog::err(‘Flavor [3] is n…’)

I also saw this error pop up, so maybe there’s some config to be done yet:

Thu Sep 8 08:27:52 2016 (22531): Fatal Error Zend OPcache cannot allocate buffer for interned strings

While the following output was produced during the DWH test:

2016-09-08 08:31:15 [0.004681] [127.0.0.1] [1651039176] [4] [API] [KalturaFrontController->getExceptionObject] ERR: exception ‘KalturaAPIException’ with message ‘Missing parameter “event”’ in /opt/kaltura/app/api_v3/lib/KalturaRequestDeserializer.php:191
Stack trace:
#0 /opt/kaltura/app/api_v3/lib/KalturaDispatcher.php(82): KalturaRequestDeserializer->buildActionArguments(Array)

2016-09-08 08:31:15 [0.004681] [127.0.0.1] [1651039176] [4] [API] [KalturaFrontController->getExceptionObject] ERR: exception ‘KalturaAPIException’ with message ‘Missing parameter “event”’ in /opt/kaltura/app/api_v3/lib/KalturaRequestDeserializer.php:191
Stack trace:
#0 /opt/kaltura/app/api_v3/lib/KalturaDispatcher.php(82): KalturaRequestDeserializer->buildActionArguments(Array)

        [tmp_name] => /opt/kaltura/web/tmp/phpsAZfNX
        [error] => 0
        [size] => 2743980

–
[tmp_name] => /opt/kaltura/web/tmp/phpsAZfNX
[error] => 0
[size] => 2743980

        [tmp_name] => /opt/kaltura/web/tmp/php5Dv511
        [error] => 0
        [size] => 5408

–
[tmp_name] => /opt/kaltura/web/tmp/php5Dv511
[error] => 0
[size] => 5408

        [tmp_name] => /opt/kaltura/web/tmp/phpWaYNYC
        [error] => 0
        [size] => 4657

–
[tmp_name] => /opt/kaltura/web/tmp/phpWaYNYC
[error] => 0
[size] => 4657

11

Hi @craig.jamieson,

Looks as though you need to regenerate the Kaltura client libs.
# service httpd stop
# service kaltura-monit stop
# find /opt/kaltura/app/cache/ -type f -exec rm {} ;
# php /opt/kaltura/app/generator/generate.php
# service httpd start
# service kaltura-monit start

Should fix it up…

Do let me know.

12

Hi @jess. Thanks for taking the time to continue to assist me with the issue. It is greatly appreciated.

Things are a little bit cleaner: the Zend OPcache error is gone, as is the error about “getter for property…”

The others still persist and the DWH sanity test fails. The exception for Flavor none-comply lists flavors 3, 5, 6, and 7 if that helps.

The exception regarding “Saving conversion log: key [object_type:[4], object_id:[0_9ppf8sjs], version:[2], object_sub_type[2], partner_id[103]] already exists’” doesn’t seem as important since at least it means one entry already exists.

This error pops up towards the end of the DWH test, right before it fails:

2016-09-08 10:42:38 [0.007411] [127.0.0.1] [1423994710] [4] [API] [KalturaFrontController->getExceptionObject] ERR: exception ‘KalturaAPIException’ with message ‘Missing parameter “event”’ in /opt/kaltura/app/api_v3/lib/KalturaRequestDeserializer.php:191
Stack trace:
#0 /opt/kaltura/app/api_v3/lib/KalturaDispatcher.php(82): KalturaRequestDeserializer->buildActionArguments(Array)

        [tmp_name] => /opt/kaltura/web/tmp/php6taNYu
        [error] => 0
        [size] => 2743980

–
[tmp_name] => /opt/kaltura/web/tmp/php6taNYu
[error] => 0
[size] => 2743980

        [tmp_name] => /opt/kaltura/web/tmp/phpnBxBbz
        [error] => 0
        [size] => 5408

–
[tmp_name] => /opt/kaltura/web/tmp/phpnBxBbz
[error] => 0
[size] => 5408

        [tmp_name] => /opt/kaltura/web/tmp/php7ZkzRB
        [error] => 0
        [size] => 4657

–
[tmp_name] => /opt/kaltura/web/tmp/php7ZkzRB
[error] => 0
[size] => 4657

I don’t have a clustered environment or anything complex, so I don’t suspect that warning applies in this case. During the generate process, there were a few warnings. Everything else looked OK though. I’ll include them just in case they are helpful:

INFO: Now generating: batchClient using Php5ClientGenerator
WARN: Service [multicenters_filesyncimportbatch] not found
WARN: Service [playready_playreadydrm] not found
WARN: Service [drm_drmprofile] not found
WARN: Missing type [KalturaHuluDistributionField]
WARN: Missing type [KalturaYouTubeDistributionField]
WARN: Missing type [KalturaYouTubeDistributionFeedSpecVersion]
WARN: Missing type [KalturaYahooDistributionField]
WARN: Missing type [KalturaYahooDistributionProcessFeedActionStatus]
WARN: Missing type [KalturaVerizonVcastDistributionField]
WARN: Missing type [KalturaUverseClickToOrderDistributionField]
WARN: Missing type [KalturaUverseDistributionField]
WARN: Missing type [KalturaUnicornDistributionField]
WARN: Missing type [KalturaQuickPlayDistributionField]
WARN: Missing type [KalturaMsnDistributionField]
WARN: Missing type [KalturaMetroPcsDistributionField]
WARN: Missing type [KalturaFreewheelGenericDistributionField]
WARN: Missing type [KalturaDailymotionDistributionField]
WARN: Missing type [KalturaCrossKalturaDistributionField]
WARN: Missing type [KalturaAttUverseDistributionField]

13

Hi @craig.jamieson,

Happy to assist:)
In regards to DWH - the test at times fails due to timing but that does not necessarily mean there’s an actual problem.
Can you please go through the checks here:


and let me know as to the results?

Don’t drop the DB before we look at the results though, if you have data there, it would be a shame to lose it needlessly.
The exception for Flavor none-comply lists flavors 3, 5, 6, and 7 just means that the source file was not good enough to be converted into these flavours or else, that performing the conversion would not result in increased quality, so it was just skipped.

The warnings during the client generation are triggered because these plugins are not enabled in /opt/kaltura/app/configurations/plugins.ini but that’s fine, I don’t believe you use any of these functionalities, so the warning can be ignored.

Are we otherwise OK and are the other sanity tests [apart from DWH] pass successfully?

14

Hi @jess,

I went through the DWH process manually and can see the plays get updated on the other end. I checked the various DWH log files (hourly, daily, dims, retention) and no errors were flagged. All the DB items: locks table clean, apache logs show up in kalturadw_ds.files, events show up in katluradw.dwh_fact_events. The other sanity tests all pass (only DWH fails), so perhaps there’s nothing further to worry about.

Thanks for the extra info regarding the flavor none-comply errors, it makes sense that they aren’t really errors. The missing parameter “event” one is still a little troubling, but perhaps it only occurs during the test, since it seems to work fine outside.

I’m also curious about one other thing. When performing the various upgrade tests, I watched a little closer on the permissions in the /opt/kaltura/app/cache folder. I checked the file /opt/kaltura/app/cache/api_v3/classMap.cache, since it seems to be one of the first loaded in most operations. The permissions seemed to get flipped to apache:apache after running the kaltura-config-all.sh script. Prior to running it, they were set to 755 with kaltura:apache as the ownership. It seems that I need to run that script, then run the command to flip the ownership to kaltura:apache recursively. Any thoughts on why this might happen?