Stuck in "converting" state

I just realized that a lot of content isn’t able to be properly viewed in the preview & embed screen, as well as is not showing up in my syndication feeds. This appears to have started over a week ago, when I was messing around with both my Default Transcoding Flavors, my thumbnail settings, and my remote storage. So I assume it’s related to one of those. Here’s a log of when I try to upload a video I’m trying to sort through right now:

For example, for the Transcoding Flavors I took off every flavor but the Source, I tried adding the 720p/2500 bitrate one but that didn’t seem to affect it any. There are also several custom thumbnail profiles in that trasncode profile as well.

2016-07-05 18:56:37 [0.000000] [588143800] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1465256427
PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with message ‘Section ‘ip-10-0-0-92’ cannot be found in /opt/kaltura/app/cache//batch/config.ini’ in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace:
#0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-92’, true)

[2016-07-05 18:53:26]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-05 18:55:37]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-05 18:57:47]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

        [tmp_name] => /opt/kaltura/web/tmp/phpyF0vI4
        [error] => 0
        [size] => 1055736


[tmp_name] => /opt/kaltura/web/tmp/phpyF0vI4
[error] => 0
[size] => 1055736

2016-07-05 18:58:48 [0.000000] [395842269] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1465256427
PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with message ‘Section ‘ip-10-0-0-92’ cannot be found in /opt/kaltura/app/cache//batch/config.ini’ in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace:
#0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-92’, true)


[2016-07-05 19:02:09]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

Hello,

Your first issue is that it seems your batch daemon is not running at all, I deduce this from:

2016-07-05 18:58:48 [0.000000] [395842269] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1465256427
PHP Fatal error: Uncaught exception 'Zend_Config_Exception' with message 'Section 'ip-10-0-0-92' cannot be found in /opt/kaltura/app/cache//batch/config.ini' in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150

in /opt/kaltura/app/configurations/batch/batch.ini, your should have something like:

[ce-rhel7.dev.kaltura.com : mainTemplate]
id      = 45866
name                                                                                            = Default Scheduler

in this example ce-rhel7.dev.kaltura.com is the hostname, the ‘id’ directive should have a unique value the config script sets. If your hostname is ip-10-0-0-92 then that’s what should be there.
It is possible you changed your hostname and did not run the reconfigure scripts, which would cause this.

If you’re using the RPMs, you should run, as root:
# /opt/kaltura/bin/kaltura-batch-config.sh
if debs, then:
# dpkg-reconfigure kaltura-batch

once the batch daemon is running again, try to upload a new entry and look at the log.

Here is my log, still having issues. The IP is incorrect, and something I thought I changed when I upgraded it a few weeks ago. Should I run this from the upgrade instructions again?

/opt/kaltura/bin/kaltura-db-update.sh
/opt/kaltura/bin/kaltura-config-all.sh

Logs when I ran the command:

[root@ip-10-0-0-92 ec2-user]# /opt/kaltura/bin/kaltura-batch-config.sh
kaltura-batch-11.14.0-1.noarch
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:

rm /opt/kaltura/app/base-config.lock

/opt/kaltura/bin/kaltura-base-config.sh

Reloading httpd:
Stopping memcached: [ OK ]
Starting memcached: [ OK ]
Starting monit: Starting Monit 5.13 daemon with http interface at [*]:2812
[ OK ]
[root@ip-10-0-0-92 ec2-user]#

Logs when I added an entry:

2016-07-06 14:05:22 [0.000000] [1815134738] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1467826548
PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with message ‘Section ‘ip-10-0-0-92’ cannot be found in /opt/kaltura/app/cache//batch/config.ini’ in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace:
#0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-92’, true)

[2016-07-06 14:06:32]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

2016-07-06 14:07:33 [0.000000] [198071463] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1467826548
PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with message ‘Section ‘ip-10-0-0-92’ cannot be found in /opt/kaltura/app/cache//batch/config.ini’ in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace:
#0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-92’, true)

[2016-07-06 14:08:43]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

2016-07-06 14:09:24 [0.006544] [45.59.236.158] [770093946] [53] [PS2] [KalturaStatement->execute] DEBUG: Sql took - 0.0059919357299805 seconds
2016-07-06 14:09:24 [0.000706] [45.59.236.158] [770093946] [54] [PS2] [KExternalErrors::dieError] ERR: exception ‘Exception’ with message ‘exiting on error 6 - requested flavor was not found’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)

        [tmp_name] => /opt/kaltura/web/tmp/phpvuCT6Q
        [error] => 0
        [size] => 21550038


[tmp_name] => /opt/kaltura/web/tmp/phpvuCT6Q
[error] => 0
[size] => 21550038

2016-07-06 14:09:43 [0.000000] [1537379074] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1467826548
PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with message ‘Section ‘ip-10-0-0-92’ cannot be found in /opt/kaltura/app/cache//batch/config.ini’ in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace:
#0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-92’, true)

[2016-07-06 14:10:54]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

2016-07-06 14:11:54 [0.000000] [1860952281] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1467826548
PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with message ‘Section ‘ip-10-0-0-92’ cannot be found in /opt/kaltura/app/cache//batch/config.ini’ in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace:
#0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-92’, true)

Is ip-10-0-0-92 resolvable from the machine itself? are you able to ping it for example? or run:
# host ip-10-0-0-92
and get an IP back?

No, I’m not sure why it had reverted to this, I’ve seen it before and it changed after I ran a config.

Anyway, here’s what I get:

[root@ip-10-0-0-92 ec2-user]# host ip-10-0-0-92
ip-10-0-0-92.us-west-2.compute.internal has address 10.0.0.92
[root@ip-10-0-0-92 ec2-user]#

OK, so, the hostname is clearly ip-10-0-0-92 and is resolvable, which is good.
Now, what do you have in /opt/kaltura/app/configurations/batch/batch.ini since it obviously does not have:

[ip-10-0-0-92 : mainTemplate]

what hostname does it have instead?

[root@ip-10-0-0-92 ec2-user]# /opt/kaltura/app/configurations/batch/batch.ini
bash: /opt/kaltura/app/configurations/batch/batch.ini: Permission denied

hmmm…which is weird since I have put in sudo -s

You’re trying to run it whereas what you should do is edit it.
Do:
# sudo vi /opt/kaltura/app/configurations/batch/batch.ini
or
# sudo nano /opt/kaltura/app/configurations/batch/batch.ini

or whatever text editor you use…

[ip-10-0-0-199 : mainTemplate]

aha…

So once I’ve updated the ini file (changed the 199 to 92),

then I run

/opt/kaltura/bin/kaltura-batch-config.sh

then try to upload another file? And see if it works?

UPDATE: Still stuck in converting…hmm maybe that bad ip is somewhere else? Should I rerun config?

The thing is, /opt/kaltura/bin/kaltura-batch-config.sh already does that for you, if you open it, you will see:

sed -i “s#@INSTALLED_HOSNAME@#hostname#” -i $BATCH_MAIN_CONF

that is, it looks for @INSTALLED_HOSNAME@ in /opt/kaltura/app/configurations/batch/batch.ini and replaces it with the output of the hostname command. My guess would be, that your hostname changed but, you did not re-run the kaltrua-base-config.sh script after it changed, therefore, /opt/kaltura/app/configurations/batch/batch.ini remained as it was after the original run of /opt/kaltura/bin/kaltura-batch-config.sh and so, the @INSTALLED_HOSNAME@ placeholder was no longer there to replace.

You should either:
# /opt/kaltura/bin/kaltura-base-config.sh
# /opt/kaltura/bin/kaltura-batch-config.sh

again or just manually change the hostname in /opt/kaltura/app/configurations/batch/batch.ini and:
# /etc/init.d/kaltura-batch restart

What concerns me about the second approach, although it will take less time [2 seconds vs. say, 2 minutes for the base-config script to end] is that you may have other places where you have the wrong hostname.

You can start by editing manually and restarting but if you run into additional problems, make sure to re-run all the config scripts or else just:
# rm /opt/kaltura/app/base-config.lock
# /opt/kaltura/bin/kaltura-config-all.sh

/opt/kaltura/bin/kaltura-config-all.sh runs all needed scripts automatically but is only the right way to go in case you have an all in 1 server, for a cluster, you need to run different scripts per node role.

starting to see some things happen now but I wonder if it’s all right, I still see some references to the IP even though I thought i changed them to our host. It removed all the assets that were still “converting” previously and some of my bulk upload jobs started coming through, but they’re all still converting. I’ll check back tomorrow buit here are some of the logs:


2016-07-06 20:07:06 [0.000409] [2076242471] [133] [BATCH] [KJobHandlerWorker::setCurrentJob] DEBUG: Start job[3596] type[10] sub-type[0] object[1] object-id[0_unkv1xx8] partner-id[108] dc[0] parent-id[] root-id[3596]
2016-07-06 20:07:06 [0.000521] [2076242471] [134] [BATCH] [KJobHandlerWorker->closeJob] ERR: exception ‘Exception’ with message ‘Timed out’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/batch/batches/KJobHandlerWorker.class.php(372): KalturaLog::err(‘Timed out’)

2016-07-06 20:07:21 [0.000405] [920341412] [13] [BATCH] [KJobHandlerWorker::setCurrentJob] DEBUG: Start job[3084] type[10] sub-type[0] object[1] object-id[0_53hh2goq] partner-id[108] dc[0] parent-id[] root-id[3084]
2016-07-06 20:07:21 [0.000592] [920341412] [14] [BATCH] [KJobHandlerWorker->closeJob] ERR: exception ‘Exception’ with message ‘Timed out’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/batch/batches/KJobHandlerWorker.class.php(372): KalturaLog::err(‘Timed out’)

2016-07-06 20:07:22 [0.000782] [45.59.236.158] [1662087766] [53] [PS2] [KalturaStatement->execute] DEBUG: Sql took - 0.00031208992004395 seconds
2016-07-06 20:07:22 [0.000641] [45.59.236.158] [1662087766] [54] [PS2] [KExternalErrors::dieError] ERR: exception ‘Exception’ with message ‘exiting on error 6 - requested flavor was not found’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)

2016-07-06 20:07:23 [0.000487] [920341412] [33] [BATCH] [KJobHandlerWorker::setCurrentJob] DEBUG: Start job[3206] type[10] sub-type[0] object[1] object-id[0_kt4zr3hj] partner-id[108] dc[0] parent-id[] root-id[3206]
2016-07-06 20:07:23 [0.000491] [920341412] [34] [BATCH] [KJobHandlerWorker->closeJob] ERR: exception ‘Exception’ with message ‘Timed out’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/batch/batches/KJobHandlerWorker.class.php(372): KalturaLog::err(‘Timed out’)

==> /opt/kaltura/log/batch/convertprofilecloser-0-2016-07-06.log <==
2016-07-06 20:07:24 [0.003501] [920341412] [54] [BATCH] [KJobHandlerWorker->closeJob] ERR: exception ‘Exception’ with message ‘Timed out’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/batch/batches/KJobHandlerWorker.class.php(372): KalturaLog::err(‘Timed out’)

        [tmp_name] => /opt/kaltura/web/tmp/phpnrl2Ax
        [error] => 0
        [size] => 21550038


[tmp_name] => /opt/kaltura/web/tmp/phpnrl2Ax
[error] => 0
[size] => 21550038

2016-07-06 20:08:03 [0.034740] [52.41.250.214] [1521434636] [27] [API] [kObjectSavedEvent->__construct] DEBUG: Event [kObjectSavedEvent] object type [PartnerLoad]
2016-07-06 20:08:03 [0.018027] [23.20.71.190] [1838114575] [3] [API] [KExternalErrors::dieError] ERR: exception ‘Exception’ with message ‘exiting on error 29 - the supplied feed is already being processed’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)

==> /opt/kaltura/log/kaltura_batch.log <==
PHP Warning: file_get_contents(/opt/kaltura/app/cache//batch/controls/KAsyncBulkUpload.0.run): failed to open stream: No such file or directory in /opt/kaltura/app/batch/scheduler/KScheduleHelperManager.class.php on line 174
PHP Stack trace:
PHP 1. {main}() /opt/kaltura/app/batch/KGenericBatchMgr.class.php:0
PHP 2. KGenericScheduler->run() /opt/kaltura/app/batch/KGenericBatchMgr.class.php:38
PHP 3. KGenericScheduler->loop() /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php:195
PHP 4. KGenericScheduler->handleRunningBatches() /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php:225
PHP 5. KScheduleHelperManager::loadRunningBatches() /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php:238
PHP 6. file_get_contents() /opt/kaltura/app/batch/scheduler/KScheduleHelperManager.class.php:174


[2016-07-06 20:08:05]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:09:16]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:10:26]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

2016-07-06 20:10:45 [0.063846] [45.59.236.158] [1783911643] [53] [PS2] [KalturaStatement->execute] DEBUG: Sql took - 0.00063014030456543 seconds
2016-07-06 20:10:45 [0.000737] [45.59.236.158] [1783911643] [54] [PS2] [KExternalErrors::dieError] ERR: exception ‘Exception’ with message ‘exiting on error 6 - requested flavor was not found’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)

[2016-07-06 20:11:37]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:12:47]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:13:58]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:15:08]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:16:19]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:17:29]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:18:39]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:19:50]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:21:01]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

[2016-07-06 20:22:11]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

2016-07-06 20:23:11 [0.000807] [45.59.236.158] [2110787421] [53] [PS2] [KalturaStatement->execute] DEBUG: Sql took - 0.00033712387084961 seconds
2016-07-06 20:23:11 [0.000696] [45.59.236.158] [2110787421] [54] [PS2] [KExternalErrors::dieError] ERR: exception ‘Exception’ with message ‘exiting on error 6 - requested flavor was not found’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)

[2016-07-06 20:23:22]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

OK, well, now it seems your batch daemon is running but there are still plenty of problems.
I honestly would recommend just re-running all the config scripts as I explained in my previous post.
It will not drop your DB unless explicitly asked to but I suggest you back it up beforehand just in case.

But, nonetheless, as a learning experience, here is what I get from the lines you posted:
It seems the permissions on /opt/kaltura/app/cache/ are wrong.
What’s the output for:
# ll -d /opt/kaltura/app/cache/
?

it should be similar to:

drwxrwxr-x. 14 apache kaltura 4096 Jun 26 10:41 /opt/kaltura/app/cache/

if the directory is not owned by the user apache and group kaltura, correct it with:

# chown apache.kaltura  /opt/kaltura/app/cache/
# chmod 775 /opt/kaltura/app/cache/

and clear all cache files with:

find /opt/kaltura/app/cache/ -type f -exec rm {} \;

Be sure NOT to delete directories under /opt/kaltura/app/cache.

Currently, from your log it seems writing to /opt/kaltura/app/cache/batch fails. Make sure this dir exists and is currently permissoned.

Another issue is with:

[2016-07-06 20:23:22]exception 'Exception' with message 'Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.' in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:

once again, related to the hostname. You should have /opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini, what do you have instead?

There also seem to be a timeout issue when making requests, you’ll need to go to the actual log and look at the lines leading to it to understand what’s going on.

It was still the 199 ip instead of 92. I did rerun the /opt/kaltura/bin/kaltura-base-config.sh before my last post and things were still like that. I’ll rerun again.

Regarding logs, I have been going through them but it’s really difficult for me to know what to do in some circumstances, I am not sure what to refer to in an instance like this:

0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)


2016-07-06 20:07:23 [0.000487] [920341412] [33] [BATCH]
[KJobHandlerWorker::setCurrentJob] DEBUG: Start job[3206] type[10]
sub-type[0] object[1] object-id[0_kt4zr3hj] partner-id[108] dc[0]
parent-id[] root-id[3206]
2016-07-06 20:07:23 [0.000491] [920341412]
[34] [BATCH] [KJobHandlerWorker->closeJob] ERR: exception 'Exception’
with message ‘Timed out’ in
/opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:

Like yes, that to me says that the KJobHandlerWorker is closing because it timed out on object 0_kt4zr3hj, but I’m not sure what to do with that information. Anyway, I’ll rerun the config.

I keep breaking it more, is there a way I can look at all my current configuration stuff so I don’t accidentally break something when trying to fix the ip problems? Then I can safely revert it without having to go to my full backup, as well as make sure exactly what parts of the config that need fixing.

For example, when trying to upload now I get the error:
0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-239’, true)

I need to change that to the correct address, so where does that address live (in the config?) and how do I change it safely/individually. Is there anyway to figure this out on my own via documentation?

Other questions include why are these error messages getting truncated, it makes it even more difficult to debug.

#0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)

I tried adding more verbosity with
sed -i ‘s@^writers.(.).filters.priority.priority\s=\s*7@writers.\1.filters.priority.priority=4@g’ /opt/kaltura/app/configurations/logger.ini

it didn’t give me any message that it was or wasn’t activated, and I didn’t notice a different. Do I need to write these messages to a log to get a full error message?

So, when I update this in nano:

[ce-rhel7.dev.kaltura.com : mainTemplate]
id = 45866

to what is missing, (now it’s a different ip, ip-10-0-0-239), and save it, view the file again, confirm the saved change, then upload a file to KMC. It gives me the exact same error, and when I open /opt/kaltura/app/cache//batch/config.ini again, I see the old IP again…what gives?

Your problem is not the IP, although of course, for a server, you’d want a fix IP. Your problem is with the hostname. Why is it constantly changing? you need the machine to have one hostname that does not change every other minute. The configuration files have directives that rely on that. Much like any server side product really…

Okay, I think I resolved that one. It’s been changing whenever we’ve been restoring it after we break the box, but otherwise it should be static. Anyway, I’ve been fixing references to it manually that show up in my errors, and now it’s repeating constantly hundreds of times


2016-07-08 16:55:38 [0.000723] [52.41.250.214] [1799309173] [296] [API] [KalturaStatement->execute] DEBUG: Sql took - 0.00024604797363281 seconds
2016-07-08 16:55:38 [0.001237] [52.41.250.214] [1799309173] [297] [API] [KalturaFrontController->getExceptionObject] ERR: exception ‘KalturaAPIException’ with message ‘Scheduler id “47578” conflicts between hosts: “ip-10-0-0-239” and “ip-10-0-0-199”’ in /opt/kaltura/app/api_v3/services/BatchControlService.php:148
Stack trace:
#0 /opt/kaltura/app/api_v3/services/BatchControlService.php(249): BatchControlService->getOrCreateScheduler(Object(KalturaScheduler))

I don’t have a line counter but I think that these are cropping up with lines like $schedulerDB->getHost()…so its still getting the wrong host, I imagine…where is that stored? I’m scared to run the config because everytime I try I break it worse.

Yup. Ran the config and broke everything again. Guess I gotta revert again.

Hi,

Now that you solved the issue with your machine keep changing hostnames, please explain exactly what does not work and I’ll gladly help you solve it.