I just realized that a lot of content isn’t able to be properly viewed in the preview & embed screen, as well as is not showing up in my syndication feeds. This appears to have started over a week ago, when I was messing around with both my Default Transcoding Flavors, my thumbnail settings, and my remote storage. So I assume it’s related to one of those. Here’s a log of when I try to upload a video I’m trying to sort through right now:
For example, for the Transcoding Flavors I took off every flavor but the Source, I tried adding the 720p/2500 bitrate one but that didn’t seem to affect it any. There are also several custom thumbnail profiles in that trasncode profile as well.
2016-07-05 18:56:37 [0.000000] [588143800] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1465256427
PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with message ‘Section ‘ip-10-0-0-92’ cannot be found in /opt/kaltura/app/cache//batch/config.ini’ in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace: #0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-92’, true)
[2016-07-05 18:53:26]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace: #0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)
[2016-07-05 18:55:37]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace: #0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)
[2016-07-05 18:57:47]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace: #0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)
Your first issue is that it seems your batch daemon is not running at all, I deduce this from:
2016-07-05 18:58:48 [0.000000] [395842269] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1465256427
PHP Fatal error: Uncaught exception 'Zend_Config_Exception' with message 'Section 'ip-10-0-0-92' cannot be found in /opt/kaltura/app/cache//batch/config.ini' in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
in /opt/kaltura/app/configurations/batch/batch.ini, your should have something like:
[ce-rhel7.dev.kaltura.com : mainTemplate]
id = 45866
name = Default Scheduler
in this example ce-rhel7.dev.kaltura.com is the hostname, the ‘id’ directive should have a unique value the config script sets. If your hostname is ip-10-0-0-92 then that’s what should be there.
It is possible you changed your hostname and did not run the reconfigure scripts, which would cause this.
If you’re using the RPMs, you should run, as root:
# /opt/kaltura/bin/kaltura-batch-config.sh
if debs, then:
# dpkg-reconfigure kaltura-batch
once the batch daemon is running again, try to upload a new entry and look at the log.
Here is my log, still having issues. The IP is incorrect, and something I thought I changed when I upgraded it a few weeks ago. Should I run this from the upgrade instructions again?
[root@ip-10-0-0-92 ec2-user]# /opt/kaltura/bin/kaltura-batch-config.sh
kaltura-batch-11.14.0-1.noarch
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
rm /opt/kaltura/app/base-config.lock
/opt/kaltura/bin/kaltura-base-config.sh
Reloading httpd:
Stopping memcached: [ OK ]
Starting memcached: [ OK ]
Starting monit: Starting Monit 5.13 daemon with http interface at [*]:2812
[ OK ]
[root@ip-10-0-0-92 ec2-user]#
Logs when I added an entry:
2016-07-06 14:05:22 [0.000000] [1815134738] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1467826548
PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with message ‘Section ‘ip-10-0-0-92’ cannot be found in /opt/kaltura/app/cache//batch/config.ini’ in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace: #0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-92’, true)
[2016-07-06 14:06:32]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace: #0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)
2016-07-06 14:07:33 [0.000000] [198071463] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1467826548
PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with message ‘Section ‘ip-10-0-0-92’ cannot be found in /opt/kaltura/app/cache//batch/config.ini’ in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace: #0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-92’, true)
[2016-07-06 14:08:43]exception ‘Exception’ with message ‘Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace: #0 /opt/kaltura/app/plugins/sphinx_search/scripts/populateFromLog.php(36): KalturaLog::err(‘Configuration f…’)
2016-07-06 14:09:24 [0.006544] [45.59.236.158] [770093946] [53] [PS2] [KalturaStatement->execute] DEBUG: Sql took - 0.0059919357299805 seconds
2016-07-06 14:09:24 [0.000706] [45.59.236.158] [770093946] [54] [PS2] [KExternalErrors::dieError] ERR: exception ‘Exception’ with message ‘exiting on error 6 - requested flavor was not found’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace: #0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)
OK, so, the hostname is clearly ip-10-0-0-92 and is resolvable, which is good.
Now, what do you have in /opt/kaltura/app/configurations/batch/batch.ini since it obviously does not have:
You’re trying to run it whereas what you should do is edit it.
Do:
# sudo vi /opt/kaltura/app/configurations/batch/batch.ini
or
# sudo nano /opt/kaltura/app/configurations/batch/batch.ini
The thing is, /opt/kaltura/bin/kaltura-batch-config.sh already does that for you, if you open it, you will see:
sed -i “s#@INSTALLED_HOSNAME@#hostname#” -i $BATCH_MAIN_CONF
that is, it looks for @INSTALLED_HOSNAME@ in /opt/kaltura/app/configurations/batch/batch.ini and replaces it with the output of the hostname command. My guess would be, that your hostname changed but, you did not re-run the kaltrua-base-config.sh script after it changed, therefore, /opt/kaltura/app/configurations/batch/batch.ini remained as it was after the original run of /opt/kaltura/bin/kaltura-batch-config.sh and so, the @INSTALLED_HOSNAME@ placeholder was no longer there to replace.
You should either:
# /opt/kaltura/bin/kaltura-base-config.sh
# /opt/kaltura/bin/kaltura-batch-config.sh
again or just manually change the hostname in /opt/kaltura/app/configurations/batch/batch.ini and:
# /etc/init.d/kaltura-batch restart
What concerns me about the second approach, although it will take less time [2 seconds vs. say, 2 minutes for the base-config script to end] is that you may have other places where you have the wrong hostname.
You can start by editing manually and restarting but if you run into additional problems, make sure to re-run all the config scripts or else just:
# rm /opt/kaltura/app/base-config.lock
# /opt/kaltura/bin/kaltura-config-all.sh
/opt/kaltura/bin/kaltura-config-all.sh runs all needed scripts automatically but is only the right way to go in case you have an all in 1 server, for a cluster, you need to run different scripts per node role.
starting to see some things happen now but I wonder if it’s all right, I still see some references to the IP even though I thought i changed them to our host. It removed all the assets that were still “converting” previously and some of my bulk upload jobs started coming through, but they’re all still converting. I’ll check back tomorrow buit here are some of the logs:
OK, well, now it seems your batch daemon is running but there are still plenty of problems.
I honestly would recommend just re-running all the config scripts as I explained in my previous post.
It will not drop your DB unless explicitly asked to but I suggest you back it up beforehand just in case.
But, nonetheless, as a learning experience, here is what I get from the lines you posted:
It seems the permissions on /opt/kaltura/app/cache/ are wrong.
What’s the output for:
# ll -d /opt/kaltura/app/cache/
?
it should be similar to:
drwxrwxr-x. 14 apache kaltura 4096 Jun 26 10:41 /opt/kaltura/app/cache/
if the directory is not owned by the user apache and group kaltura, correct it with:
find /opt/kaltura/app/cache/ -type f -exec rm {} \;
Be sure NOT to delete directories under /opt/kaltura/app/cache.
Currently, from your log it seems writing to /opt/kaltura/app/cache/batch fails. Make sure this dir exists and is currently permissoned.
Another issue is with:
[2016-07-06 20:23:22]exception 'Exception' with message 'Configuration file [/opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini] not found.' in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
once again, related to the hostname. You should have /opt/kaltura/app/configurations/sphinx/populate/ip-10-0-0-92.ini, what do you have instead?
There also seem to be a timeout issue when making requests, you’ll need to go to the actual log and look at the lines leading to it to understand what’s going on.
It was still the 199 ip instead of 92. I did rerun the /opt/kaltura/bin/kaltura-base-config.sh before my last post and things were still like that. I’ll rerun again.
Regarding logs, I have been going through them but it’s really difficult for me to know what to do in some circumstances, I am not sure what to refer to in an instance like this:
0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)
Like yes, that to me says that the KJobHandlerWorker is closing because it timed out on object 0_kt4zr3hj, but I’m not sure what to do with that information. Anyway, I’ll rerun the config.
I keep breaking it more, is there a way I can look at all my current configuration stuff so I don’t accidentally break something when trying to fix the ip problems? Then I can safely revert it without having to go to my full backup, as well as make sure exactly what parts of the config that need fixing.
For example, when trying to upload now I get the error:
0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct(’/opt/kaltura/ap…’, ‘ip-10-0-0-239’, true)
I need to change that to the correct address, so where does that address live (in the config?) and how do I change it safely/individually. Is there anyway to figure this out on my own via documentation?
Other questions include why are these error messages getting truncated, it makes it even more difficult to debug.
I tried adding more verbosity with
sed -i ‘s@^writers.(.).filters.priority.priority\s=\s*7@writers.\1.filters.priority.priority=4@g’ /opt/kaltura/app/configurations/logger.ini
it didn’t give me any message that it was or wasn’t activated, and I didn’t notice a different. Do I need to write these messages to a log to get a full error message?
to what is missing, (now it’s a different ip, ip-10-0-0-239), and save it, view the file again, confirm the saved change, then upload a file to KMC. It gives me the exact same error, and when I open /opt/kaltura/app/cache//batch/config.ini again, I see the old IP again…what gives?
Your problem is not the IP, although of course, for a server, you’d want a fix IP. Your problem is with the hostname. Why is it constantly changing? you need the machine to have one hostname that does not change every other minute. The configuration files have directives that rely on that. Much like any server side product really…
Okay, I think I resolved that one. It’s been changing whenever we’ve been restoring it after we break the box, but otherwise it should be static. Anyway, I’ve been fixing references to it manually that show up in my errors, and now it’s repeating constantly hundreds of times
–
2016-07-08 16:55:38 [0.000723] [52.41.250.214] [1799309173] [296] [API] [KalturaStatement->execute] DEBUG: Sql took - 0.00024604797363281 seconds
2016-07-08 16:55:38 [0.001237] [52.41.250.214] [1799309173] [297] [API] [KalturaFrontController->getExceptionObject] ERR: exception ‘KalturaAPIException’ with message ‘Scheduler id “47578” conflicts between hosts: “ip-10-0-0-239” and “ip-10-0-0-199”’ in /opt/kaltura/app/api_v3/services/BatchControlService.php:148
Stack trace: #0 /opt/kaltura/app/api_v3/services/BatchControlService.php(249): BatchControlService->getOrCreateScheduler(Object(KalturaScheduler))
I don’t have a line counter but I think that these are cropping up with lines like $schedulerDB->getHost()…so its still getting the wrong host, I imagine…where is that stored? I’m scared to run the config because everytime I try I break it worse.