Batch script execution failed after yum update

Hello all

As suggested by Jess I post here the same question posted on Github and still not solved. https://github.com/kaltura/platform-install-packages/issues/523

Server is Centos 6.3, 16 GB ram.

After yum update from of Kaltura ce 10.10 to 11.8 the batch script execution is failing and no new media is processed.

/etc/init.d/kaltura-batch status
Service Batch Manager isn’t running [ FAILED ]

I looks like no PID lock file is created.

[root@videohost ~]# /etc/init.d/kaltura-batch restart
Stopping Batch Manager…
Killing Batch Manager with PID and related workers [ OK ]
Starting Batch Manager…
Batch Manager started with PID [ OK ]
[root@videohost ~]# ls -la /opt/kaltura/log/batch/batch.pid
ls: cannot access /opt/kaltura/log/batch/batch.pid: No such file or directory

If I restart kaltura-batch from monit it fail and a bunch of defunct processes appear at ps -aux

kaltura 22109 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]
kaltura 22121 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]
kaltura 22123 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]
kaltura 22125 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]
kaltura 22127 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]
kaltura 22129 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]
kaltura 22131 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]
kaltura 22133 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]
kaltura 22135 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]
kaltura 22137 0.0 0.0 0 0 ? Z 21:44 0:00 [sh]

and so on, about 20 defunct processes

Below the last lines of /opt/kaltura/log/kaltura_batch.log

tail kaltura_batch.log

2016-02-17 14:52:36 [0.000000] [641611149] [1] [BATCH]
[KSchedulerConfig->load] NOTICE: loading configuration
/opt/kaltura/app/configurations/batch at 1453830755

PHP Fatal error: Uncaught exception ‘Zend_Config_Exception’ with
message ‘Section ‘videohost.netv.it’ cannot be found in
/opt/kaltura/app/cache//batch/config.ini’ in
/opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150

Stack trace:

#0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59):
Zend_Config_Ini->__construct(’/opt/kaltura/ap…’,
‘videohost.netv…’, true)

#1 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(42): KSchedulerConfig->load()

#2 /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php(88): KSchedulerConfig->__construct(’/opt/kaltura/ap…’)

#3 /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php(70): KGenericScheduler->loadConfig(’/opt/kaltura/ap…’)

#4 /opt/kaltura/app/batch/KGenericBatchMgr.class.php(37): KGenericScheduler->__construct(’/usr/bin/php’, ‘/opt/kaltura/ap…’)

#5 {main}

thrown in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php on line 150

The hostname is resolvable and correct.

there are no difference between
/opt/kaltura/app/cache//batch/config.ini and
/opt/kaltura/app/configurations/batch/batch.ini

they are identical, and hostname at

/app/configurations/batch/batch.ini:

[.* : mainTemplate]

is correct

I tried to clear cache with:

find /opt/kaltura/app/cache -type f -exec rm {} ;

and restart the daemon but no difference.

Sanity test return all ok until partner ID

[Check memcached daemon is started by Monit] [PASSED, RC: 0] - [52.919132422]

[check daemon memcached init status] [PASSED, RC: 0] - [.003007207]

[kaltura-html5lib ver in KMC config.ini] [PASSED, RC: 0] - [.056952055]

[kaltura-kdp3 ver in KDP3 config.ini] [PASSED, RC: 0] - [.010422104]

[kaltura-kmc ver in KMC config.ini] [PASSED, RC: 0] - [.014156152]

[Get KMC SWFs] [PASSED, RC: 0] - [.120932115]

[check_testme_page] [FAILED, RC: 1] - [.032193084]

[check_kmc_index_page] [PASSED, RC: 0] - [.013382364]

[check_admin_console_index_page] [PASSED, RC: 0] - [.120580998]

[check_studio_index_page] [PASSED, RC: 0] - [.016036960]

[check_clipapp_index_page] [PASSED, RC: 0] - [.371713134]

[Create Partner] [FAILED, RC: 255] - [.070936344]

Partner creation failed. I will skip all tests that require it.

[Red5 file upload] [FAILED, RC: 2] - [.416108913]

Any help to solve this issue? Everything was working fine before the update and all post install script were ok with no error after upgrade.

Many thanks
ifprojects

Hello,

I’d recommend you start by:
# /etc/init.d/kaltura-batch stop
then make sure there are no zombie batch processes around. Batch processes are PHP CLI procs so you can check that with:
# ps fax|grep php

note that there may be other unrelated procs that use PHP CLI to run but if the system is only running Kaltura, it should only be used for batch and the kaltura-populate daemon.

Then, upgrade to 11.9.0 which was released as stable yesterday and re-run all the config scripts.
Then lets see where we end up.

Thanks,

Hi Jess, thankyou for replying.

Before reading your reply I did stop and start the batch a few times and after stopping the batch the defunct processes clear up and disappear. Restarting batch is as before wth no PID and all the zombie processes etc…

I have also upgraded to 11.9.0 and I did re run all the config files and this time not using the answer file and retyping all the data in. Same thing and kaltura batch do not start and zombies etc… as before.

I must admit I was so fedup I runned yum remove “kaltura” …

Again this was a fully working installation and no modification of any kind has been made to the server prior to the upgrade that created the batch problem.

Thanks anyway

Hi Jess, I have same problem and upgrade 11.8.0 to 11.9.0 yesterday, but batch script execution is failing and converting not working. Kaltura-batch restarts.

[root@media ~]# /etc/init.d/kaltura-batch restart Stopping Batch Manager.... Killing Batch Manager with PID and related workers [ OK ] Starting Batch Manager.... Batch Manager started with PID [ OK ]

kaltlog:


2016-03-02 13:16:12 [0.000000] [1813965679] [1] [BATCH] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1456919911
PHP Warning: file_put_contents(/batch.pid): failed to open stream: Permission denied in /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php on line 97
2016-03-02 13:16:12 [0.042543] [1813965679] [2] [BATCH] [KGenericScheduler->loadConfig] INFO: 2.0.2

==> /opt/kaltura/log/kaltura_prod.log <==
2016-03-02 13:16:18 [9.000846] [xx.xx.xx.xx] [728976448] [12] [PS2] [KExternalErrors::dieError] ERR: exception ‘Exception’ with message ‘exiting on error 5 - required file was not found’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/alpha/apps/kaltura/lib/KExternalErrors.class.php(128): KalturaLog::err(‘exiting on erro…’)

Any help to solve this issue?
Many thanks,
Vassy

Hi,

Like I wrote in https://github.com/kaltura/platform-install-packages/issues/525
in your particular case, I see:
PHP Warning: file_put_contents(/batch.pid): failed to open stream: Permission denied in /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php on line 97

This is obviously wrong. If you open /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php in line 97 you will see:
file_put_contents($pid, getmypid())
And $pid is set by:
$pid = $this->schedulerConfig->getPidFileDir() . ‘/batch.pid’;

So, my educated guess is you have:
logDir = /

in your /opt/kaltura/app/configurations/batch/batch.ini
which in fact should be:
logDir = /opt/kaltura/log/batch/

You can of course fix this manually but I suspect, if you have this issue, you probably have many other similar ones and you should just rerun the kaltura-config-all.sh script to fix them.

Hi Jess, thank you for replying.

In my /opt/kaltura/app/configurations/batch/batch.ini:
logDir = /opt/kaltura/log/batch/

Thanks,
Vassy

Same for me

logDir = /opt/kaltura/log/batch/

Jess I did a test 2 days ago just to see if at start kaltura-batch was going at least to try to write the PID file in the right place or not. I’ve created an empty file /opt/kaltura/log/batch/batch.pid with kaltura as user/group and then restarted kaltura-batch and my dummy file disappeared, so the script cancelled the dummy file but didn’t write his own. I don’t know if this could lead to some guessing or not.

Thanks

@ifprojects, @jess, If see in admin_console -> momitoring -> batch, path to pid file is /opt/kaltura/var/run/batch.pid. If i change /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php line 97
from $pid = $this->schedulerConfig->getPidFileDir() . '/batch.pid';
to $pid = '/opt/kaltura/var/run/batch.pid';
batch Running and all is OK.
I guess $this->schedulerConfig->getPidFileDir() returns incorrect path. This is temporary solution for me, but works.

Regards,
Vassy

Hi vassybg,

You got me very excited on this one. I tried the same solution and restarted the server couple of times and also ran config-all script but it did not work for me.

//old line    $pid = $this->schedulerConfig->getPidFileDir() . '/batch.pid';
//new line    $pid = '/opt/kaltura/var/run/batch.pid';

Can you please let me know your Kaltura and OS version?

Thanks,
talhagithub

Hi talhagithub,

I only edit /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php line 97 and restart monit. I get path from admin_console -> monitoring -> batch for batch pid path.
Kaltura 11.9.0
Centos 6.7

Regards,
Vassy

Interesting to note that path from admin_console -> monitoring -> batch for batch pid path is

/opt/kaltura/log/batch/batch.pid

for version 11.8.0

@ifprojects, try to edit line 97 to $pid = '/opt/kaltura/log/batch/batch.pid';

Regaerds,
Vassy

Will do, thanks.

Regards
IF