The convert jobs alway pending. Please help?

phong_hunterist · August 17, 2016, 9:06am

Please look at the pictures.
The problem is some users can not convert their uploaded video. When I check from batch management from admin_console, it’s show me the tasks pedding forever.
Any help?
Thank you very much.

[root@BATCH1 batch 2501-PROD]# tail -f convert-0-2016-08-17.log
2016-08-17 18:07:32 [0.000091] [1417268031] [12] [BATCH] [KJobHandlerWorker->run] INFO: 0 jobs to handle
2016-08-17 18:07:32 [0.000075] [1417268031] [13] [BATCH] [KJobHandlerWorker->run] INFO: Queue size: 0 sent to scheduler
2016-08-17 18:07:32 [0.000209] [1417268031] [14] [BATCH] [KalturaClientBase->doQueue] NOTICE: service url: [https://front1.2501pf.com]
2016-08-17 18:07:32 [0.000207] [1417268031] [15] [BATCH] [KalturaClientBase->doCurl] NOTICE: curl: https://front1.2501pf.com/api_v3/service/batch/action/getQueueSize
2016-08-17 18:07:32 [0.000081] [1417268031] [16] [BATCH] [KalturaClientBase->doCurl] NOTICE: post: {"format":"3","ignoreNull":true,"clientTag":"batch: BATCH1 KAsyncConvert index: 0 sessionId: 1417268031","apiVersion":"3.3.0","workerQueueFilter":{"objectType":"KalturaWorkerQueueFilter","schedulerId":"88957","workerId":"70","jobType":"0","filter":{"objectType":"KalturaBatchJobFilter","jobTypeEqual":"0","jobSubTypeIn":"1,2,99,3,fastStart.FastStart,segmenter.Segmenter,mp4box.Mp4box,vlc.Vlc,document.ImageMagick,201,202,quickTimeTools.QuickTimeTools"}},"partnerId":"-1","ks":"NGRjNWIxY2U0YWVjNWUxZWI3MmIzMDlkOGE2NmM5OTAyYzRjMjRhM3wtMTs7MTQ3NDAxNjg1MjsyOzE0NzE0MjQ4NTIuNTAzMjtiYXRjaFVzZXI7ZGlzYWJsZWVudGl0bGVtZW50Oy0xOw==","kalsig":"30471164ac5fa66f794b44ebf04087c7"}
2016-08-17 18:07:32 [0.037679] [1417268031] [17] [BATCH] [KalturaClientBase->doQueue] NOTICE: server: [front1.2501pf.com], session: [1213212600]
2016-08-17 18:07:32 [0.000102] [1417268031] [18] [BATCH] [KalturaClientBase->doQueue] NOTICE: result (serialized): i:234;
2016-08-17 18:07:32 [0.000091] [1417268031] [19] [BATCH] [KalturaClientBase->doQueue] NOTICE: result (object dump): 234
2016-08-17 18:07:32 [0.000081] [1417268031] [20] [BATCH] [KalturaClientBase->doQueue] NOTICE: execution time for [https://front1.2501pf.com/api_v3/service/batch/action/getQueueSize]: [0.038242816925049]
2016-08-17 18:07:32 [0.000260] [1417268031] [21] [BATCH] [KBatchBase->done] INFO: Done after [0.25762701034546] seconds
2016-08-17 18:07:34 [0.000000] [591649317] [1] [BATCH] [KBatchBase->__construct] DEBUG: ___________________________________________________________________________________

jess · August 17, 2016, 8:48pm

Hello,

Start by checking all relevant logs under /opt/kaltura/log/batch/*.log and /opt/kaltura/log/kaltura_batch.log with:

# grep -A 1 -B 1 --color "ERR:\|PHP\|trace\|CRIT\|\[error\]" /opt/kaltura/log/batch/*`date +%Y-%m-%d`*.log /opt/kaltura/log/kaltura_batch.log

Once you find errors, I can further help you debug them.

phong_hunterist · August 18, 2016, 2:52am

Thank you very much.
The errors showed from log when I uploaded new video.
Anyway, The problem pending converting just happened with some accounts in kaltura.

tail -f /opt/kaltura/log/batch/*`date +%Y-%m-%d`*.log /opt/kaltura/log/kaltura_batch.log | grep -A 1 -B 1 --color "ERR:\|trace\|CRIT\|\[error\]"
' in /opt/kaltura/app/batch/client/KalturaClientBase.php:401
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(1122): KalturaClientBase->doQueue()
--
' in /opt/kaltura/app/batch/client/KalturaClientBase.php:401
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(1122): KalturaClientBase->doQueue()

jess · August 18, 2016, 10:09am

Hi,

You need to open the relevant log file and look at the lines leading to the final error.
Then paste the lines leading to it plus the full stack trace so I can help you further.

phong_hunterist · August 18, 2016, 10:41am

Thank you very much.
I rechecked again by log schedulerId and workerId but all of both return NULL.
I think this is the reason to make this->getJobs() method from KJobHandlerWorker.class.php return NULL.
Could yo help me know what happened? I’m very confusing this case

jess · August 18, 2016, 11:01am

Like I said, paste the relevant entries from the log so I can help you further.
Also look at /opt/kaltura/log/kaltura_batch.log

The UI output does not help me, I’m afraid.

Also, you can always try to rerun:
# /opt/kaltura/bin/kaltura-batch-config.sh
or, on a deb based install:
# dpkg-reconfigure kaltura-batch

phong_hunterist · August 19, 2016, 2:40am

I just see only error from /opt/kaltura/log/batch/partnerloadcleanup-0-2016-08-18.err.log

PHP Fatal error:  Uncaught exception 'KalturaClientException' with message 'failed to unserialize server result
' in /opt/kaltura/app/batch/client/KalturaClientBase.php:401
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(1122): KalturaClientBase->doQueue()
#1 /opt/kaltura/app/batch/batches/Cleanup/KAsyncPartnerLoadCleanup.class.php(28): KalturaBatchService->updatePartnerLoadTable()
#2 /opt/kaltura/app/batch/batches/Cleanup/KAsyncPartnerLoadCleanupExe.php(11): KAsyncPartnerLoadCleanup->run()
#3 {main}
  thrown in /opt/kaltura/app/batch/client/KalturaClientBase.php on line 401

And I also check /opt/kaltura/log/batch//convert-0-2016-08-18.log But it alway returned 0 sent to scheduler

2016-08-19 11:29:37 [0.000099] [1210349954] [12] [BATCH] [KalturaClientBase->doQueue] NOTICE: execution time for [https://front1.com/api_v3/service/batch/action/getExclusiveJobs]: [0.16400408744812]
2016-08-19 11:29:37 [0.000104] [1210349954] [13] [BATCH] [KJobHandlerWorker->run] INFO: 0 jobs to handle
2016-08-19 11:29:37 [0.000073] [1210349954] [14] [BATCH] [KJobHandlerWorker->run] INFO: Queue size: 0 sent to scheduler
2016-08-19 11:29:37 [0.000220] [1210349954] [15] [BATCH] [KalturaClientBase->doQueue] NOTICE: service url: [https://front1.2501pf.com]
2016-08-19 11:29:37 [0.000199] [1210349954] [16] [BATCH] [KalturaClientBase->doCurl] NOTICE: curl: https://front1.com/api_v3/service/batch/action/getQueueSize
2016-08-19 11:29:37 [0.000080] [1210349954] [17] [BATCH] [KalturaClientBase->doCurl] NOTICE: post: {"format":"3","ignoreNull":true,"clientTag":"batch: BATCH1 KAsyncConvert index: 0 sessionId: 1210349954","apiVersion":"3.3.0","workerQueueFilter":{"objectType":"KalturaWorkerQueueFilter","schedulerId":"88957","workerId":"70","jobType":"0","filter":{"objectType":"KalturaBatchJobFilter","jobTypeEqual":"0","jobSubTypeIn":"1,2,99,3,fastStart.FastStart,segmenter.Segmenter,mp4box.Mp4box,vlc.Vlc,document.ImageMagick,201,202,quickTimeTools.QuickTimeTools"}},"partnerId":"-1","ks":"ZTZhYTliNDYxOWY2ODllNWMwZGU2NGIzMGIxNjZjNjA1NGUzYzc4NnwtMTs7MTQ3NDE2NTc3NzsyOzE0NzE1NzM3NzcuNTk7YmF0Y2hVc2VyO2Rpc2FibGVlbnRpdGxlbWVudDstMTs=","kalsig":"8e751475aa15cfe44d04ac44c20312c0"}
2016-08-19 11:29:37 [0.039682] [1210349954] [18] [BATCH] [KalturaClientBase->doQueue] NOTICE: server: [front1.com], session: [1074452967]
2016-08-19 11:29:37 [0.000120] [1210349954] [19] [BATCH] [KalturaClientBase->doQueue] NOTICE: result (serialized): i:240;
2016-08-19 11:29:37 [0.000122] [1210349954] [20] [BATCH] [KalturaClientBase->doQueue] NOTICE: result (object dump): 240
2016-08-19 11:29:37 [0.000096] [1210349954] [21] [BATCH] [KalturaClientBase->doQueue] NOTICE: execution time for [https://front1.com/api_v3/service/batch/action/getQueueSize]: [0.040301084518433]
2016-08-19 11:29:37 [0.000244] [1210349954] [22] [BATCH] [KBatchBase->done] INFO: Done after [0.24942278862] seconds

Any help?

phong_hunterist · August 19, 2016, 9:15am

Sorry, I have one more question

function getExclusiveJobs(KalturaExclusiveLockKey $lockKey, $maxExecutionTime, $numberOfJobs, KalturaBatchJobFilter $filter = null, $jobType = null, $maxOffset = null)
	{
		$kparams = array();
		$this->client->addParam($kparams, "lockKey", $lockKey->toParams());
		$this->client->addParam($kparams, "maxExecutionTime", $maxExecutionTime);
		$this->client->addParam($kparams, "numberOfJobs", $numberOfJobs);
		if ($filter !== null)
			$this->client->addParam($kparams, "filter", $filter->toParams());
		$this->client->addParam($kparams, "jobType", $jobType);
		$this->client->addParam($kparams, "maxOffset", $maxOffset);
		$this->client->queueServiceActionCall("batch", "getExclusiveJobs", $kparams);
		if ($this->client->isMultiRequest())
			return $this->client->getMultiRequestResult();
		$resultObject = $this->client->doQueue();
		$this->client->throwExceptionIfError($resultObject);
		$this->client->validateObjectType($resultObject, "array");
		return $resultObject;
	}

What log file will catch error from this function?

Thank you very much

phong_hunterist · August 25, 2016, 3:38am

tail -f /opt/kaltura/log/kaltura_apache_errors_ssl.log
[Wed Aug 24 17:21:06 2016] [error] [client 52.196.26.215] PHP Fatal error: Call to a member function getJobTypeQuota() on a non-object in /opt/kaltura/app/alpha/lib/model/PartnerLoadPeer.php on line 180
[Wed Aug 24 17:22:07 2016] [error] [client 52.196.26.215] PHP Fatal error: Call to a member function getJobTypeQuota() on a non-object in /opt/kaltura/app/alpha/lib/model/PartnerLoadPeer.php on line 180
[Wed Aug 24 17:23:08 2016] [error] [client 52.196.26.215] PHP Fatal error: Call to a member function getJobTypeQuota() on a non-object in /opt/kaltura/app/alpha/lib/model/PartnerLoadPeer.php on line 180
[Wed Aug 24 17:24:10 2016] [error] [client 52.196.26.215] PHP Fatal error: Call to a member function getJobTypeQuota() on a non-object in /opt/kaltura/app/alpha/lib/model/PartnerLoadPeer.php on line 180
[Wed Aug 24 17:25:10 2016] [error] [client 52.196.26.215] PHP Fatal error: Call to a member function getJobTypeQuota() on a non-object in /opt/kaltura/app/alpha/lib/model/PartnerLoadPeer.php on line 180
[Wed Aug 24 17:26:12 2016] [error] [client 52.196.26.215] PHP Fatal error: Call to a member function getJobTypeQuota() on a non-object in /opt/kaltura/app/alpha/lib/model/PartnerLoadPeer.php on line 180

That I found on Kaltura document but I dont understand clearly
Job Quota - A quota can be configured to limit the number of concurrently running jobs in the Kaltura deployment. There is a system-wide default as well as defaults for specific job types. It is also possible to configure the quota differently per each account.
https://knowledge.kaltura.com/kaltura-media-transcoding-services-and-technology

Any idea?
Thank you for your help.