Red5 file upload and DWH cycle failed after upgrade 11.6

1 
[root@kaltura ~]# /opt/kaltura/bin/kaltura-sanity.sh
[Space on /] [PASSED, RC: 0] - [.003674649]
[Space on /opt/kaltura/web] [PASSED, RC: 0] - [.003546695]
[Check httpd daemon status] [PASSED, RC: 0] - [.023354439]
Napping 1 seconds to allow the daemon httpd to finish its init after monit restarted it..
Napping 2 seconds to allow the daemon httpd to finish its init after monit restarted it..
Napping 3 seconds to allow the daemon httpd to finish its init after monit restarted it..
[Check httpd daemon is started by Monit] [PASSED, RC: 0] - [4.261568337]
[check daemon httpd init status] [PASSED, RC: 0] - [.004759549]
[Check kaltura-sphinx daemon status] [PASSED, RC: 0] - [.027469306]
[Check kaltura-sphinx daemon is started by Monit] [PASSED, RC: 0] - [10.567329368]
[check daemon kaltura-sphinx init status] [PASSED, RC: 0] - [.004421807]
[Check kaltura-batch daemon status] [PASSED, RC: 0] - [.026433933]
Napping 1 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
Napping 2 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
Napping 3 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
Napping 4 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
Napping 5 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
Napping 6 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
[Check kaltura-batch daemon is started by Monit] [PASSED, RC: 0] - [6.212003099]
[check daemon kaltura-batch init status] [PASSED, RC: 0] - [.004606432]
[Check kaltura-monit daemon status] [PASSED, RC: 0] - [.023144375]
[check daemon kaltura-monit init status] [PASSED, RC: 0] - [.004642978]
[Check memcached daemon status] [PASSED, RC: 0] - [.022893296]
Napping 1 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 2 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 3 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 4 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 5 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 6 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 7 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 8 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 9 seconds to allow the daemon memcached to finish its init after monit restarted it..
[Check memcached daemon is started by Monit] [PASSED, RC: 0] - [9.442849173]
[check daemon memcached init status] [PASSED, RC: 0] - [.004633895]
[kaltura-html5lib ver in KMC config.ini] [PASSED, RC: 0] - [.102884332]
[kaltura-kdp3 ver in KDP3 config.ini] [PASSED, RC: 0] - [.040406185]
[kaltura-kmc ver in KMC config.ini] [PASSED, RC: 0] - [.045240760]
[Get KMC SWFs] [PASSED, RC: 0] - [.127690504]
[check_testme_page] [PASSED, RC: 0] - [14.247045186]
[check_kmc_index_page] [PASSED, RC: 0] - [.020835387]
[check_admin_console_index_page] [PASSED, RC: 0] - [.480929346]
[check_studio_index_page] [PASSED, RC: 0] - [.013889421]
[check_clipapp_index_page] [PASSED, RC: 0] - [.495025523]
[Create Partner] [PASSED, RC: 0] - [1.436053838]
[Local dropfolder creation] [PASSED, RC: 0] - [.369388974]
[Create flavor param] [PASSED, RC: 0] - [.410269582]
[Delete flavor param] [PASSED, RC: 0] - [.182209818]
[HTTP notification: 
No HTTP templates exist.
 Try running:
php /opt/kaltura/app/tests/standAloneClient/exec.php /opt/kaltura/app/tests/standAloneClient/httpNotificationsTemplate.xml /tmp/out.xml
Passing -2 as partner ID] [PASSED, RC: 0] - [.410269582]
[Mail notification: ID: 18, Subject: Your video is ready to be played!, Mail body: Hello world:)] [PASSED, RC: 0] - [.410269582]
[Upload content kaltura_logo_animated_blue.flv] [PASSED, RC: 0] - [.598495385]
Napping 10 seconds to allow entry 0_e4nhu25c to digest.. 
Napping 10 seconds to allow entry 0_e4nhu25c to digest.. 
Napping 10 seconds to allow entry 0_e4nhu25c to digest.. 
Napping 10 seconds to allow entry 0_e4nhu25c to digest.. 
[kaltura_logo_animated_blue.flv - 0_e4nhu25c status] [PASSED, RC: 0] - [40.610887962]
[Create thumb for 0_e4nhu25c] [PASSED, RC: 0] - [.408577792]
[Clipping 0_e4nhu25c] [PASSED, RC: 0] - [.440951533]
[Trimming 0_e4nhu25c] [PASSED, RC: 0] - [.415825266]
[Mock playback 0_e4nhu25c] [PASSED, RC: 0] - [.228473347]
[Adding captions to 0_e4nhu25c] [PASSED, RC: 0] - [.310487828]
[Searching for Example||Deja||Bold in 0_e4nhu25c metadata] [PASSED, RC: 0] - [.191962089]
[Entry reconversion of 0_e4nhu25c succeeded] [PASSED, RC: 0] - [.318695066]
Napping 30 seconds to allow mail to be sent out.. 
[Found an email sending entry for mb-kaltura.yasar.edu.tr@kaltura.com[PID is 135] in /var/log/maillog] [PASSED, RC: 0] - [30.010206196]
Testing analytics, be patient..

Please note: if you are running this test on a clustered ENV, it will fail but this does not mean there is an actual problem.
The tech information as to why is available here: 
https://github.com/kaltura/platform-install-packages/issues/106#issuecomment-42837404

[DWH cycle] [FAILED, RC: 255] - [37.203860812]
[Upload content kaltura_logo_animated_green.flv] [PASSED, RC: 0] - [.710325315]
[Upload bulk using CSV] [PASSED, RC: 0] - [.384476196]
[Upload bulk using XML] [PASSED, RC: 0] - [.288162145]
[Create player] [PASSED, RC: 0] - [.192981908]
[Generate thumb] [PASSED, RC: 0] - [.308571705]
[Delete partner] [PASSED, RC: 0] - [.480202869]
[Red5 file upload] [FAILED, RC: 2] - [.071434530]


Thank you for running Kaltura! To keep Kaltura viable, stable and well tested, please join the community and help by contributing sanity tests that verify overall platform stability: http://bit.ly/kaltura-ci , and by contributing to the project roadmap by solving simple tasks and challenges: http://bit.ly/kaltura-tasks.

kaltlog is below

[root@kaltura ~]# kaltlog
==> /opt/kaltura/log/batch/dbcleanup-0-2016-01-22.err.log <==
PHP Fatal error:  Uncaught exception 'KalturaClientException' with message 'couldn't connect to host' in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(1299): KalturaClientBase->doQueue()
--
==> /opt/kaltura/log/batch/jobsuspender-0-2016-01-22.err.log <==
PHP Fatal error:  Uncaught exception 'KalturaClientException' with message 'couldn't connect to host' in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(1137): KalturaClientBase->doQueue()
--
==> /opt/kaltura/log/batch/partnerloadcleanup-0-2016-01-22.err.log <==
PHP Fatal error:  Uncaught exception 'KalturaClientException' with message 'couldn't connect to host' in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(1122): KalturaClientBase->doQueue()
2

Hello,

The errors you attached are unrelated to Red5, I assume they are older entries from when the Apache was down, perhaps while sanity was testing if it automatically is started by monit, which is one of its tests.

The red5 test is basically:
# $BASE_DIR/bin/fms_push_file_monitor.sh $BASE_DIR/web/content/templates/entry/data/kmc_logo.flv rtmp://$RED5_HOST/oflaDemo/

I suggest you start by making sure it is running and that you can:
# telnet $RED5_HOST 1935

Then try running the script manually from the shell to see what’s causing it to fail, it is basically just trying to use ffmpeg to push $BASE_DIR/web/content/templates/entry/data/kmc_logo.flv onto the red5 server.

3

Hi @jess

I restarted to red5 and it is working

[root@kaltura ~]# /etc/init.d/red5 restart
Shutting down red5:                                        [  OK  ]
Starting red5:                                             [  OK  ]

and

[root@kaltura ~]# $BASE_DIR/bin/fms_push_file_monitor.sh $BASE_DIR/web/content/templates/entry/data/kmc_logo.flv rtmp://$RED5_HOST/oflaDemo/
OK:) kmc_logo.flv successfully uploaded.
[root@kaltura ~]# clear

Thank u so much,

4

Hi @jess

latest kaltura-sanity.sh is below.

What can I do for “DWH cycle”

[Space on /] [PASSED, RC: 0] - [.009406532]
[Space on /opt/kaltura/web] [PASSED, RC: 0] - [.007400507]
[Check httpd daemon status] [PASSED, RC: 0] - [.039511887]
Napping 1 seconds to allow the daemon httpd to finish its init after monit restarted it..
Napping 2 seconds to allow the daemon httpd to finish its init after monit restarted it..
[Check httpd daemon is started by Monit] [PASSED, RC: 0] - [3.382115322]
[check daemon httpd init status] [PASSED, RC: 0] - [.014017628]
[Check kaltura-sphinx daemon status] [PASSED, RC: 0] - [.029714048]
[Check kaltura-sphinx daemon is started by Monit] [PASSED, RC: 0] - [10.662804304]
[check daemon kaltura-sphinx init status] [PASSED, RC: 0] - [.004678448]
[Check kaltura-batch daemon status] [PASSED, RC: 0] - [.058917932]
Napping 1 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
Napping 2 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
Napping 3 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
Napping 4 seconds to allow the daemon kaltura-batch to finish its init after monit restarted it..
[Check kaltura-batch daemon is started by Monit] [PASSED, RC: 0] - [4.324412626]
[check daemon kaltura-batch init status] [PASSED, RC: 0] - [.004683733]
[Check kaltura-monit daemon status] [PASSED, RC: 0] - [.054235003]
[check daemon kaltura-monit init status] [PASSED, RC: 0] - [.004722068]
[Check memcached daemon status] [PASSED, RC: 0] - [.022979797]
Napping 1 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 2 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 3 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 4 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 5 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 6 seconds to allow the daemon memcached to finish its init after monit restarted it..
Napping 7 seconds to allow the daemon memcached to finish its init after monit restarted it..
[Check memcached daemon is started by Monit] [PASSED, RC: 0] - [7.567631758]
[check daemon memcached init status] [PASSED, RC: 0] - [.009380563]
[kaltura-html5lib ver in KMC config.ini] [PASSED, RC: 0] - [.219768804]
[kaltura-kdp3 ver in KDP3 config.ini] [PASSED, RC: 0] - [.040594155]
[kaltura-kmc ver in KMC config.ini] [PASSED, RC: 0] - [.106309723]
[Get KMC SWFs] [PASSED, RC: 0] - [.213552802]
[check_testme_page] [PASSED, RC: 0] - [.247388406]
[check_kmc_index_page] [PASSED, RC: 0] - [.031769408]
[check_admin_console_index_page] [PASSED, RC: 0] - [.165990473]
[check_studio_index_page] [PASSED, RC: 0] - [.016855941]
[check_clipapp_index_page] [PASSED, RC: 0] - [.850103020]
[Create Partner] [PASSED, RC: 0] - [3.377361684]
[Local dropfolder creation] [PASSED, RC: 0] - [.657807539]
[Create flavor param] [PASSED, RC: 0] - [.446652963]
[Delete flavor param] [PASSED, RC: 0] - [.287236209]
[HTTP notification: 
No HTTP templates exist.
 Try running:
php /opt/kaltura/app/tests/standAloneClient/exec.php /opt/kaltura/app/tests/standAloneClient/httpNotificationsTemplate.xml /tmp/out.xml
Passing -2 as partner ID] [PASSED, RC: 0] - [.446652963]
[Mail notification: ID: 19, Subject: Your video is ready to be played!, Mail body: Hello world:)] [PASSED, RC: 0] - [.446652963]
[Upload content kaltura_logo_animated_blue.flv] [PASSED, RC: 0] - [.836243931]
Napping 10 seconds to allow entry 0_uylmm2s6 to digest.. 
Napping 10 seconds to allow entry 0_uylmm2s6 to digest.. 
Napping 10 seconds to allow entry 0_uylmm2s6 to digest.. 
Napping 10 seconds to allow entry 0_uylmm2s6 to digest.. 
Napping 10 seconds to allow entry 0_uylmm2s6 to digest.. 
Napping 10 seconds to allow entry 0_uylmm2s6 to digest.. 
Napping 10 seconds to allow entry 0_uylmm2s6 to digest.. 
[kaltura_logo_animated_blue.flv - 0_uylmm2s6 status] [PASSED, RC: 0] - [71.946863802]
[Create thumb for 0_uylmm2s6] [PASSED, RC: 0] - [.340496505]
[Clipping 0_uylmm2s6] [PASSED, RC: 0] - [.616048970]
[Trimming 0_uylmm2s6] [PASSED, RC: 0] - [.464692100]
[Mock playback 0_uylmm2s6] [PASSED, RC: 0] - [.231632959]
[Adding captions to 0_uylmm2s6] [PASSED, RC: 0] - [.310532155]
[Searching for Example||Deja||Bold in 0_uylmm2s6 metadata] [PASSED, RC: 0] - [.408221129]
[Entry reconversion of 0_uylmm2s6 succeeded] [PASSED, RC: 0] - [.360172975]
Napping 30 seconds to allow mail to be sent out.. 
[Found an email sending entry for mb-kaltura.yasar.edu.tr@kaltura.com[PID is 136] in /var/log/maillog] [PASSED, RC: 0] - [30.030372088]
Testing analytics, be patient..

Please note: if you are running this test on a clustered ENV, it will fail but this does not mean there is an actual problem.
The tech information as to why is available here: 
https://github.com/kaltura/platform-install-packages/issues/106#issuecomment-42837404

[DWH cycle] [FAILED, RC: 255] - [89.035043440]
[Upload content kaltura_logo_animated_green.flv] [PASSED, RC: 0] - [.903079458]
[Upload bulk using CSV] [PASSED, RC: 0] - [.468246911]
[Upload bulk using XML] [PASSED, RC: 0] - [.292459314]
[Create player] [PASSED, RC: 0] - [.207157641]
[Generate thumb] [PASSED, RC: 0] - [.308131048]
[Delete partner] [PASSED, RC: 0] - [.537847487]
[Red5 file upload] [PASSED, RC: 0] - [2.979111711]


Thank you for running Kaltura! To keep Kaltura viable, stable and well tested, please join the community and help by contributing sanity tests that verify overall platform stability: http://bit.ly/kaltura-ci , and by contributing to the project roadmap by solving simple tasks and challenges: http://bit.ly/kaltura-tasks.
5

Hi,

Like I wrote before, you need to look at /opt/kaltura/dwh/logs, looking for string ERROR and then working your way up the log file to understand what causes it.
If you need further assistance, please paste the ERROR and lines leading to it.

6

Hi @jess,

Error log file is here

Thanks in advance.

7

Hi,

This is just the lines that contain the actual string “ERROR” but like I said, you need to look at the lines leading to it.
Also, you have “Lock is already seized”, look at: https://github.com/kaltura/platform-install-packages/blob/Kajam-11.7.0/doc/kaltura-packages-faq.md#analytics-issues

which explains how to do basic troubleshooting and also removing locks.