Looking for help interpreting sanity check

1

Hi,

New installation of 10.2, ran kaltura-sanity.sh. Mostly looks good, but a few problems. Any help/suggestion interpreting this would be welcome :smile:

[root@Kmedia ~]# kaltura-sanity.sh

[Space on /] [PASSED, RC: 0] - [.007037028]
[Space on /opt/kaltura/web] [PASSED, RC: 0] - [.004906727]
[Check httpd daemon status] [PASSED, RC: 0] - [.036216137]
Napping 1 seconds to allow the daemon httpd to finish its init after monit restarted itā€¦
ā€¦ x 60
Napping 1 seconds to allow the daemon httpd to finish its init after monit restarted itā€¦
[Check httpd daemon is started by Monit] [FAILED, RC: 1] - [63.710936934]
[check daemon httpd init status] [PASSED, RC: 0] - [.006503049]
[Check kaltura-sphinx daemon status] [PASSED, RC: 0] - [.106633651]
Napping 1 seconds to allow the daemon kaltura-sphinx to finish its init after monit restarted itā€¦
ā€¦ x 60
Napping 1 seconds to allow the daemon kaltura-sphinx to finish its init after monit restarted itā€¦
[Check kaltura-sphinx daemon is started by Monit] [FAILED, RC: 1] - [64.857607907]
[check daemon kaltura-sphinx init status] [PASSED, RC: 0] - [.008610824]
[Check kaltura-batch daemon status] [FAILED, RC: 1] - [.086186348]
Napping 1 seconds to allow the daemon kaltura-batch to finish its init after monit restarted itā€¦
[Check kaltura-batch daemon is started by Monit] [PASSED, RC: 0] - [1.140704292]
[check daemon kaltura-batch init status] [PASSED, RC: 0] - [.008415255]
[Check kaltura-monit daemon status] [PASSED, RC: 0] - [.029573450]
[check daemon kaltura-monit init status] [PASSED, RC: 0] - [.008793379]
[Check memcached daemon status] [PASSED, RC: 0] - [.046177191]
Napping 1 seconds to allow the daemon memcached to finish its init after monit restarted itā€¦
ā€¦ x 60
Napping 1 seconds to allow the daemon memcached to finish its init after monit restarted itā€¦
[Check memcached daemon is started by Monit] [FAILED, RC: 1] - [62.622150262]
[check daemon memcached init status] [PASSED, RC: 0] - [.008603989]
[kaltura-html5lib ver in KMC config.ini] [PASSED, RC: 0] - [.173217783]
[kaltura-kdp3 ver in KDP3 config.ini] [PASSED, RC: 0] - [.104853911]
[kaltura-kmc ver in KMC config.ini] [PASSED, RC: 0] - [.115506721]
[Get KMC SWFs] [PASSED, RC: 0] - [.770672277]
[check_testme_page] [PASSED, RC: 0] - [10.842049017]
[check_kmc_index_page] [PASSED, RC: 0] - [.304659386]
[check_admin_console_index_page] [PASSED, RC: 0] - [.624825223]
[check_studio_index_page] [PASSED, RC: 0] - [1.186695565]
[Create Partner] [PASSED, RC: 0] - [2.907739866]
[Local dropfolder creation succeeded] [PASSED, RC: 0] - [1.167132174]
[Create flavor param] [PASSED, RC: 0] - [1.557222655]
[Delete flavor param] [PASSED, RC: 0] - [1.089803185]
[Upload content kaltura_logo_animated_blue.flv] [PASSED, RC: 0] - [2.106538482]
Napping 10 seconds to allow entry 0_x8x6gesr to digestā€¦
Napping 10 seconds to allow entry 0_x8x6gesr to digestā€¦
Napping 10 seconds to allow entry 0_x8x6gesr to digestā€¦
Napping 10 seconds to allow entry 0_x8x6gesr to digestā€¦
Napping 10 seconds to allow entry 0_x8x6gesr to digestā€¦
Napping 10 seconds to allow entry 0_x8x6gesr to digestā€¦
Napping 10 seconds to allow entry 0_x8x6gesr to digestā€¦
Napping 10 seconds to allow entry 0_x8x6gesr to digestā€¦
Napping 10 seconds to allow entry 0_x8x6gesr to digestā€¦
[kaltura_logo_animated_blue.flv - 0_x8x6gesr status] [FAILED, RC: 1] - [-.924220785]
Napping 30 seconds to allow mail to be sent outā€¦
[Found an email sending entry for mb-Kmedia.nipissingu.ca@kaltura.com[PID is 102] in /var/log/maillog] [PASSED, RC: 0] - [30.046611555]
Testing analytics, be patientā€¦

Please note: if you are running this test on a clustered ENV, it will fail but this does not mean there is an actual problem.
The tech information as to why is available here:

[DWH cycle] [FAILED, RC: 255] - [1.460081954]
[Upload content kaltura_logo_animated_green.flv] [PASSED, RC: 0] - [1.833282291]
[Upload bulk using CSV] [PASSED, RC: 0] - [1.281106643]
[Upload bulk using XML] [PASSED, RC: 0] - [1.231636143]
[Create player] [PASSED, RC: 0] - [.817816070]
[Generate thumb] [FAILED, RC: 255] - [.726898284]
[Delete parnter] [PASSED, RC: 0] - [.662459233]
[Red5 file upload] [SKIPPED as OflaDemo isnā€™t configured]
see: https://github.com/kaltura/platform-install-packages/blob/master/doc/install-kaltura-redhat-based.md#configure-red5-server

Thank you for running Kaltura! To keep Kaltura viable, stable and well tested, please join the community and help by contributing sanity tests that verify overall platform stability: http://bit.ly/kaltura-ci , and by contributing to the project roadmap by solving simple tasks and challenges: http://bit.ly/kaltura-tasks.

[root@Kmedia ~]#

2

Hello,

In general, things seem healthy enough. Only prob seems to be the entry did not convert but sometimes this happens because it just takes too long to finish, for example if you have a long queue and only one batch daemon.
Did you try uploading manually?

3

Hi yes. I did try to upload one manually, and it is not processing at all* Iā€™m also getting an ā€˜Internal Server Errorā€™ from analytics. The kaltura-sphinx service seems to be running.

A little historyā€¦

This install started as a 9.18. Everything seemed OK. I created a publisher, did a couple upload tests, all seemed well. I then tried adding ssl, a process I struggled with, but with your help, eventually got through. Along the way, I updated to 10.2. So now, I have 10.2 with ssl but

  • conversion of uploads isnā€™t working
  • analytics produces internal errors
  • creating a second publisher did not produce the sample media (Iā€™m guessing the conversion issues)
  • Universal Studio players require an update, but the update fails with ā€œError updating UIConf:ā€ Attempting to duplicate a player produces the same result
  • preview in the content for the publisher that was working (when it was 9.18 with no ssl) no longer plays

It seems three issues (analytics, conversions, uiconfs). I donā€™t mind blowing everything away and starting fresh, but troubleshooting may be a good learning experience too. What do you advise Jess?

4

Firstly, yes, I do think troubleshooting this can teach you a lot about the system and is a good idea:)
Lets start with the batchā€¦
Is it currently running?
Can be verified with:
# /etc/init.d/kaltura-batch status
or:
# ps fax|grep batch
If it is not running, let us look at /opt/kaltura/log/kaltura_batch.log and find out why.
If it IS running, lets check /opt/kaltura/log/batch/*log to see what errors weā€™re getting.

Once this is resolved we can move on to other things.

5

Hi Jess,

Lets start with the batchā€¦
Is it currently running?
Can be verified with:

# /etc/init.d/kaltura-batch status

Service Batch isnā€™t running

If it is not running, let us look at /opt/kaltura/log/kaltura_batch.log and find out why.

# tail -n 50 /opt/kaltura/log/kaltura_batch.log

lots of blocks like thisā€¦

2015-01-22 10:34:23 [1996871653] [KSchedulerConfig->load] NOTICE: loading configuration /opt/kaltura/app/configurations/batch at 1421771741
PHP Fatal error:  Uncaught exception 'Zend_Config_Exception' with message 'Section 'Kmedia.nipissingu.ca' cannot be found in /opt/kaltura/app/cache//batch/config.ini' in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php:150
Stack trace:
#0 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(59): Zend_Config_Ini->__construct('/opt/kaltura/ap...', 'Kmedia.nipissin...', true)
#1 /opt/kaltura/app/batch/scheduler/KSchedulerConfig.class.php(42): KSchedulerConfig->load()
#2 /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php(81): KSchedulerConfig->__construct('/opt/kaltura/ap...')
#3 /opt/kaltura/app/batch/scheduler/KGenericScheduler.class.php(63): KGenericScheduler->loadConfig('/opt/kaltura/ap...')
#4 /opt/kaltura/app/batch/KGenericBatchMgr.class.php(37): KGenericScheduler->__construct('/usr/bin/php', '/opt/kaltura/ap...')
#5 {main}
  thrown in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Config/Ini.php on line 150
6

OK, our error of interest is:
PHP Fatal error: Uncaught exception ā€˜Zend_Config_Exceptionā€™ with message ā€˜Section ā€˜Kmedia.nipissingu.caā€™ cannot be found in /opt/kaltura/app/cache//batch/config.iniā€™ in

To fix it, you can re-run /opt/kaltura/bin/kaltura-batch-config.sh

7

Yes. That got batch going, which did the conversions on my test upload and the missing sample media on the new publisher. Also sent the email for the new publisher.

One down!

8

Good.

Now by order:
- analytics produces internal errors
Look at the /opt/kaltura/dwh/logs for thge string ā€˜ERRORā€™ and lets see

  • creating a second publisher did not produce the sample media (Iā€™m guessing the conversion issues)
    This probably works now, correct?
  • Universal Studio players require an update, but the update fails with ā€œError updating UIConf:ā€ Attempting to duplicate a player produces the same result
    How about when creating a new player?
  • preview in the content for the publisher that was working (when it was 9.18 with no ssl) no longer plays
    Please run a sniffer and let us see the calls it is making.
9

Sorry about the delayā€¦ my notification of reply went to junk this time. Anywaysā€¦

WOWā€¦ a lot of logs!

Some of the older ones have

/opt/kaltura/dwh/logs/etl_daily-20150114.log:ERROR 14-01 00:59:09,108 - Run kjbVar job - Unable to open file appender for file [${LogPath}/log_${jobVar}.log] : org.pentaho.di.core.exception.KettleFileException: 
/opt/kaltura/dwh/logs/etl_daily-20150114.log:ERROR 14-01 00:59:09,109 - Run kjbVar job - org.pentaho.di.core.exception.KettleFileException: 
/opt/kaltura/dwh/logs/etl_daily-20150114.log:ERROR 14-01 00:59:09,113 - Abort job 1 - Aborting job.

BUT. I think the issue giving me internal error messages in the GUI is that all the newer logs only have

[root@Kmedia logs]# tail etl_hourly-20150123-10.log
sh: /opt/kaltura/bin/pentaho/kitchen.sh: No such file or directory

Thatā€™s correct

Creating a new player, and then duplicating that seemed to work as expected

Iā€™ll work on that shortly, and get back to youā€¦ juggling a couple balls today :smile:

Thanks Jess!

10

In regards to media not playing, although I was getting no message in the preview and embed screen, I do get a message ā€˜Media not foundā€™ when previewing a specific flavour. In the case of a file I just uploaded, these are the detailsā€¦

Entry Name: 1test
Entry Id: 0_ihfuxwsq
Flavor Name: Source
Flavor Asset Id: 0_u7zcz32b
Bitrate: 642
Codec: avc1
Dimensions: 640 x 480
Format:
Size (KB): 61747
Status: OK

These are whatā€™s in /opt/kaltura/web/content/entry/data/0/0

-rw-rw-r--. 1 kaltura kaltura    32920 Jan 23 13:27 0_ihfuxwsq_0_bz5eg5qq_0.conv.log
-rw-rw-r--. 1 kaltura kaltura 72318056 Jan 23 13:27 0_ihfuxwsq_0_bz5eg5qq_2.mp4
-rw-rw-r--. 1 kaltura kaltura    31937 Jan 23 13:25 0_ihfuxwsq_0_qzcxm6w4_0.conv.log
-rw-rw-r--. 1 kaltura kaltura 63178157 Jan 23 13:25 0_ihfuxwsq_0_qzcxm6w4_2.mp4
-rw-rw-r--. 1 kaltura kaltura    29549 Jan 23 13:23 0_ihfuxwsq_0_odx3eym2_0.conv.log
-rw-rw-r--. 1 kaltura kaltura 45147353 Jan 23 13:23 0_ihfuxwsq_0_odx3eym2_2.mp4
-rw-rw-r--. 1 kaltura kaltura    90603 Jan 23 13:21 0_ihfuxwsq_0_ubu2extx_12.jpg
-rw-r-----. 1 apache  kaltura 63196492 Jan 23 13:21 0_ihfuxwsq_0_u7zcz32b_12.mov

As to the analytics, I found kitchen.sh at

/opt/kaltura/pentaho/pdi/kitchen.sh

Should I make a link?

11

Hi @jess,

I updated to 10.3 this morning, and that didnā€™t make any difference to these remaining issues. I may just have to nuke everything and start fresh. I have to get something running :smile:

v

12

@jess

I have this same issue with DWH stats not displaying and DWH is the only thing to fail duing my sanity check. I ran through the steps in this troubleshooting page but encountered several errors.

My stats are still not working and I have not found any specific errors related to stats when running ā€˜kaltlogā€™. What do you recommend on doing to track down this issue. I resolved it on my other system but this one is not taking the changes as expected. The option of dropping the data and reinstalling is not available in this situation as it is a live system.

13

Please see my reply to: Still Struggling with DWH cycle - new install