Hi there!
Like the title says - I have absolutely no activity recorded/displayed in the form of any reports. Please help me troubleshoot. Obviously, during sanity check the DWH cycle fails…
Here’s the sanity check sql file in /tmp - http://pastebin.com/v56g3T8R
And here’s the grep ERROR result on all logs - http://pastebin.com/qR1q3aDW
All pointers and help will be much appreciated!
Thanks.
Hello,
Indeed quite a few errors as far as Analytics goes…
The rest seems to be fine, which is good:)
For analytics, please look at the logs under /opt/kaltura/dwh/logs, starting just above the first ERROR and paste the relevant output so we can better understand the cause.
Also, what MySQL version are you working with?
Jess, thanks for your assistance, but I’m really not sure what you’re asking me to do here. Please elaborate.
mysql --version
mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1Jess,
While patiently waiting for some community support, I’m also posting the output of running > etl_hourly.sh
Pretty much the only error I’m getting is (at the very end of the log):
sh: /opt/kaltura/bin/pentaho/kitchen.sh: No such file or directory
Does this give us a better idea how to proceed with troubleshooting?
Thanks!
OK, scratch that. I managed to mess it all up to the point where a fresh install was needed.
OS is CentOS 7 now, database is:
mysql Ver 15.1 Distrib 5.5.44-MariaDB, for Linux (x86_64) using readline 5.1
Still, the problem with reporting persists - analytics is not working, and this time around it doesn’t display zero values - it simply shows the entire reporting screen in KMC greyed out. “Internal server error occurred” pops up the first time.
Sanity check fails during the DWH cycle with:
INSERT INTO csi_log VALUES(NULL,‘DWH cycle’,‘centos’,1447519792,“Sent event [KalturaStatsEventType::WIDGET_LOADED] Sent event [KalturaStatsEventType::MEDIA_LOADED] Sent event [KalturaStatsEventType::PLAY] Sent event [KalturaStatsEventType::PLAY_REACHED_25] Sent event [KalturaStatsEventType::PLAY_REACHED_50] Sent event [KalturaStatsEventType::PLAY_REACHED_75] Sent event [KalturaStatsEventType::PLAY_REACHED_100] Executing [logrotate -f /etc/logrotate.d/kaltura_apache]Redirecting to /bin/systemctl reload httpd.service failed”,255,11.770679758,‘11.1.0’);
I can do “service httpd restart” from the shell without any issues.
Meanwhile, the daily/hourly logs contain a single line:
sh: /opt/kaltura/bin/pentaho/kitchen.sh: No such file or directory
Here’s grep for ERROR:
grep ERROR /opt/kaltura/dwh/logs/*
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,900 - Register server - Unexpected error
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,901 - Register server - org.pentaho.di.core.exception.KettleStepException:
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,902 - register_etl_server - Errors detected!
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,902 - register_etl_server - Errors detected!
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,904 - Kitchen - Finished with errors
Ideas please?
Also, there are some error when running kaltura-run-dwh.sh
./kaltura-run-dwh.sh
reading config file /etc/logrotate.d/kaltura_apache
Handling 4 logs
rotating pattern: /opt/kaltura/log/kaltura_apache_errors.log forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 7373
considering log /opt/kaltura/log/kaltura_apache_errors.log
log needs rotating
rotating log /opt/kaltura/log/kaltura_apache_errors.log, log->rotateCount is 5
dateext suffix '-20151114’
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
destination /opt/kaltura/log/kaltura_apache_errors.log-20151114.gz already exists, skipping rotation
switching euid to 0 and egid to 0
rotating pattern: /opt/kaltura/log/kaltura_apache_access.log forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 7373
considering log /opt/kaltura/log/kaltura_apache_access.log
log needs rotating
rotating log /opt/kaltura/log/kaltura_apache_access.log, log->rotateCount is 5
dateext suffix '-20151114’
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding old rotated logs failed
renaming /opt/kaltura/log/kaltura_apache_access.log to /opt/kaltura/log/kaltura_apache_access.log-20151114
compressing log with: /bin/gzip
switching uid to 0 and gid to 7373
running last action script
switching euid to 0 and egid to 0
Redirecting to /bin/systemctl reload httpd.service
switching euid to 0 and egid to 0
rotating pattern: /opt/kaltura/log/kaltura_apache_errors_ssl.log forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 7373
considering log /opt/kaltura/log/kaltura_apache_errors_ssl.log
log /opt/kaltura/log/kaltura_apache_errors_ssl.log does not exist – skipping
not running postrotate script, since no logs were rotated
switching euid to 0 and egid to 0
rotating pattern: /opt/kaltura/log/kaltura_apache_access_ssl.log forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 7373
considering log /opt/kaltura/log/kaltura_apache_access_ssl.log
log /opt/kaltura/log/kaltura_apache_access_ssl.log does not exist – skipping
not running last action script, since no logs will be rotated
switching euid to 0 and egid to 0
ERROR 1305 (42000) at line 1: PROCEDURE kalturadw.get_data_for_operational does not exist
ERROR 1305 (42000) at line 1: PROCEDURE kalturadw.mark_operational_sync_as_done does not exist
Ok…having run this, now there’s plenty in the DWH logs.
Daily log:
INFO 14-11 22:23:33,337 - daily - Starting entry [Check if update_dims_lock is seized]
ERROR 14-11 22:23:33,523 - Check if update_dims_lock is seized - An error occurred executing this job entry:
An error occurred executing SQL:
SELECT * FROM locks
WHERE lock_name = ‘update_dims_lock’ AND lock_state = 0
Table ‘kalturadw_ds.locks’ doesn’t exist
update_dims:
INFO 14-11 22:23:29,619 - Kitchen - Start of run.
INFO 14-11 22:23:29,744 - Update Dimensions - Start of job execution
ERROR 14-11 22:23:29,927 - Update Dimensions - A serious error occurred during job execution: org.pentaho.di.core.exception.KettleJobException:
Unable to begin processing by logging start in logtable etl_log
Couldn’t execute SQL: LOCK TABLES etl_log WRITE
Table ‘kalturalog.etl_log’ doesn’t exist
INFO 14-11 22:23:29,928 - Kitchen - Finished!
ERROR 14-11 22:23:29,928 - Kitchen - Finished with errors
INFO 14-11 22:23:29,928 - Kitchen - Start=2015/11/14 22:23:29.620, Stop=2015/11/14 22:23:29.928
INFO 14-11 22:23:29,928 - Kitchen - Processing ended after 0 seconds.
Hourly:
INFO 14-11 22:23:29,619 - Kitchen - Start of run.
INFO 14-11 22:23:29,744 - Update Dimensions - Start of job execution
ERROR 14-11 22:23:29,927 - Update Dimensions - A serious error occurred during job execution: org.pentaho.di.core.exception.KettleJobException:
Unable to begin processing by logging start in logtable etl_log
Couldn’t execute SQL: LOCK TABLES etl_log WRITE
Table ‘kalturalog.etl_log’ doesn’t exist
INFO 14-11 22:23:29,928 - Kitchen - Finished!
ERROR 14-11 22:23:29,928 - Kitchen - Finished with errors
INFO 14-11 22:23:29,928 - Kitchen - Start=2015/11/14 22:23:29.620, Stop=2015/11/14 22:23:29.928
INFO 14-11 22:23:29,928 - Kitchen - Processing ended after 0 seconds.
[root@centos logs]# cat etl_hourly-20151114-22.log
INFO 14-11 22:23:25,849 - Kitchen - Start of run.
INFO 14-11 22:23:25,940 - hourly - Start of job execution
INFO 14-11 22:23:25,943 - hourly - Starting entry [Register etl_server]
INFO 14-11 22:23:25,944 - Register etl_server - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/register_etl_server.ktr]
INFO 14-11 22:23:25,990 - register_etl_server - Dispatching started for transformation [register_etl_server]
INFO 14-11 22:23:26,076 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
INFO 14-11 22:23:26,215 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 14-11 22:23:26,219 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,246 - Set hostname command - Optimization level not specified. Using default of 9.
INFO 14-11 22:23:26,328 - Set hostname command - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,331 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,332 - Select hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,333 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,334 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
ERROR 14-11 22:23:26,350 - Register server - Unexpected error
ERROR 14-11 22:23:26,350 - Register server - org.pentaho.di.core.exception.KettleStepException:
Error in step, asking everyone to stop because of:
Error looking up row in database
Table ‘kalturadw_ds.etl_servers’ doesn’t exist
at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.processRow(InsertUpdate.java:307) at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40) at java.lang.Thread.run(Thread.java:745)Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Error looking up row in database
Table ‘kalturadw_ds.etl_servers’ doesn’t exist
at org.pentaho.di.core.database.Database.getLookup(Database.java:3120) at org.pentaho.di.core.database.Database.getLookup(Database.java:3093) at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.lookupValues(InsertUpdate.java:80) at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.processRow(InsertUpdate.java:290) ... 2 moreCaused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table ‘kalturadw_ds.etl_servers’ doesn’t exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:389)
at com.mysql.jdbc.Util.getInstance(Util.java:372)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:980)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2535)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1911)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2034)
at org.pentaho.di.core.database.Database.getLookup(Database.java:3101)
… 5 more
INFO 14-11 22:23:26,351 - Register server - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
INFO 14-11 22:23:26,351 - register_etl_server - register_etl_server
INFO 14-11 22:23:26,351 - register_etl_server - register_etl_server
ERROR 14-11 22:23:26,351 - register_etl_server - Errors detected!
INFO 14-11 22:23:26,353 - hourly - Finished job entry [Register etl_server] (result=[false])
INFO 14-11 22:23:26,353 - hourly - Job execution finished
INFO 14-11 22:23:26,353 - Kitchen - Finished!
ERROR 14-11 22:23:26,353 - Kitchen - Finished with errors
INFO 14-11 22:23:26,353 - Kitchen - Start=2015/11/14 22:23:25.850, Stop=2015/11/14 22:23:26.353
INFO 14-11 22:23:26,353 - Kitchen - Processing ended after 0 seconds.
cat etl_perform_retention_policy-20151114.log
INFO 14-11 22:23:36,563 - Kitchen - Start of run.
INFO 14-11 22:23:36,650 - perform retention policy - Start of job execution
ERROR 14-11 22:23:36,833 - perform retention policy - A serious error occurred during job execution: org.pentaho.di.core.exception.KettleJobException:
Unable to begin processing by logging start in logtable etl_log
Couldn’t execute SQL: LOCK TABLES etl_log WRITE
Table ‘kalturalog.etl_log’ doesn’t exist
INFO 14-11 22:23:36,834 - Kitchen - Finished!
ERROR 14-11 22:23:36,834 - Kitchen - Finished with errors
INFO 14-11 22:23:36,834 - Kitchen - Start=2015/11/14 22:23:36.564, Stop=2015/11/14 22:23:36.834
INFO 14-11 22:23:36,834 - Kitchen - Processing ended after 0 seconds.