Hi Rahul,
I am glad analytics now works well.
As for batches, I am not 100% I understand… do you see hosts that were previously installed as batches and are no longer active and want to remove them? if so, see:
if I did not correctly understand your question, please reiterate.
Thanks,
I have nothing like this in my logs. I see the entries in the database and the files and not errors are returned with the following command.
Hello Jess,
We also are having issues:
1 - imbedded video
2 - views DO NOT increase when we play the video
3 - To make things more confusing we did run the scripts from URL
WHEN we ran this the views updated to the CORRECT views this first time we ran it.
4 - After that the views did NOT again show up. We ran the scripts in 3 above and again it did not update.
5 - Found your post here and did the
grep –Fn ‘GET //api_v3/index.php?service=stats&action=collect’ kaltura_apache_access.log
I cannot find any of the GET statements.
6 - however if I just for part of the key idea I believe behind the command GET I searched for this:
grep ‘action=collect’ kaltura_apache_access.log
135.23.132.132 - - [18/Jun/2016:22:41:17 -0400] “GET /api_v3/index.php?service=stats&apiVersion=3.1&expiry=86400&clientTag=kwidget%3Av2.34&format=1&ignoreNull=1&action=collect&event:eventType=2&event:clientVer=2.34&event:currentPoint=0&event:duration=76&event:eventTimestamp=1466304029226&event:isFirstInSession=false&event:objectType=KalturaStatsEvent&event:partnerId=101&event:sessionId=a91d718f-0602-3d15-0fb6-0160fa692861&event:uiconfId=23448242&event:seek=false&event:entryId=0_r540cjj6&event:widgetId=_101&event:referrer=http%253A%252F%252Fwww.indievue.com%252F&kalsig=b603ee86cd52cfcce3cb6d363e90c55a HTTP/1.1” 200 410 0/61299 “http://www.indievue.com/” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36” “-” 135.23.132.132 “-” “vs.indievue.com” 5958 364083668 - 906 “-” “-” “-” “-” “no-store, no-cache, must-revalidate, post-check=0, pre-check=0” -
135.23.132.132 - - [18/Jun/2016:22:41:17 -0400] “GET /api_v3/index.php?service=stats&apiVersion=3.1&expiry=86400&clientTag=kwidget%3Av2.34&format=1&ignoreNull=1&action=collect&event:eventType=1&event:clientVer=2.34&event:currentPoint=0&event:duration=76&event:eventTimestamp=1466304029314&event:isFirstInSession=false&event:objectType=KalturaStatsEvent&event:partnerId=101&event:sessionId=a91d718f-0602-3d15-0fb6-0160fa692861&event:uiconfId=23448242&event:seek=false&event:entryId=0_r540cjj6&event:widgetId=_101&event:referrer=http%253A%252F%252Fwww.indievue.com%252F&kalsig=2ee9063deaf949fab6d11a05872bf348 HTTP/1.1” 200 410 0/39438 “http://www.indievue.com/” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36” “-” 135.23.132.132 “-” “vs.indievue.com” 5950 299720959 - 906 “-” “-” “-” “-” “no-store, no-cache, must-revalidate, post-check=0, pre-check=0” -
135.23.132.132 - - [18/Jun/2016:22:41:57 -0400] “GET /api_v3/index.php?service=stats&apiVersion=3.1&expiry=86400&clientTag=kwidget%3Av2.34&format=1&ignoreNull=1&action=collect&event:eventType=2&event:clientVer=2.34&event:currentPoint=0&event:duration=76&event:eventTimestamp=1466304069479&event:isFirstInSession=false&event:objectType=KalturaStatsEvent&event:partnerId=101&event:sessionId=4f1ca75a-b841-a306-7005-b80d477e45e5&event:uiconfId=23448242&event:seek=false&event:entryId=0_r540cjj6&event:widgetId=_101&event:referrer=http%253A%252F%252Fwww.indievue.com%252F&kalsig=1829075cb77205aa09fd7861203f89e3 HTTP/1.1” 200 411 0/14307 “http://www.indievue.com/” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36” “-” 135.23.132.132 “-” “vs.indievue.com” 5951 1105144487 - 906 “-” “-” “-” “-” “no-store, no-cache, must-revalidate, post-check=0, pre-check=0” -
135.23.132.132 - - [18/Jun/2016:22:41:57 -0400] “GET /api_v3/index.php?service=stats&apiVersion=3.1&expiry=86400&clientTag=kwidget%3Av2.34&format=1&ignoreNull=1&action=collect&event:eventType=1&event:clientVer=2.34&event:currentPoint=0&event:duration=76&event:eventTimestamp=1466304069546&event:isFirstInSession=false&event:objectType=KalturaStatsEvent&event:partnerId=101&event:sessionId=4f1ca75a-b841-a306-7005-b80d477e45e5&event:uiconfId=23448242&event:seek=false&event:entryId=0_r540cjj6&event:widgetId=_101&event:referrer=http%253A%252F%252Fwww.indievue.com%252F&kalsig=f3d3b05aae66dcd642b04495bab1e7ea HTTP/1.1” 200 411 0/13677 “http://www.indievue.com/” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36” “-” 135.23.132.132 “-” “vs.indievue.com” 5958 1973129878 - 906 “-” “-” “-” “-” “no-store, no-cache, must-revalidate, post-check=0, pre-check=0” -
7 - doing this command #more kaltura_apache_errors.log yields these errors.
[Sat Jun 18 04:50:48 2016] [error] [client 185.25.148.240] script ‘/opt/kaltura/app/alpha/web/testproxy.php’ not found or unable to stat
[Sat Jun 18 08:22:45 2016] [error] [client 169.229.3.91] Invalid URI in request Wg6\xad\xa8\x98\x
c9R\xcelZ\xaf\vudQ\x82\xe5o\x91\xb8\xb7\xbd\xfd\xcfb^$[:j\xd3\x10\xebX\x9aw\xcb~\x07\x17\xd0n\xcd
[Sat Jun 18 09:56:14 2016] [error] [client 185.25.151.159] script ‘/opt/kaltura/app/alpha/web/testproxy.php’ not found or unable to stat
[Sat Jun 18 19:35:02 2016] [error] [client 185.25.151.159] script ‘/opt/kaltura/app/alpha/web/testproxy.php’ not found or unable to stat
Jess what are we doing wrong - where do we look.
AND Just as an aside we found the hourly script.
8 - I have this 8th point which I was yelled at a few times for asking by colleagues. They just want to find the update script and error. But my thinking is that we are on the wrong track?
If someone views an imbedded video, should the view not go up 1 counter IMMEDIATELY? Why 1 hour or 1 day as I saw in another post.
Anyway can you help us Kaltura peoples.
thanks danny
Hello All,
Sorry for putting so much detail in the original note. Can anyone help us please with the Kaltura issues we have. thanks danny
Hi Danny,
Not sure I understand what the current situation is…
Let’s take a step back. Analytics should update every four hours. The way it works is this:
It is also important to note that plays made from with in the KMC will not be counted, only plays made by embedding the player into external pages will show.
For a step by step debugging, please follow:
And let’s see where we’re stuck.
Thanks,
Really appreciate it. This will help a lot I think.
There is one other question I have which really does not make sense to me if you think about the action.
The play video and collection of the stats is really important. I get that. I understand that. Even on youtube a “view” is extremely important.
So in our discussion here I see the update of the stats is 4 hours in your response. We actually tracked the scripts down and we thought it was 1 hour.
My question is this. Since it is so important, would we be able to instead of making it a crontab every four hours, set it to be a process that when a view does happen and stats are collected, a process is set off to record that stats right away. This can be on the “fly” with multiple views being queued until the current view is calculated. As an IT programmer (old one) I can totally see how we could do it. But in this technology perhaps there is a reason it is not being done. Why ?
I hope I explained that curiosity question better though long winded. thanks danny
I run into the same issue, and even after re-install of DHW i receive this error related to the Plugins
Check if the plugin is available in the plugins subdirectory of the Kettle distribution
/opt/kaltura/dwh/logs/log_events_events.log:2016/06/30 16:14:20 - file_processing_job - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Unable to load class for step/plugin with id [ConcurrentSafeCombinationLookup]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.
/opt/kaltura/dwh/logs/log_events_events.log:2016/06/30 16:14:20 - file_processing_job - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_events_events.log:2016/06/30 16:14:20 - file_processing_job - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.trans.step.StepMeta.(StepMeta.java:239)
/opt/kaltura/dwh/logs/log_events_events.log:2016/06/30 16:14:20 - file_processing_job - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : … 10 more
root@video:~#
Any idea?
I have copied missing plugins over from /opt/kaltura/dwh/pentaho-plugins
to /opt/kaltura/pentaho/pdi/plugins/steps
After I proceed with Acess Log Testing, and came to following result:
grep ERROR /opt/kaltura/dwh/logs/*
/opt/kaltura/dwh/logs/etl_perform_retention_policy-20160630.log:ERROR 30-06 16:40:43,558 - Lock is already seized - Row nr 1 causing abort : [retention_lock], [1], [0], [null]
/opt/kaltura/dwh/logs/etl_perform_retention_policy-20160630.log:ERROR 30-06 16:40:43,559 - Lock is already seized - Aborting after having seen 1 rows.
/opt/kaltura/dwh/logs/etl_perform_retention_policy-20160630.log:ERROR 30-06 16:40:43,559 - seize_lock_by_name - Errors detected!
/opt/kaltura/dwh/logs/etl_perform_retention_policy-20160630.log:ERROR 30-06 16:40:43,564 - seize_lock_by_name - Errors detected!
/opt/kaltura/dwh/logs/etl_update_dims-20160630-16.log:ERROR 30-06 16:39:26,946 - Mapping with Field Input - Unexpected error
/opt/kaltura/dwh/logs/etl_update_dims-20160630-16.log:ERROR 30-06 16:39:26,947 - Mapping with Field Input - org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_update_dims-20160630-16.log:ERROR 30-06 16:39:27,010 - Mapping with Field Input - Unexpected error
/opt/kaltura/dwh/logs/etl_update_dims-20160630-16.log:ERROR 30-06 16:39:27,011 - Mapping with Field Input - org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_update_dims-20160630-16.log:ERROR 30-06 16:39:27,053 - iterate_sequences - Errors detected!
/opt/kaltura/dwh/logs/etl_update_dims-20160630-16.log:ERROR 30-06 16:39:27,054 - iterate_sequences - Errors detected!
Do I understand correct, this error is because the log gotten alrady rotated before, even I uploaded for testing the access log kaltura_apache_access.log-20160630.gz freshly, where I see the played Videos proberply ?
Edit: https://github.com/kaltura/platform-install-packages/issues/94
I am trying to run the command later today when I have access from the development teams.
grep ERROR /opt/kaltura/dwh/logs/*
However just to state we have an issue here and we with your help trying to find out. But here is what happens:
We run the commands piecemeal…
from /opt/kaltura/bin/kaltura-run-dwh.sh except for the first 2 lines:
i…e
su kaltura -c "/opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh"
su kaltura -c "/opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh
-p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh"
su kaltura -c "/opt/kaltura/dwh/etlsource/execute/etl_daily.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh"
su kaltura -c "/opt/kaltura/dwh/etlsource/execute/etl_perform_retention_policy.sh
-p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh"
su kaltura -c "/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh
/opt/kaltura/log/cron.log"
if it works that mean our cron job is not doing it right,
because when I try to run it from root or kaltura the following didn’t work.
00 * * * * kaltura /opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
00 * * * * kaltura
/opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
59 0,4,8,12,16,20 * * * kaltura
/opt/kaltura/dwh/etlsource/execute/etl_daily.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
30 12 * * * kaltura
/opt/kaltura/dwh/etlsource/execute/etl_perform_retention_policy.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
0 10 * * * kaltura
/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log
the difference is the cron job…1. doesn’t have -c and the whole lines are not in quotes…
Just stating but well have the error log soon. thanks for help guys.
Ok guys we really need help please.
Again to review the situation.
The analytics are not showing the PLAY VIEWS correctly.
We did use the command
grep ERROR /opt/kaltura/dwh/logs/*
It comes up completely blank. no errors.
One of the managers told me this:
1 - they think it is something with crobtab.
When they ran the installers from kaltura package, the installers updated the cron.
Once everything was setup that was when they noticed this was not working properly.
2 - IF they run the command that is being discussed here:
/etc/cron.d/kaltura-dwh
THey run this or individually the commands within the script they all work.
3 - We notice that the first time MANUALLY running it worked for the script.
4 - Automatically does NOT work on updating the analtics.
5 - We are wondering if it is about the ownership of the scripts and does it matter who runs it?
Can you please help us. Please we need help and how can we get a time response?
thanks danny
@jess
I drop the db again and ran configuration script which showed the following after.
/opt/kaltura/bin/kaltura-dwh-config.sh /tmp/kaltura_17_05_01_19.ans
kaltura-dwh-11.3.0-1.noarch
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
# rm /opt/kaltura/app/base-config.lock
# /opt/kaltura/bin/kaltura-base-config.sh
Deploying analytics warehouse DB, please be patient as this may take a while...
Output is logged to /opt/kaltura/dwh/logs/dwh_setup.log.
sending incremental file list
MySQLInserter/
sent 133 bytes received 25 bytes 316.00 bytes/sec
total size is 2646419 speedup is 16749.49
sending incremental file list
MappingFieldRunner/
sent 143 bytes received 25 bytes 336.00 bytes/sec
total size is 90670 speedup is 539.70
sending incremental file list
GetFTPFileNames/
sent 137 bytes received 25 bytes 324.00 bytes/sec
total size is 7308893 speedup is 45116.62
sending incremental file list
FetchFTPFile/
sent 126 bytes received 25 bytes 302.00 bytes/sec
total size is 3687964 speedup is 24423.60
sending incremental file list
sent 80 bytes received 12 bytes 184.00 bytes/sec
total size is 34770 speedup is 377.93
current version 5999
DWH configured.
sent 131 bytes received 25 bytes 312.00 bytes/sec
total size is 5783119 speedup is 37071.28
sending incremental file list
DimLookup/
This is the results of the Sanity check.
[kaltura_logo_animated_blue.flv - 0_v7s0l1d7 status] [FAILED, RC: 1] - [-.465484982]
Napping 30 seconds to allow mail to be sent out…
[Found an email sending entry for mb-tig03.hiphopservers.com@kaltura.com[PID is 111] in /var/log/maillog] [PASSED, RC: 0] - [30.112048594]
Testing analytics, be patient…
Please note: if you are running this test on a clustered ENV, it will fail but this does not mean there is an actual problem.
The tech information as to why is available here:
[DWH cycle] [FAILED, RC: 255] - [325.877967047]
Hi
I have the same problem, analytic not updated automatically, i notice that it update once per day. What i found in database
SELECT * FROM dwh_fact_events order by event_time DESC;
last event_time is 2017-01-23 03:51:06
when i check the /var/log/cron logs for this time i found logrotate is called
Jan 23 03:51:01 KaltureServer run-parts(/etc/cron.daily)[17299]: starting logrotate
Jan 23 03:56:53 KaltureServer run-parts(/etc/cron.daily)[17821]: finished logrotate
#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
its run /etc/logrotate.d/
ConsoleKit httpd kaltura_api kaltura_base kaltura_populate mysqld nginx.rpmsave yum
dracut kaltura_apache kaltura_apps kaltura_batch kaltura_sphinx nginx syslog
After this time all Kaltura Cron job running but it doesn’t reflect any update to “dwh_fact_events” or Kaltura GUI.
should I rotate the logs before all Kaltura Cron job running? Any advice will be highly appreciated
Kindly find below # less /var/log/cron as “/opt/kaltura/log/cron.log” only include timestamp without any information
Jan 23 03:51:01 KaltureServer anacron[11091]: Job cron.daily' started Jan 23 03:51:01 KaltureServer run-parts(/etc/cron.daily)[17299]: starting logrotate Jan 23 03:56:53 KaltureServer run-parts(/etc/cron.daily)[17821]: finished logrotate Jan 23 03:56:53 KaltureServer run-parts(/etc/cron.daily)[17299]: starting makewhatis.cron Jan 23 03:56:55 KaltureServer run-parts(/etc/cron.daily)[17947]: finished makewhatis.cron Jan 23 03:56:55 KaltureServer run-parts(/etc/cron.daily)[17299]: starting mlocate.cron Jan 23 03:57:14 KaltureServer run-parts(/etc/cron.daily)[17985]: finished mlocate.cron Jan 23 03:57:14 KaltureServer anacron[11091]: Jobcron.daily’ terminated (mailing output)
Jan 23 03:57:14 KaltureServer anacron[11091]: Normal exit (1 job run)
Jan 23 04:00:01 KaltureServer CROND[18180]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 04:00:01 KaltureServer CROND[18181]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 04:00:01 KaltureServer CROND[18182]: (kaltura) CMD (/opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pd
i/kitchen.sh)
Jan 23 04:00:01 KaltureServer CROND[18183]: (kaltura) CMD (/opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/penta
ho/pdi/kitchen.sh)
Jan 23 04:01:01 KaltureServer CROND[20132]: (root) CMD (run-parts /etc/cron.hourly)
Jan 23 04:01:01 KaltureServer run-parts(/etc/cron.hourly)[20132]: starting 0anacron
Jan 23 04:01:01 KaltureServer run-parts(/etc/cron.hourly)[20141]: finished 0anacron
Jan 23 04:10:01 KaltureServer CROND[22534]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 04:15:01 KaltureServer CROND[23110]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 04:20:01 KaltureServer CROND[23860]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 04:30:01 KaltureServer CROND[24788]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 04:30:01 KaltureServer CROND[24789]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 04:40:01 KaltureServer CROND[26020]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 04:45:01 KaltureServer CROND[26834]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 04:50:01 KaltureServer CROND[27346]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 04:59:01 KaltureServer CROND[27934]: (kaltura) CMD (/opt/kaltura/dwh/etlsource/execute/etl_daily.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi
/kitchen.sh)
Jan 23 05:00:01 KaltureServer CROND[28483]: (kaltura) CMD (/opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/penta
ho/pdi/kitchen.sh)
Jan 23 05:00:01 KaltureServer CROND[28482]: (kaltura) CMD (/opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pd
i/kitchen.sh)
Jan 23 05:00:01 KaltureServer CROND[28481]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 05:00:01 KaltureServer CROND[28484]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 05:01:01 KaltureServer CROND[30001]: (root) CMD (run-parts /etc/cron.hourly)
Jan 23 05:01:01 KaltureServer run-parts(/etc/cron.hourly)[30001]: starting 0anacron
Jan 23 05:01:01 KaltureServer run-parts(/etc/cron.hourly)[30010]: finished 0anacron
Jan 23 05:10:01 KaltureServer CROND[30634]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 05:15:01 KaltureServer CROND[31033]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 05:20:01 KaltureServer CROND[31449]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 05:30:01 KaltureServer CROND[32384]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 05:30:01 KaltureServer CROND[32383]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 05:40:01 KaltureServer CROND[33425]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 05:45:01 KaltureServer CROND[34010]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 05:50:01 KaltureServer CROND[34473]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 06:00:01 KaltureServer CROND[35655]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Hi @nabil_naim1,
Analytics is updated at one day intervals by design, it is not meant to be realtime.
You can change the intervals somewhat in /etc/cron.d/kaltura-dwh but you must consider that each process takes a while to complete [depending on the volume to process, of course].
I see you wrote:
last event_time is 2017-01-23 03:51:06
So, seems as though the stats are correctly populated into the DB. Do you not see the data in KMC or when making direct API requests to the ‘report’ API or were you just asking about the update intervals?
thank you for your reply Jess,
I can see analytics update through GUI but it’s updated once per day as you mention, i read about update statistics every 4 hours? also even if the analytics is updated but video plays in content tab doesn’t updated either daily or hourly ? its always 0 even if its list in top content?
also, can you briefly list the update sequence as I can’t find a description for each job what it does exactly
e.g. 1. rotate logs
2. /opt/Kaltura/DWH/etlsource/execute/etl_update_dims.sh update the database table 1,2
Hi @nabil_naim1,
Some of the jobs happen every hour, some happen every 4 hours and some happen once a day, I wouldn’t recommend messing about with it, however, as I said, the process is not meant to be realtime.
As for the play count in the KMC->Content tab, the script responsible for updating that is:
/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh
Called in /etc/cron.d/kaltura-dwh, please see this post as to potential issues with it and how to resolve them:
Thank you again 
when i run it from shell using
su kaltura -c "/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh"
the content play number updated successfully ?!!!
although using cron job it doesn’t
00 * * * * kaltura /opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
00 * * * * kaltura /opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
59 0,4,8,12,16,20 * * * kaltura /opt/kaltura/dwh/etlsource/execute/etl_daily.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
30 12 * * * kaltura /opt/kaltura/dwh/etlsource/execute/etl_perform_retention_policy.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
0 10 * * * kaltura /opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log
i check also # cat /var/log/cron | grep “00:10” -B2 -A2
Jan 23 00:01:02 KaltureServer anacron[25786]: Jobs will be executed sequentially
Jan 23 00:01:02 KaltureServer anacron[25786]: Normal exit (0 jobs run)
Jan 23 00:10:01 KaltureServer CROND[29111]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 00:15:01 KaltureServer CROND[32064]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 00:20:01 KaltureServer CROND[34724]: (root) CMD (/usr/lib64/sa/sa1 1 1)
As you will notice in the kaltura-dwh hours is only “0” so i fix it to “00”, is that should cause a problem ?
Hi @nabil_naim1,
0 10 should work the same as 00 10 does but I agree that for consistency’s sake it should be corrected. You should still see an entry like:
10:00:01 ce-front0 CROND[9729]: (kaltura) CMD (/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log)
in /var/log/cron, if you don’t, then we have a problem but I imagine you will:)
Another thing to look at is /opt/kaltura/log/cron.log, anything there to indicate a problem?
You are right, this means cron job can’t update play number automatically
Jan 23 09:45:01 KaltureServer CROND[15319]: (apache) CMD (/opt/kaltura/app/alpha/crond/kaltura/clear_cache.sh 2>&1 >> /opt/kaltura/log/cron.log)
Jan 23 09:50:01 KaltureServer CROND[15813]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 10:00:01 KaltureServer CROND[17936]: (kaltura) CMD (/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log)
Jan 23 10:00:01 KaltureServer CROND[17937]: (kaltura) CMD (/opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh)
Jan 23 10:00:01 KaltureServer CROND[17939]: (kaltura) CMD (/opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh)
cat /opt/kaltura/log/cron.log only contains
Mon Jan 23 04:00:01 EET 2017
Mon Jan 23 04:15:01 EET 2017
Mon Jan 23 04:30:01 EET 2017
Mon Jan 23 04:45:01 EET 2017
Mon Jan 23 05:00:01 EET 2017
Mon Jan 23 05:15:01 EET 2017
Mon Jan 23 05:30:01 EET 2017
Mon Jan 23 05:45:01 EET 2017
…
Still i have the same issue running “/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh” from cron job doesn’t update the content video plays although when i run it from script it works fine
#less /var/log/cron | grep -i "dwh_plays_views_sync.sh"
shows
Jan 26 10:00:01 KaltureServer CROND[4005]: (kaltura) CMD (/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log)
cat /opt/kaltura/log/cron.log
shows
Thu Jan 26 14:30:01 EET 2017
Thu Jan 26 14:45:01 EET 2017
Thu Jan 26 15:00:02 EET 2017
Thu Jan 26 15:15:01 EET 2017
Thu Jan 26 15:30:01 EET 2017
Thu Jan 26 15:45:01 EET 2017
Thu Jan 26 16:00:01 EET 2017
Thu Jan 26 16:15:01 EET 2017
Thu Jan 26 16:30:01 EET 2017
Thu Jan 26 16:45:01 EET 2017
(Only time Stamp)
cron job
00 10 * * * kaltura /opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log
script run from bash
[root@KaltureServer log]# su kaltura
[kaltura@KaltureServer log]$ /opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh
after this i find all media plays in content tab update
Also below command shows nothing
Can you change:
00 10 * * * kaltura /opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log
to read:
00 10 * * * kaltura /opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log 2>&1
``
?
this would also redirect output sent to STDERR to /opt/kaltura/log/cron.log. Perhaps there are errors during the execution when it's done by crond