Trouble setting up auto log rotation

Hello

I am having Kaltura Cluster 11.0 CE installed on my server.For the analytics data to reflect in front-end, I have used following scripts from [https://github.com/kaltura/platform-install-packages/blob/master/doc/kaltura-packages-faq.md][1]

rm /opt/kaltura/dwh/logs/*

logrotate -vvv -f /etc/logrotate.d/kaltura_apache

su kaltura -c “/opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh”

su kaltura -c “/opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh”

su kaltura -c “/opt/kaltura/dwh/etlsource/execute/etl_daily.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh”

su kaltura -c “/opt/kaltura/dwh/etlsource/execute/etl_perform_retention_policy.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh”

su kaltura -c “/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log”

Except the # logrotate -vvv -f /etc/logrotate.d/kaltura_apache command, all other commands are run from my DWH-Analytics Server and the # logrotate -vvv -f /etc/logrotate.d/kaltura_apache is run from NFS server.

I want my analytics data on Kaltura Server to fetch on to my front-end after regular intervals, so I have set the above command to be run on DWH in my cronjob for fixed intervals except the # logrotate -vvv -f /etc/logrotate.d/kaltura_apache command which I have to run manually each time on NFS server.Then only it generates log in /opt/kaltura/dwh.

Please suggest some measure so that i can set the logrotate command to be run automatically in NFS server after fixed intervals of time so that my analytics data can come to front-end in regular periods.

Hello,

The log rotation should run from the front servers as it rotates the access logs and places them on the NFS.
There is no need to this manually, when installing kaltura-front the following symlink should be created:
# ll /etc/logrotate.d/kaltura_apache
lrwxrwxrwx. 1 root root 56 Oct 30 08:12 /etc/logrotate.d/kaltura_apache -> /opt/kaltura/app/configurations/logrotate/kaltura_apache

That ensures rotation of the access logs.
As for DWH, that is done here:
# ll /etc/cron.d/kaltura-dwh
lrwxrwxrwx. 1 root root 40 Oct 28 11:18 /etc/cron.d/kaltura-dwh -> /opt/kaltura/app/configurations/cron/dwh

And that symlink is created when installing kaltura-dwh, which you should only install on one machine. It is quite important that this package should not be installed on more than one node since if two instances of the scripts inside the cronjob file run in parallel, it will cause locks to be stuck in the kalturadw_ds.locks table.

Hello Jess,

Thanks for your response as always :smile:

The issue is:

ll /etc/logrotate.d/kaltura_apache

lrwxrwxrwx. 1 root root 56 Oct 30 08:12 /etc/logrotate.d/kaltura_apache -> /opt/kaltura/app/configurations/logrotate/kaltura_apache

The above symlink is created in my Kaltura-front server and also

ll /etc/cron.d/kaltura-dwh

lrwxrwxrwx. 1 root root 40 Oct 28 11:18 /etc/cron.d/kaltura-dwh -> /opt/kaltura/app/configurations/cron/dwh

the above symlink is created when installing kaltura-dwh on our analytics server,and this package is installed on only one machine i.e. on our dedicated DWH-analytics server.

The #logrotate -vvv -f /etc/logrotate.d/kaltura_apache command runs only once a day.
It has run today also and the log file is created in /opt/kaltura/web/logs in DWH server, the details of log file are:
-rw-r–r-- 1 kaltura apache 10499103 Nov 5 03:44 abc.com-kaltura_apache_access.log-20151105-03.gz
but the analytics data is not coming in KMC analytics and the number of plays/views are also not updated in content tab in KMC. Please help in this regard.

Also,how can we set up logrotate command to run after regular intervals like the other dwh scripts set up in the /opt/kaltura/app/configurations/cron/dwh file so that it can rotate logs periodically.

Hello @Mehul,

Welcome, as always.
To troubleshoot why you are not seeing analytics in KMC, follow:


Run the queries there, check the logs under /opt/kaltura/dwh/logs and lets see what the issue is

As for:

You can run logrotate -vvv -f /etc/logrotate.d/kaltura_apache from the crontab for each front node at smaller intervals. I suggest you place a job under /etc/cron.d/

Hello Jess,

I got your point.Can you please tell me how much time does it take to update the analytics data(no. of videos played) in our Kaltura server?

Because right now,I am able to see analytics data upto of tommorrow but not of today’s even though all the commands ran properly and the log files are also generated in /opt/kaltura/web/logs and also in /opt/kaltura/dwh/logs with proper permission.But now we have to run the commands manually to export the data on our kaltura server.

Hello Jess,

Please look in to this.I am facing trouble resolving this issue.

Hello,

The time it takes will depend on how many entries there are to update…

Hi Jess

I have setup logrorate scripts on NFS server in cron tab to run
at 12:00 PM:- logrotate -vvv -f /etc/logrotate.d/kaltura_apache

after that i set following scripts in /etc/cron.d/kaltura-dwh to run at following time

12 :05 PM:- /opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
12:20 PM:- /opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh at 12:15 PM
12:35PM :- /opt/kaltura/dwh/etlsource/execute/etl_daily.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
12:50 PM :-/opt/kaltura/dwh/etlsource/execute/etl_perform_retention_policy.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
13:05 PM :- /opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log

But after that also there is nothing in analytic reports but
when i run all commands manually one by one , It works and shows analytical reports in KMC.

Except crontab, how can I configure the scripts to run automatically?

Hello,

I am a bit confused…
logrotate -vvv -f /etc/logrotate.d/kaltura_apache
needs to run on the front machines since it rotates the local logs from them onto the NFS box.
Also, post changing that, what user are you running the cron jobs as? perhaps it is a permissions issue.

Hi

We have run logrotate -vvv -f /etc/logrotate.d/kaltura_apache on front machine i.e NFS server

First server :- NFS server (front machine + sphinx+ Batch+ nginx+ NFS) separate

Second Server :- DWH server separate
Third Server :- Mysql Server Separate
Fourth Server:- Secondary Batch server

and we are running cron jobs with kaltura:apache user and group.

logrotate -vvv -f /etc/logrotate.d/kaltura_apache on NFS server

and

/opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
/opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
/opt/kaltura/dwh/etlsource/execute/etl_daily.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
/opt/kaltura/dwh/etlsource/execute/etl_perform_retention_policy.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh
/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log

on DWH server.

Hi @Mehul,

Under what user are you running the job?

Hi Jess,

We are running both the logrotate script(on NFS Server) and all other DWH Scripts(on DWH Server) under kaltura user.

Hi,

Does it work when you run it directly from shell as the kaltura user? if not, does it work when running as root? if so, then it is a permissions issue, make sure that:
/opt/kaltura/dwh/logs
/opt/kaltura/dwh/pentaho-plugins
/opt/kaltura/dwh/cycles
/opt/kaltura/pentaho/pdi
are owned by the kaltura user

Hello Jess,

Please excuse me for last reply,I am not running it as kaltura user directly.I am running the scripts directly from the shell as root and its working fine.

Yes,all the below directories are owned by kaltura user but the group is root.

How can I configure the dwh scripts to run as kaltura user.