Kaltura CE 10.13.0 released - DWH Trouble

jess · July 27, 2015, 2:12pm

Hi Mathieu,

When using aptitude/apt-get, saying asking to ‘install’ something that already exists will upgrade it.
so it basically is the parallel of saying:
# yum update “kaltura”

You then need to reconfigure some packages as stated here:

github.com

kaltura/platform-install-packages/blob/master/doc/install-kaltura-deb-based.md#upgrade-kaltura

# Installing Kaltura on a Single Server (deb)
This guide describes deb installation of an all-in-one Kaltura server and applies to deb based Linux distros.

The processes was tested on Debian 8 and Ubuntu 14.04 but is expected to work on other versions. 

*When installing on Ubuntu Xenial [16.04], please refer to [this howto](install-kaltura-xenial.md) instead.*

If you tried a different deb based distro and failed, please report it.


[Kaltura Inc.](http://corp.kaltura.com) also provides commercial solutions and services including pro-active platform monitoring, applications, SLA, 24/7 support and professional services. If you're looking for a commercially supported video platform  with integrations to commercial encoders, streaming servers, eCDN, DRM and more - Start a [Free Trial of the Kaltura.com Hosted Platform](http://corp.kaltura.com/free-trial) or learn more about [Kaltura' Commercial OnPrem Edition™](http://corp.kaltura.com/Deployment-Options/Kaltura-On-Prem-Edition). Please note that this service in only offered for RHEL based distros. 

#### Table of Contents
[Prerequites](pre-requisites.md)

[Step-by-step Installation](install-kaltura-deb-based.md#step-by-step-installation)

[Unattended Installation](install-kaltura-deb-based.md#unattended-installation)

[Nginx SSL Configuration](nginx-ssl-config.md)

This file has been truncated. show original

Mathieu · July 27, 2015, 2:42pm

Hi Jess,

J face to a problem on update:

Ign http://installrepo.kaltura.org jupiter/main Translation-en

apt-get clean && apt-get update don’t fix this.

Do you think that it’s a good idea to add this one for update ?
http://installrepo.origin.kaltura.org/repo/apt/debian-10.16.0/

Thanks

Mathieu

jess · July 27, 2015, 3:04pm

Hey Mathieu,

Since we’re using CDN, its possible that your edge node was not yet updated. You can change installrepo.kaltura.org to installrepo.origin.kaltura.org for now but please remember to change it back later as we ideally would prefer you use CDN.

No need to add debiab-10.16.0, debian is a symlink to latest stable which is now 10.16.0:)

Mathieu · July 27, 2015, 3:53pm

Hi Jess,

I upgrade with installrepo.origin
Don’t worry, I change it after upgrade.

apt-get dist-upgrade
Les paquets suivants seront mis à jour :
  kaltura-base kaltura-html5-studio kaltura-html5lib kaltura-kmc
Réception de : 1 http://installrepo.origin.kaltura.org/repo/apt/debian/ jupiter/main kaltura-base all 10.16.0-2+b2 [79,6 MB]
Réception de : 2 http://installrepo.origin.kaltura.org/repo/apt/debian/ jupiter/main kaltura-html5-studio all 2.0.0-1 [641 kB]
Réception de : 3 http://installrepo.origin.kaltura.org/repo/apt/debian/ jupiter/main kaltura-html5lib all 2.33-1+b1 [228 MB]
Réception de : 4 http://installrepo.origin.kaltura.org/repo/apt/debian/ jupiter/main kaltura-kmc all 5.39.6-1 [355 MB]

I do : dpkg-reconfigure kaltura-batch and dpkg-reconfigure kaltura-front

root@kaltest:~# dpkg-reconfigure kaltura-batch
Shutting down:[FAILED]
Service Batch not running
 System start/stop links for /etc/init.d/kaltura-batch already exist.
 * Reloading web server apache2                                                                                                                                                                                                               *
 * Starting daemon monitor monit

Don’t need to modify /opt/kaltura/dwh/etlsource/cycles/register_files.ktr (kaltura-dwh was the latest version)

I run my script for cron and I have the same error message on:
etl_hourly-20150727-14.log & log_events_events.log
The function call createFolder is not valid. (script#3) (script#3)

I continue the investigation
grep is my best friend

Mathieu

jess · July 27, 2015, 3:55pm

Hi Mathieu,

grep is indeed a good pal:)
I will also check on my end.
What MySQL ver are you using BTW?

Mathieu · July 27, 2015, 4:03pm

Hi Jess,

I got Mysql version 5.5.44

root@kaltest:~# dpkg -l | grep mysql
ii libdbd-mysql-perl 4.025-1 amd64 Perl5 database interface to the MySQL database
ii libmysqlclient18:amd64 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database client library
ii mysql-client-5.5 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database client binaries
ii mysql-client-core-5.5 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database core client binaries
ii mysql-common 5.5.44-0ubuntu0.14.04.1 all MySQL database common files, e.g. /etc/mysql/my.cnf
ii mysql-server 5.5.44-0ubuntu0.14.04.1 all MySQL database server (metapackage depending on the latest version)
ii mysql-server-5.5 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database server binaries and system database setup
ii mysql-server-core-5.5 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database server binaries
ii php5-mysql 5.5.9+dfsg-1ubuntu4.11 amd64 MySQL module for php5

Thanks for your help
I keep you informed

Mathieu

jess · July 28, 2015, 11:24am

can you please:

Check the value of ExportPath variable in /opt/kaltura/dwh/.kettle/kettle.properties (and check that kaltura user has
permissions on this directory)
Send the output for this query:
SELECT c.process_id, process_name
FROM kalturadw_ds.cycles c LEFT JOIN kalturadw_ds.processes p
ON c.process_id = p.id
GROUP BY c.process_id

Thanks,

Mathieu · July 28, 2015, 12:02pm

Hi Jess,

ExportPath = @TMP_DIR@/
root@kaltest:/opt# grep -r 'TMP_DIR' give me:
kaltura/dwh/etlsource/scripts/ip2location/update_ip2location.sh:TMP_DIR=/tmp/
It’s the only file where TMP_DIR is declared.
I change the owner of /tmp from root to kaltura.

I run the script and I got the same error.

mysql>

 SELECT c.process_id, process_name FROM kalturadw_ds.cycles c LEFT JOIN kalturadw_ds.processes p ON c.process_id = p.id GROUP BY c.process_id LIMIT 0 , 30;
+------------+--------------+
| process_id | process_name |
+------------+--------------+
|          1 | events       |
+------------+--------------+
1 row in set (0.00 sec)

For information, I came back to the version before my update because I got error on permission. KMC was unavailable.

Mathieu

jess · July 28, 2015, 12:27pm

Hi Mathieu,

Please edit the file and change @TMP_DIR@ to ‘/opt/kaltura/tmp/’ and retry.
This token should have been replaced during the config phase of the run of /var/lib/dpkg/info/kaltura-base.postinst around line 351 [depending on the version you have installed].

Mathieu · July 28, 2015, 12:38pm

Hi Jess,

Wonderfull, it works without error.
I keep you inform if all the analytics display well

Thanks for all

Mathieu

jess · July 28, 2015, 1:33pm

Great. Please do. Note that you can play some entry right now [not from KMC though, embed code should do] and then rerun all the scripts to test. In general, stats will be updated daily unless you force it by running manually.

Mathieu · July 29, 2015, 9:05pm

Hi Jess,

I have to more read the documentation about analytics include in player.
I keep you inform

Mathieu

Mathieu · July 30, 2015, 10:10am

Hi Jess,

I read the documentation without success.
I have no error anymore in logs but in Analytics tab:

Content Report >> Top Content: display no graph
Content Report >> Content Drop-off : display no graph
Content Report >> Content Interactions: display no graph
Content Report >> Content Contributions: display graph without problem
Users & Community reports >> Top Contributors: display graph without problem
Users & Community reports >> Geographics Distribution: display no graph
etc …

I don’t know what I forget to do.
My Embeded player seems to be right configured.

All is working fine except these analytics.

Have you got an idea ?

Thanks

Mathieu

Mathieu · July 30, 2015, 10:36am

Hi,

I’m going to try this issue:

I’ll keep you informed

Mathieu

jess · July 30, 2015, 10:49am

Hello Mathieu,

Can you please ensure /opt/kaltura/web/logs/$HOSTNAME-kaltura_apache_access.log-*gz files exist and contain entries similar to:
’//api_v3/index.php?service=stats&action=collect’?

If so, next step would be to ensure these were processed and exist in:
mysql> select * from kalturadw_ds.files where insert_time >=%Y%m%d;

Please look at:
https://github.com/kaltura/platform-install-packages/blob/Jupiter-10.17.0/doc/kaltura-packages-faq.md#analytics-issues

And see if it helps, if you see one of the steps has failed, please let me know which and we can take it from there.

Thanks,

Mathieu · July 30, 2015, 12:16pm

Hi Jess,

The logs exist with right contains.
I just notice a trouble with owner.

root@kaltest:/opt/kaltura/web# ll | grep logs
drwxr-xr-x  2 root     root    4096 juil. 30 14:01 logs/

and

root@kaltest:/opt/kaltura/web/logs# ll
total 1956
drwxr-xr-x 2 root    root       4096 juil. 30 14:01 ./
drwxrwxr-x 8 root    root       4096 juil. 20 12:17 ../
-rw-rw-r-- 1 kaltura www-data 240183 juil. 20 18:11 kaltest-kaltura_apache_access.log-20150720-18.gz
-rw-rw-r-- 1 kaltura www-data 166612 juil. 21 06:56 kaltest-kaltura_apache_access.log-20150721-06.gz
-rw-rw-r-- 1 kaltura www-data 336765 juil. 22 06:48 kaltest-kaltura_apache_access.log-20150722-06.gz
-rw-r--r-- 1 root    root     395022 juil. 23 06:37 kaltest-kaltura_apache_access.log-20150723-06.gz
-rw-r--r-- 1 root    root     229677 juil. 24 06:28 kaltest-kaltura_apache_access.log-20150724-06.gz
-rw-r--r-- 1 root    root     107840 juil. 25 06:30 kaltest-kaltura_apache_access.log-20150725-06.gz
-rw-r--r-- 1 root    root        523 juil. 26 06:44 kaltest-kaltura_apache_access.log-20150726-06.gz
-rw-r--r-- 1 root    root        545 juil. 27 06:00 kaltest-kaltura_apache_access.log-20150727-06.gz
-rw-r--r-- 1 root    root       2702 juil. 27 12:24 kaltest-kaltura_apache_access.log-20150727-14.gz
-rw-r--r-- 1 root    root      20763 juil. 28 13:48 kaltest-kaltura_apache_access.log-20150728-13.gz
-rw-r--r-- 1 root    root      11909 juil. 28 14:28 kaltest-kaltura_apache_access.log-20150728-14.gz
-rw-r--r-- 1 root    root      48557 juil. 28 15:53 kaltest-kaltura_apache_access.log-20150728-15.gz
-rw-r--r-- 1 root    root     158665 juil. 29 06:50 kaltest-kaltura_apache_access.log-20150729-06.gz
-rw-r--r-- 1 root    root     156947 juil. 29 23:08 kaltest-kaltura_apache_access.log-20150729-23.gz
-rw-r--r-- 1 root    root      82508 juil. 30 06:32 kaltest-kaltura_apache_access.log-20150730-06.gz

For SQL:

mysql> select * from kalturadw_ds.locks;
+---------+------------------+---------------------+------------+
| lock_id | lock_name        | lock_time           | lock_state |
+---------+------------------+---------------------+------------+
|       1 | daily_lock       | 2015-07-29 23:09:42 |          0 |
|       2 | retention_lock   | 2015-07-29 23:09:52 |          0 |
|       3 | hourly_kaltest   | 2015-07-29 23:09:26 |          0 |
|       4 | update_dims_lock | 2015-07-29 23:09:35 |          0 |
+---------+------------------+---------------------+------------+


mysql> select * from kalturadw_ds.files where insert_time >='%Y%m%d';
+---------+-----------------------------------------------+-------------+-------------+---------------------+----------+---------------+-------+-----------+--------------+------------+----------+--------------------+--------+
| file_id | file_name                                     | file_status | prev_status | insert_time         | run_time | transfer_time | lines | err_lines | file_size_kb | process_id | cycle_id | compression_suffix | subdir |
+---------+-----------------------------------------------+-------------+-------------+---------------------+----------+---------------+-------+-----------+--------------+------------+----------+--------------------+--------+
|       1 | kaltest-kaltura_apache_access.log-20150720-18 | IN_CYCLE    | NULL        | 2015-07-27 14:50:24 | NULL     | NULL          |  NULL |      NULL |          234 |          1 |        1 | gz                 |        |
|       2 | kaltest-kaltura_apache_access.log-20150721-06 | IN_CYCLE    | NULL        | 2015-07-27 14:50:24 | NULL     | NULL          |  NULL |      NULL |          162 |          1 |        1 | gz                 |        |
|       3 | kaltest-kaltura_apache_access.log-20150722-06 | IN_CYCLE    | NULL        | 2015-07-27 14:50:24 | NULL     | NULL          |  NULL |      NULL |          328 |          1 |        1 | gz                 |        |
|       4 | kaltest-kaltura_apache_access.log-20150723-06 | IN_CYCLE    | NULL        | 2015-07-27 14:50:24 | NULL     | NULL          |  NULL |      NULL |          385 |          1 |        1 | gz                 |        |
|       5 | kaltest-kaltura_apache_access.log-20150724-06 | IN_CYCLE    | NULL        | 2015-07-27 14:50:24 | NULL     | NULL          |  NULL |      NULL |          224 |          1 |        1 | gz                 |        |
|       6 | kaltest-kaltura_apache_access.log-20150725-06 | IN_CYCLE    | NULL        | 2015-07-27 14:50:24 | NULL     | NULL          |  NULL |      NULL |          105 |          1 |        1 | gz                 |        |
|       7 | kaltest-kaltura_apache_access.log-20150726-06 | IN_CYCLE    | NULL        | 2015-07-27 14:50:24 | NULL     | NULL          |  NULL |      NULL |            0 |          1 |        1 | gz                 |        |
|       8 | kaltest-kaltura_apache_access.log-20150727-06 | IN_CYCLE    | NULL        | 2015-07-27 14:50:24 | NULL     | NULL          |  NULL |      NULL |            0 |          1 |        1 | gz                 |        |
|       9 | kaltest-kaltura_apache_access.log-20150727-14 | IN_CYCLE    | NULL        | 2015-07-27 14:50:24 | NULL     | NULL          |  NULL |      NULL |            2 |          1 |        1 | gz                 |        |
|      10 | kaltest-kaltura_apache_access.log-20150728-13 | IN_CYCLE    | NULL        | 2015-07-28 13:48:17 | NULL     | NULL          |  NULL |      NULL |           20 |          1 |        2 | gz                 |        |
|      11 | kaltest-kaltura_apache_access.log-20150728-14 | IN_CYCLE    | NULL        | 2015-07-28 14:28:48 | NULL     | NULL          |  NULL |      NULL |           11 |          1 |        3 | gz                 |        |
|      12 | kaltest-kaltura_apache_access.log-20150728-15 | IN_CYCLE    | NULL        | 2015-07-28 15:53:37 | NULL     | NULL          |  NULL |      NULL |           47 |          1 |        4 | gz                 |        |
|      13 | kaltest-kaltura_apache_access.log-20150729-06 | IN_CYCLE    | NULL        | 2015-07-29 23:09:14 | NULL     | NULL          |  NULL |      NULL |          154 |          1 |        5 | gz                 |        |
|      14 | kaltest-kaltura_apache_access.log-20150729-23 | IN_CYCLE    | NULL        | 2015-07-29 23:09:14 | NULL     | NULL          |  NULL |      NULL |          153 |          1 |        5 | gz                 |        |
+---------+-----------------------------------------------+-------------+-------------+---------------------+----------+---------------+-------+-----------+--------------+------------+----------+--------------------+--------+
14 rows in set, 1 warning (0.01 sec)

mysql> select * from kalturadw.dwh_fact_events where event_date_id >="%Y%m%d";
Empty set, 1 warning (0.00 sec)

I try to change the owner or I follow the analytics issues ? : Drop the current DWH DBs and Reinstall and config DWH.

Thanks

Mathieu

Mathieu · July 30, 2015, 12:21pm

Change the owner of files and directory provide the same thing …no graph

jess · July 30, 2015, 12:29pm

Hi Mathieu,

Ownership is fine, the process only needs read permissions [and exec to enter the dir] and it has these.
Do you see actual collect requests in the access logs?

Mathieu · July 30, 2015, 1:46pm

Hi Jess,

Sorry but I don’t understand your last question ?
What do you mean exactly ?

Thanks

jess · July 30, 2015, 1:58pm

Hi,

I mean, open the files under /opt/kaltura/web/logs/kaltest-kaltura_apache_access.log-* and make sure you actually see calls for ‘//api_v3/index.php?service=stats&action=collect’ if you do not, then no analytics calls were made by the player/…