Kaltura CE 10.13.0 released - DWH Trouble

J faced to a problem with dwh
Ubuntu 14.04 LTS

First :
sh: 0: Can’t open /opt/kaltura/bin/pentaho/kitchen.sh
in log /opt/kaltura/dwh/logs/etl_daily-20150722.log

modify : /opt/kaltura/dwh/etlsource/execute/etl_daily.sh
Replace KITCHEN=$BASE_DIR/bin/pentaho/kitchen.sh
By KITCHEN=$BASE_DIR/pentaho/pdi/kitchen.sh

modify : /opt/kaltura/dwh/etlsource/execute/etl_daily.sh
Replace KITCHEN=$BASE_DIR/bin/pentaho/kitchen.sh
By KITCHEN=$BASE_DIR/pentaho/pdi/kitchen.sh

After this j get an other error :
Check if update_dims_lock is seized - An error occurred executing this job entry:
Error occured while trying to connect to the database
Error connecting to database: (using class org.gjt.mm.mysql.Driver)
Communications link failure

J modify the tcpwrapper /etc/hosts.allow
mysqld : 127.0.0.1

It works fine but with a little error :
desc = org.pentaho.di.core.exception.KettleDatabaseException:
Couldn’t execute SQL: call register_file(‘kaltest-kaltura_apache_access.log-20150722-06’, 1, 328,9, ‘gz’, ‘’)

Incorrect number of arguments for PROCEDURE kalturadw_ds.register_file; expected 5, got 6

fields = null
codes = ExecSQL001

I continue the investigation

Mathieu

Hello,

What should really happen is for /etc/cron.d/kaltura-dwh to pass -k with the correct location to pentaho, I will take care of it and release a new package but you can change it on your local file for now.

About the locks, can you look here?

as for the register_file(‘kaltest-kaltura_apache_access.log-20150722-06’, 1, 328,9, ‘gz’, ‘’) I will check on that and get back to you.

Also, when you say ‘little error’ do you mean you’re getting stats despite the error?

Are you file parsed and does:
mysql> select * from kalturadw_ds.files where insert_time >=%Y%m%d;

return actual results?

Thanks,

Hi Jess,

I’m working with Mathieu, it’s seems that the PROCEDURE register_file receive too much argument.
We have this issue on each cycles when parsing log file to analytic. We try to purge and recreate the dwh databases. Without any correction. Still the same messages in ETL Logs.

Like that for try I add an argument in sql procedure and only one error in ETL LOG this time for expected 6, got 5
And a part of analytic working.

Hello,

Can you try changing /opt/kaltura/dwh/etlsource/cycles/register_files.ktr line 359

file_size_kb = size / 1024;
to:
file_size_kb = floor(size / 1024);

And let me know if that helps?

Thanks,

Hello Jess,

It’s better. We can see more analytics on the KMC.
We have only two same errors on the logs : etl_hourly-20150727-14.log & log_events_events.log

The function call createFolder is not valid. (script#3) (script#3)

The first one on etl_hourly:

ERROR 27-07 14:50:26,665 - Create output_path - Unexpected error
ERROR 27-07 14:50:26,665 - Create output_path - org.pentaho.di.core.exception.KettleValueException:
Javascript error:
org.mozilla.javascript.EvaluatorException: The function call createFolder is not valid. (script#3) (script#3)

        at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.addValues(ScriptValuesMod.java:457)
        at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.processRow(ScriptValuesMod.java:688)
        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.mozilla.javascript.EvaluatorException: org.mozilla.javascript.EvaluatorException: The function call createFolder is not valid. (script#3) (script#3)
        at org.mozilla.javascript.DefaultErrorReporter.runtimeError(DefaultErrorReporter.java:109)
        at org.mozilla.javascript.Context.reportRuntimeError(Context.java:938)
        at org.mozilla.javascript.Context.reportRuntimeError(Context.java:994)
        at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesAddedFunctions.createFolder(ScriptValuesAddedFunctions.java:1816)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:161)
        at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:413)
        at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:97)
        at org.mozilla.javascript.gen.c13._c0(script:3)
        at org.mozilla.javascript.gen.c13.call(script)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:398)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3065)
        at org.mozilla.javascript.gen.c13.call(script)
        at org.mozilla.javascript.gen.c13.exec(script)
        at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.addValues(ScriptValuesMod.java:376)
        ... 3 more

The second one on log_events_events.log:

2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Unexpected error
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : org.pentaho.di.core.exception.KettleValueException:
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Javascript error:
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : org.mozilla.javascript.EvaluatorException: The function call createFolder is not valid. (script#3) (script#3)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.addValues(ScriptValuesMod.java:457)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.processRow(ScriptValuesMod.java:688)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at java.lang.Thread.run(Thread.java:745)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Caused by: org.mozilla.javascript.EvaluatorException: org.mozilla.javascript.EvaluatorException: The function call createFolder is not valid. (script#3) (script#3)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.DefaultErrorReporter.runtimeError(DefaultErrorReporter.java:109)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.Context.reportRuntimeError(Context.java:938)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.Context.reportRuntimeError(Context.java:994)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesAddedFunctions.createFolder(ScriptValuesAddedFunctions.java:1816)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at java.lang.reflect.Method.invoke(Method.java:606)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:161)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:413)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:97)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.gen.c13._c0(script:3)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.gen.c13.call(script)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:398)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3065)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.gen.c13.call(script)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.mozilla.javascript.gen.c13.exec(script)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.addValues(ScriptValuesMod.java:376)
2015/07/27 14:50:26 - Create output_path.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :   ... 3 more
2015/07/27 14:50:26 - Create output_path.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)

Thanks for your help

Mathieu

Hi Mathieu,

Might be a permission issue. How are you running the DWH jobs? from CLI manually or letting it run from crond?
Under what user are they running?

Also, what’s the output of
# dpkg -l kaltura-base
# dpkg -l kaltura-dwh
# dpkg -l kaltura-pentaho

Just to better understand what versions you’re running.

Hi Jess,

J run the DWH Jobs with this liitle script:

#!/bin/bash

rm /opt/kaltura/dwh/logs/*
logrotate -vvv -f /etc/logrotate.d/kaltura_apache
su - kaltura -c "/opt/kaltura/dwh/etlsource/execute/etl_hourly.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh"
su - kaltura -c "/opt/kaltura/dwh/etlsource/execute/etl_update_dims.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh"
su - kaltura -c "/opt/kaltura/dwh/etlsource/execute/etl_daily.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh"
su - kaltura -c "/opt/kaltura/dwh/etlsource/execute/etl_perform_retention_policy.sh -p /opt/kaltura/dwh -k /opt/kaltura/pentaho/pdi/kitchen.sh"
su - kaltura -c "/opt/kaltura/app/alpha/scripts/dwh/dwh_plays_views_sync.sh >> /opt/kaltura/log/cron.log"

Here’s the version installed:

ii kaltura-base 10.15.0-1+b9 all Kaltura Open Source Video Platform
ii kaltura-dwh 10.10.0-1+b14 amd64 Kaltura Open Source Video Platform - Analytics
ii kaltura-pentaho 4.2.1 amd64 Pentaho Open Source Data Integration Community Edition (CE).

Thanks

Hi Mathieu,

Script looks fine.
Can you make sure /opt/kaltura/pentaho/pdi/ is owned by the kaltura user recursively?

Also, there are already newer versions out, can you upgrade? it does not yet include the change to /opt/kaltura/dwh/etlsource/cycles/register_files.ktr though so, if you do upgrade, you will need to manually refix that.

Hi Jess,

Yes pdi is owned recursively by Kaltura

J can try an upgrade but I’m a little bit confused with this command, I’m a newbie with Kaltura :
aptitude install ~Nkaltura
Could you explain me what I can do exactly ? I must install all kaltura package after update the repo or a script exists ?

Thanks

Mathieu

Hi Mathieu,

When using aptitude/apt-get, saying asking to ‘install’ something that already exists will upgrade it.
so it basically is the parallel of saying:
# yum update “kaltura

You then need to reconfigure some packages as stated here:

Hi Jess,

J face to a problem on update:

Ign http://installrepo.kaltura.org jupiter/main Translation-en

apt-get clean && apt-get update don’t fix this.

Do you think that it’s a good idea to add this one for update ?
http://installrepo.origin.kaltura.org/repo/apt/debian-10.16.0/

Thanks

Mathieu

Hey Mathieu,

Since we’re using CDN, its possible that your edge node was not yet updated. You can change installrepo.kaltura.org to installrepo.origin.kaltura.org for now but please remember to change it back later as we ideally would prefer you use CDN.

No need to add debiab-10.16.0, debian is a symlink to latest stable which is now 10.16.0:)

Hi Jess,

I upgrade with installrepo.origin
Don’t worry, I change it after upgrade.

apt-get dist-upgrade
Les paquets suivants seront mis à jour :
  kaltura-base kaltura-html5-studio kaltura-html5lib kaltura-kmc
Réception de : 1 http://installrepo.origin.kaltura.org/repo/apt/debian/ jupiter/main kaltura-base all 10.16.0-2+b2 [79,6 MB]
Réception de : 2 http://installrepo.origin.kaltura.org/repo/apt/debian/ jupiter/main kaltura-html5-studio all 2.0.0-1 [641 kB]
Réception de : 3 http://installrepo.origin.kaltura.org/repo/apt/debian/ jupiter/main kaltura-html5lib all 2.33-1+b1 [228 MB]
Réception de : 4 http://installrepo.origin.kaltura.org/repo/apt/debian/ jupiter/main kaltura-kmc all 5.39.6-1 [355 MB]

I do : dpkg-reconfigure kaltura-batch and dpkg-reconfigure kaltura-front

root@kaltest:~# dpkg-reconfigure kaltura-batch
Shutting down:[FAILED]
Service Batch not running
 System start/stop links for /etc/init.d/kaltura-batch already exist.
 * Reloading web server apache2                                                                                                                                                                                                               *
 * Starting daemon monitor monit

Don’t need to modify /opt/kaltura/dwh/etlsource/cycles/register_files.ktr (kaltura-dwh was the latest version)

I run my script for cron and I have the same error message on:
etl_hourly-20150727-14.log & log_events_events.log
The function call createFolder is not valid. (script#3) (script#3)

I continue the investigation
grep is my best friend :wink:

Mathieu

Hi Mathieu,

grep is indeed a good pal:)
I will also check on my end.
What MySQL ver are you using BTW?

Hi Jess,

I got Mysql version 5.5.44

root@kaltest:~# dpkg -l | grep mysql
ii libdbd-mysql-perl 4.025-1 amd64 Perl5 database interface to the MySQL database
ii libmysqlclient18:amd64 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database client library
ii mysql-client-5.5 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database client binaries
ii mysql-client-core-5.5 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database core client binaries
ii mysql-common 5.5.44-0ubuntu0.14.04.1 all MySQL database common files, e.g. /etc/mysql/my.cnf
ii mysql-server 5.5.44-0ubuntu0.14.04.1 all MySQL database server (metapackage depending on the latest version)
ii mysql-server-5.5 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database server binaries and system database setup
ii mysql-server-core-5.5 5.5.44-0ubuntu0.14.04.1 amd64 MySQL database server binaries
ii php5-mysql 5.5.9+dfsg-1ubuntu4.11 amd64 MySQL module for php5

Thanks for your help
I keep you informed

Mathieu

can you please:

  • Check the value of ExportPath variable in /opt/kaltura/dwh/.kettle/kettle.properties (and check that kaltura user has
    permissions on this directory)
  • Send the output for this query:
    SELECT c.process_id, process_name
    FROM kalturadw_ds.cycles c LEFT JOIN kalturadw_ds.processes p
    ON c.process_id = p.id
    GROUP BY c.process_id

Thanks,

Hi Jess,

  • ExportPath = @TMP_DIR@/
    root@kaltest:/opt# grep -r 'TMP_DIR' give me:
    kaltura/dwh/etlsource/scripts/ip2location/update_ip2location.sh:TMP_DIR=/tmp/
    It’s the only file where TMP_DIR is declared.
    I change the owner of /tmp from root to kaltura.

I run the script and I got the same error.

mysql>

 SELECT c.process_id, process_name FROM kalturadw_ds.cycles c LEFT JOIN kalturadw_ds.processes p ON c.process_id = p.id GROUP BY c.process_id LIMIT 0 , 30;
+------------+--------------+
| process_id | process_name |
+------------+--------------+
|          1 | events       |
+------------+--------------+
1 row in set (0.00 sec)

For information, I came back to the version before my update because I got error on permission. KMC was unavailable.

Mathieu

Hi Mathieu,

Please edit the file and change @TMP_DIR@ to ‘/opt/kaltura/tmp/’ and retry.
This token should have been replaced during the config phase of the run of /var/lib/dpkg/info/kaltura-base.postinst around line 351 [depending on the version you have installed].

Hi Jess,

Wonderfull, it works without error.
I keep you inform if all the analytics display well

Thanks for all

Mathieu

Great. Please do. Note that you can play some entry right now [not from KMC though, embed code should do] and then rerun all the scripts to test. In general, stats will be updated daily unless you force it by running manually.