Analytics Not Updating

siactive · December 2, 2015, 4:27pm

Hello,

Recently I moved our backend Kaltura storage to a Linux NFS share (over from Windows, couldn’t resolve a few permissions problems) And everything seems to be in a good working state except for my analytics page, which has not had any player updates since the move.

Following the instructions from the GitHub page,

I have no locked Processes
The apache access logs are listed from today up until the storage change
The last event time I have for data entry insertion was on the day before the switchover, nothing since

I have tried running kaltura-run-dwh.sh from both my DWH and API nodes, but no updates have come through. Any help would be appreciated.

siactive · December 2, 2015, 7:44pm

Also, it looks like my “Top Contributors” Report is still working correctly, although it appears to be the only one.

I was looking in DWH logs, and I’m seeing a few errors in etl_daily, etl_hourly, etl_update_dims, perform_aggregations.

/opt/kaltura/dwh/logs/etl_daily-20151202.log:ERROR 02-12 13:16:01,903 - perform aggregations - org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_daily-20151202.log:ERROR 02-12 13:16:01,905 - Run kjbVar job - An error occurred executing this job entry :
/opt/kaltura/dwh/logs/etl_daily-20151202.log:ERROR 02-12 13:16:01,906 - Run kjbVar job - org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_daily-20151202.log:ERROR 02-12 13:16:02,001 - Abort job 1 - Aborting job.
/opt/kaltura/dwh/logs/etl_hourly-20151202-13.log:ERROR 02-12 13:15:52,571 - process_events - org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_hourly-20151202-13.log:ERROR 02-12 13:15:52,573 - file_processing_job - An error occurred executing this job entry :
/opt/kaltura/dwh/logs/etl_hourly-20151202-13.log:ERROR 02-12 13:15:52,574 - file_processing_job - org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_hourly-20151202-14.log:ERROR 02-12 14:00:05,605 - Register server - Unexpected error
/opt/kaltura/dwh/logs/etl_hourly-20151202-14.log:ERROR 02-12 14:00:05,606 - Register server - org.pentaho.di.core.exception.KettleStepException:
/opt/kaltura/dwh/logs/etl_hourly-20151202-14.log:ERROR 02-12 14:00:05,606 - register_etl_server - Errors detected!
/opt/kaltura/dwh/logs/etl_hourly-20151202-14.log:ERROR 02-12 14:00:05,606 - register_etl_server - Errors detected!
/opt/kaltura/dwh/logs/etl_hourly-20151202-14.log:ERROR 02-12 14:00:05,614 - Kitchen - Finished with errors
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,343 - Mapping (sub-transformation) - Unable to load the mapping transformation because of an error : org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,344 - Mapping (sub-transformation) - org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,344 - Mapping (sub-transformation) - Error initializing step [Mapping (sub-transformation)]
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,354 - Mapping (sub-transformation) - Unable to load the mapping transformation because of an error : org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,355 - Mapping (sub-transformation) - org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,355 - Mapping (sub-transformation) - Error initializing step [Mapping (sub-transformation)]
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,356 - iterate_sequences - Step [Mapping (sub-transformation).0] failed to initialize!
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,356 - iterate_sequences - Step [Mapping (sub-transformation).1] failed to initialize!
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,357 - Iterate sequences - Unable to prepare for execution of the transformation
/opt/kaltura/dwh/logs/etl_update_dims-20151202-13.log:ERROR 02-12 13:15:57,357 - Iterate sequences - org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/etl_update_dims-20151202-14.log:ERROR 02-12 14:00:05,185 - Update Dimensions - A serious error occurred during job execution: org.pentaho.di.core.exception.KettleJobException:
/opt/kaltura/dwh/logs/etl_update_dims-20151202-14.log:ERROR 02-12 14:00:05,185 - Kitchen - Finished with errors
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : org.pentaho.di.core.exception.KettleException:
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Unexpected error during transformation metadata load
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Error reading object from XML file
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Unable to load step info from XML step nodeorg.pentaho.di.core.exception.KettleStepLoaderException:
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Unable to load class for step/plugin with id [MapperFieldRunner]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Unable to load class for step/plugin with id [MapperFieldRunner]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_aggregation_perform_aggregations.log:2015/12/02 13:16:01 - perform aggregations - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :    at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1195)

In addition, I am seeing my apache nodes passing the stats via GET //api_v3/index.php?service=stats&action=collect. Will continue to investigate.

jess · December 3, 2015, 12:29pm

Hi @siactive,

Might be a permissions issue, when you moved files from the original volume, did you do so with -p [preserve permissions]? if it is a permissions issue, it can be easily determined by trying to run the scripts as root and checking whether or not you still get errors. If you still get them, it isn’t about permissions and we need to look further into the lines around the actual error to understand where the issue stems from.

siactive · December 3, 2015, 12:51pm

Hi Jess,

When running the individual scripts inside kaltura-run-dwh.sh as root, I’m still seeing the errors appear in the /dwh/logs folder. Is there a particular one you’d like me to grab more output of?

jess · December 3, 2015, 12:56pm

Hi Nick,

Just:
# rm /opt/kaltura/dwh/logs/*
then run all scripts and:
# grep ERROR /opt/kaltura/dwh/logs/*
the go to each log that has the string in it and check around the final error string to see what the actual problem is.
When you find it, we can solve it:)

siactive · December 3, 2015, 1:19pm

Jess,

Each step seems to be unable to find a plugin, seeing this in etl_daily,update_dims, and log_aggregations

Unexpected error during transformation metadata load

Error reading object from XML file

Unable to load step info from XML step nodeorg.pentaho.di.core.exception.KettleStepLoaderException:
Unable to load class for step/plugin with id [MapperFieldRunner]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.


Unable to load class for step/plugin with id [MapperFieldRunner]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.




at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1195)
at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:611)
at org.pentaho.di.job.Job.execute(Job.java:503)
at org.pentaho.di.job.Job.execute(Job.java:642)
at org.pentaho.di.job.Job.execute(Job.java:420)
at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:63)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.pentaho.di.core.exception.KettleXMLException:
Error reading object from XML file

Unable to load step info from XML step nodeorg.pentaho.di.core.exception.KettleStepLoaderException:
Unable to load class for step/plugin with id [MapperFieldRunner]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.

jess · December 3, 2015, 1:32pm

Not sure what exactly happened on the FS but I’d suggest:
# yum reinstall kaltura-pentaho
# yum reinstall kaltura-dwh
and then:
# kaltrua-dwh-config.sh
again.

Note that none of these steps should drop the existing data. Nonetheless, I suggest you back up the relevant DBs prior to doing so, just in case…

siactive · December 3, 2015, 2:53pm

Jess,

I reinstalled pentaho, dwh, and dwh config script. I am still receiving the same errors inside of /dwh/logs

ERROR 03-12 09:45:57,951 - Mapping (sub-transformation) - Unable to load the mapping transformation because of an error : org.pentaho.di.core.exception.KettleException:
Unable to load the specified mapping

Error reading object from XML file

Unable to load step info from XML step nodeorg.pentaho.di.core.exception.KettleStepLoaderException:
Unable to load class for step/plugin with id [MapperFieldRunner]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.


Unable to load class for step/plugin with id [MapperFieldRunner]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.

Any other ideas?

jess · December 3, 2015, 3:04pm

Hi Nick,

What MySQL version is this?

siactive · December 3, 2015, 3:15pm

Jess it is:
Ver 15.1 Distrib 5.5.44-MariaDB, pulled from the Cent Repo in October

jess · December 3, 2015, 3:26pm

That might account for it then. Make sure you’re not running in strict mode.

siactive · December 3, 2015, 3:34pm

Hi Jess,

Checked the databases using SELECT @@SQL_MODE; no mode has been set, is there a mode that does need to be set?

+------------+-------------------+
| @@SQL_MODE | @@GLOBAL.SQL_MODE |
+------------+-------------------+
|            |                   |
+------------+-------------------+

siactive · December 4, 2015, 5:22pm

SO as a bit of an experiment, I rolled out a brand new VM this morning, installed DWH services and and it too is getting those Java Plugin errors. I’m not sure if they are the cause of my issue, especially since nothing was done to the current DWH server (Aside from mounting the new NFS share). I’m going to roll through all the permissions inside the new NFS share, as that has really been the only environment change I’ve had since the analytics stopped working. Will update if I find anything.

siactive · January 7, 2016, 12:57pm

Hello,

Just wanted to update this thread. After continuing my troubleshooting, I found out the reason I was receiving Java class errors in my DWH logs. I was missing several plugins in the /opt/kalt/pentaho/pdi/plugins/steps/ directory, specifically in my case MappingFieldRunner/ MySQLInserter/ ShapeFileReader3/. After replacing these and running the DWH script from kaltura/bin/, I am not receiving any errors in the logs. I am now waiting to see if the KMC updates.

siactive · January 7, 2016, 2:18pm

So after running /kaltura-run-dwh.sh, no errors entries are in the DWH log files, everything looks good. However, the KMC is still not updating certain stats such as Top Content, really anything to do with play statistics it seems.

So, going off of the Analytics troubleshooting guide again, there are no locks, no errors in DWH logs.

In kalturadw_ds.files, I see the apache access logs from the past month are "IN_CYCLE"
In Kalturadw.dwh_facts_events there are 0 entries, even after playing embedded videos (Outside of the KMC of course)

Looking at the apache_access_logs on my web servers, I am seeing what I believe are the correct GET requests,

$IP- - [07/Jan/2016:09:00:13 -0500] "GET /api_v3/index.php?service=stats&apiVersion=3.1&expiry=86400&clientTag=kwidget%3Av2.37.3&format=1&ignoreNull=1&action=collect&event:eventType=13&event

Anything else I can do to troubleshoot this issue? I feel like I am fairly close to getting analytics working again.

siactive · January 8, 2016, 3:24pm

I’m going to mark this issue as “closed”, while I could not get my original DWH server to work, I built a new VM with MariaDB, installed the DWH roles onto it, and pointed my Front End nodes to it (via answer file) and the analytics are working again.