Analytics Fail - Table has no parititon

voneddie · August 7, 2020, 6:30am

Hi
The analytics have never worked.

I have checked the table kalturadw_ds.locks
2 | retention_lock | 2020-08-07 08:20:43 | 1

Then
update locks set lock_state=0;

Then
/opt/kaltura/bin/kaltura-run-dwh.sh

Cheking the log i see

/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Unexpected error
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : org.pentaho.di.core.exception.KettleStepException:
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Error while running this step!
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Couldn’t execute SQL: Call kalturadw.calc_entries_sizes(20200804)
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Table has no partition for value 20200804
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:208)
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at java.lang.Thread.run(Thread.java:748)
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage.log:2020/08/07 08:20:36 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Couldn’t execute SQL: Call kalturadw.calc_entries_sizes(20200804)

Can you help me ?

jess · August 7, 2020, 10:02am

Hello @voneddie,

What MySQL version are you using?

Cheers,

voneddie · August 7, 2020, 10:36am

Hi

mysql Ver 15.1 Distrib 5.5.65-MariaDB, for Linux (x86_64) using readline 5.1

jess · August 9, 2020, 11:30am

Hello,

Please backup/remove the old logs under /opt/kaltura/dwh/logs/*log, then, re-run /opt/kaltura/bin/kaltura-run-dwh.sh, archive all said logs, upload them to a server I can access and send me a link. I’ll take a look.

Cheers,

voneddie · August 10, 2020, 8:24am

etl_update dims
/opt/kaltura/pentaho/pdi/kitchen.sh: line 11: cd: /root: Permission denied
INFO 10-08 10:13:33,751 - Kitchen - Start of run.
INFO 10-08 10:13:33,827 - Update Dimensions - Start of job execution
INFO 10-08 10:13:34,133 - Update Dimensions - Starting entry [Check if daily_lock is seized]
INFO 10-08 10:13:34,141 - Update Dimensions - Starting entry [Seize update dims lock]
INFO 10-08 10:13:34,141 - Seize update dims lock - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/seize_lock_by_name.ktr]
INFO 10-08 10:13:34,198 - seize_lock_by_name - Dispatching started for transformation [seize_lock_by_name]
INFO 10-08 10:13:34,267 - get_update_dims_lock_name - Dispatching started for transformation [get_update_dims_lock_name]
INFO 10-08 10:13:34,306 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:34,311 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,314 - Set lock name = dims_lock - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,316 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,317 - Get lock name - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,318 - Create lock states - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,332 - Register lock - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,336 - Lock is free - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,336 - Get free lock - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,338 - Get Now - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,342 - Seize lock - Finished processing (I=1, O=0, R=1, W=1, U=1, E=0)
INFO 10-08 10:13:34,343 - Update Dimensions - Starting entry [get_last_updated_at]
INFO 10-08 10:13:34,344 - get_last_updated_at - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//dimensions/get_last_updated_at.ktr]
INFO 10-08 10:13:34,355 - get_last_updated_at - Dispatching started for transformation [get_last_updated_at]
INFO 10-08 10:13:34,389 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:34,395 - Table input - Finished processing (I=1, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:34,417 - Write to log -
------------> Linenr 1------------------------------
last_updated_at = 2020/08/10 09:11:47.000

====================
INFO 10-08 10:13:34,420 - Set Last Updated Variables - Setting environment variables…
INFO 10-08 10:13:34,421 - Write to log - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,422 - Set Last Updated Variables - Set variable LastUpdatedAt to value [2020/08/10 09:11:47.000]
INFO 10-08 10:13:34,422 - Set Last Updated Variables - Set variable NewLastUpdatedAt to value [2020/08/10 10:13:34.419]
INFO 10-08 10:13:34,424 - Get now - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,426 - Set Last Updated Variables - Finished after 1 rows.
INFO 10-08 10:13:34,426 - Set Last Updated Variables - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,429 - Update Dimensions - Starting entry [get_operational_replication_synced_at]
INFO 10-08 10:13:34,429 - get_operational_replication_synced_at - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//dimensions/get_operational_replication_synced_at.ktr]
INFO 10-08 10:13:34,476 - get_operational_replication_synced_at - Dispatching started for transformation [get_operational_replication_synced_at]
INFO 10-08 10:13:34,520 - Get Variables - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,523 - Check if replication table exists - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,528 - Write to log -
------------> Linenr 1------------------------------
operational_replication_synced_at = 2020/08/10 10:13:34.527

====================
INFO 10-08 10:13:34,528 - Set SyncedAt Variable - Setting environment variables…
INFO 10-08 10:13:34,528 - Set SyncedAt Variable - Set variable OperationalReplicationSyncedAt to value [2020/08/10 10:13:34.527]
INFO 10-08 10:13:34,531 - If test.heartbeat exists - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,533 - Get now as synced at - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,543 - Write to log - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,544 - Set SyncedAt Variable - Finished after 1 rows.
INFO 10-08 10:13:34,544 - Set SyncedAt Variable - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,546 - Update Dimensions - Starting entry [Get Sequence]
INFO 10-08 10:13:34,546 - Get Sequence - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/dimensions/get_sequences.ktr]
INFO 10-08 10:13:34,553 - get_sequences - Dispatching started for transformation [get_sequences]
INFO 10-08 10:13:34,577 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:34,578 - Table input - Finished processing (I=12, O=0, R=0, W=12, U=0, E=0)
INFO 10-08 10:13:34,584 - Copy rows to result - Finished processing (I=0, O=0, R=12, W=12, U=0, E=0)
INFO 10-08 10:13:34,586 - Update Dimensions - Starting entry [Iterate sequences]
INFO 10-08 10:13:34,587 - Iterate sequences - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/dimensions/iterate_sequences.ktr]
INFO 10-08 10:13:34,600 - iterate_sequences - Dispatching started for transformation [iterate_sequences]
INFO 10-08 10:13:34,627 - Execute Sequence - Dispatching started for transformation [Execute Sequence]
INFO 10-08 10:13:34,664 - Execute Sequence - Dispatching started for transformation [Execute Sequence]
INFO 10-08 10:13:34,678 - Get rows from result - Finished processing (I=0, O=0, R=12, W=12, U=0, E=0)
INFO 10-08 10:13:34,682 - input - Finished processing (I=0, O=0, R=6, W=6, U=0, E=0)
INFO 10-08 10:13:34,683 - input - Finished processing (I=0, O=0, R=6, W=6, U=0, E=0)
INFO 10-08 10:13:34,709 - Refresh BISources Tables - Dispatching started for transformation [Refresh BISources Tables]
INFO 10-08 10:13:34,787 - Update Partners - Dispatching started for transformation [Update Partners]
INFO 10-08 10:13:34,809 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,844 - parse_partner_custom_data - Dispatching started for transformation [parse_partner_custom_data]
INFO 10-08 10:13:34,874 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:34,874 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:34,875 - Table input - Finished processing (I=7, O=0, R=6, W=7, U=0, E=0)
INFO 10-08 10:13:34,953 - Table input - Finished processing (I=12, O=0, R=6, W=12, U=0, E=0)
INFO 10-08 10:13:34,954 - Get Variables - Finished processing (I=0, O=0, R=12, W=12, U=0, E=0)
INFO 10-08 10:13:34,965 - Get Variables - Finished processing (I=0, O=0, R=7, W=7, U=0, E=0)
INFO 10-08 10:13:34,966 - Read Table Names - Finished reading query, closing connection.
INFO 10-08 10:13:34,967 - Calculator - Finished processing (I=0, O=0, R=7, W=7, U=0, E=0)
INFO 10-08 10:13:34,967 - Calculator - Finished processing (I=0, O=0, R=12, W=12, U=0, E=0)
INFO 10-08 10:13:34,968 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:34,986 - create Sqls - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:35,017 - Read Table Names - Finished processing (I=32, O=0, R=0, W=32, U=0, E=0)
INFO 10-08 10:13:35,107 - Read Table Partners - Finished reading query, closing connection.
INFO 10-08 10:13:35,259 - Write to log -
------------> Linenr 1------------------------------
table_name = entry_media_source
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,270 - Write to log -
------------> Linenr 2------------------------------
table_name = entry_media_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,271 - Write to log -
------------> Linenr 3------------------------------
table_name = entry_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,271 - create Sqls - Finished processing (I=0, O=0, R=32, W=32, U=0, E=0)
INFO 10-08 10:13:35,283 - Write to log -
------------> Linenr 4------------------------------
table_name = entry_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,289 - Insert new rows - Finished reading query, closing connection.
INFO 10-08 10:13:35,289 - Insert new rows - Finished processing (I=0, O=0, R=32, W=32, U=0, E=0)
INFO 10-08 10:13:35,291 - Update Locations - Dispatching started for transformation [Update Locations]
INFO 10-08 10:13:35,292 - Write to log -
------------> Linenr 5------------------------------
table_name = partner_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,295 - Write to log -
------------> Linenr 6------------------------------
table_name = partner_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,297 - Write to log -
------------> Linenr 7------------------------------
table_name = ui_conf_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,298 - Write to log -
------------> Linenr 8------------------------------
table_name = ui_conf_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,299 - Write to log -
------------> Linenr 9------------------------------
table_name = user_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,300 - Write to log -
------------> Linenr 10------------------------------
table_name = moderation_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,302 - Write to log -
------------> Linenr 11------------------------------
table_name = widget_security_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,303 - Write to log -
------------> Linenr 12------------------------------
table_name = widget_security_policy
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,304 - Write to log -
------------> Linenr 13------------------------------
table_name = event_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,306 - Write to log -
------------> Linenr 14------------------------------
table_name = control
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,319 - update rows - Finished reading query, closing connection.
INFO 10-08 10:13:35,320 - update rows - Finished processing (I=0, O=0, R=32, W=32, U=0, E=0)
INFO 10-08 10:13:35,321 - Write to log -
------------> Linenr 15------------------------------
table_name = gender
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,321 - Write to log -
------------> Linenr 16------------------------------
table_name = partner_group_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,321 - Write to log -
------------> Linenr 17------------------------------
table_name = file_sync_object_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,321 - Write to log -
------------> Linenr 18------------------------------
table_name = file_sync_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,322 - Write to log -
------------> Linenr 19------------------------------
table_name = ready_behavior
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,322 - Write to log -
------------> Linenr 20------------------------------
table_name = asset_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,322 - Write to log -
------------> Linenr 21------------------------------
table_name = creation_mode
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,322 - Write to log -
------------> Linenr 22------------------------------
table_name = batch_job_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,322 - Write to log -
------------> Linenr 23------------------------------
table_name = batch_job_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,322 - Write to log -
------------> Linenr 24------------------------------
table_name = batch_job_error_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,322 - Write to log -
------------> Linenr 25------------------------------
table_name = fms_app
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,323 - Write to log -
------------> Linenr 26------------------------------
table_name = partner_vertical
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,323 - Write to log -
------------> Linenr 27------------------------------
table_name = partner_class_of_service
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,323 - Write to log -
------------> Linenr 28------------------------------
table_name = batch_job_object_type
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,323 - Write to log -
------------> Linenr 29------------------------------
table_name = batch_job_exec_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,323 - Write to log -
------------> Linenr 30------------------------------
table_name = upload_token_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,323 - Write to log -
------------> Linenr 31------------------------------
table_name = category_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,323 - Write to log -
------------> Linenr 32------------------------------
table_name = category_entry_status
inserted rows = 0
updated rows = 0

====================
INFO 10-08 10:13:35,334 - Write to log - Finished processing (I=0, O=0, R=32, W=32, U=0, E=0)
INFO 10-08 10:13:35,386 - update_entries - Dispatching started for transformation [update_entries]
INFO 10-08 10:13:35,508 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:35,514 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:35,561 - Read Table entry - Finished reading query, closing connection.
INFO 10-08 10:13:35,571 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:35,594 - Update Kusers - Dispatching started for transformation [Update Kusers]
INFO 10-08 10:13:35,715 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:35,729 - Read Table Kusers - Finished reading query, closing connection.
INFO 10-08 10:13:35,768 - update_flavor_asset - Dispatching started for transformation [update_flavor_asset]
INFO 10-08 10:13:35,860 - Read Flavor Asset - Finished reading query, closing connection.
INFO 10-08 10:13:35,875 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:35,894 - Update widget - Dispatching started for transformation [Update widget]
INFO 10-08 10:13:35,988 - update_file_sync - Dispatching started for transformation [update_file_sync]
INFO 10-08 10:13:36,005 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,024 - Attempt to insert to flavor asset - Connected to database [KalturaDW] (commit=1000)
INFO 10-08 10:13:36,051 - Read Table widget - Finished reading query, closing connection.
INFO 10-08 10:13:36,094 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:36,095 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,132 - update_flavor_params_conversion_profile - Dispatching started for transformation [update_flavor_params_conversion_profile]
INFO 10-08 10:13:36,168 - update_media_info - Dispatching started for transformation [update_media_info]
INFO 10-08 10:13:36,288 - Flavor Params Conversions Profile - Finished reading query, closing connection.
INFO 10-08 10:13:36,295 - Read Media Info - Finished reading query, closing connection.
INFO 10-08 10:13:36,297 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,305 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,326 - update_permissions - Dispatching started for transformation [update_permissions]
INFO 10-08 10:13:36,403 - update_flavor_params - Dispatching started for transformation [update_flavor_params]
INFO 10-08 10:13:36,470 - Read Table Permissions - Finished reading query, closing connection.
INFO 10-08 10:13:36,475 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,497 - update_upload_token - Dispatching started for transformation [update_upload_token]
INFO 10-08 10:13:36,537 - Read flavor params - Finished reading query, closing connection.
INFO 10-08 10:13:36,548 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:36,552 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,553 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,571 - Mapping (sub-transformation) - Finished processing (I=0, O=0, R=6, W=0, U=0, E=0)
INFO 10-08 10:13:36,590 - update_flavor_params_output - Dispatching started for transformation [update_flavor_params_output]
INFO 10-08 10:13:36,605 - Attempt to insert to flavor params - Connected to database [KalturaDW] (commit=1000)
INFO 10-08 10:13:36,672 - Read Flavor Params Output - Finished reading query, closing connection.
INFO 10-08 10:13:36,682 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,703 - Update new flavor params - Finished reading query, closing connection.
INFO 10-08 10:13:36,714 - update_category - Dispatching started for transformation [update_category]
INFO 10-08 10:13:36,740 - Read Table category - Finished reading query, closing connection.
INFO 10-08 10:13:36,744 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,758 - update_category_entry - Dispatching started for transformation [update_category_entry]
INFO 10-08 10:13:36,817 - Read Table category_entry - Finished reading query, closing connection.
INFO 10-08 10:13:36,821 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,851 - Update UI Conf - Dispatching started for transformation [Update UI Conf]
INFO 10-08 10:13:36,907 - Read Table UI Conf - Finished reading query, closing connection.
INFO 10-08 10:13:36,913 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:36,948 - Update Convertsion Profile - Dispatching started for transformation [Update Convertsion Profile]
INFO 10-08 10:13:36,977 - Read Conversion Profile - Finished reading query, closing connection.
INFO 10-08 10:13:36,981 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,026 - update_batch_job_sep - Dispatching started for transformation [update_batch_job_sep]
INFO 10-08 10:13:37,146 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:37,155 - input - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,184 - Mapping (sub-transformation) - Finished processing (I=0, O=0, R=6, W=0, U=0, E=0)
INFO 10-08 10:13:37,246 - Update Dimensions - Starting entry [Set new last updated as the operantional synced at]
INFO 10-08 10:13:37,247 - Update Dimensions - Starting entry [set_last_updated_at]
INFO 10-08 10:13:37,248 - set_last_updated_at - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//dimensions/set_last_updated_at.ktr]
INFO 10-08 10:13:37,256 - set_last_updated_at - Dispatching started for transformation [set_last_updated_at]
INFO 10-08 10:13:37,262 - Get Variables - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,262 - Write to log -
------------> Linenr 1------------------------------
new_last_updated_at = 2020/08/10 10:13:34.527
last_updated_parameter_id = 00000000000000000000002

====================
INFO 10-08 10:13:37,268 - Write to log - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,271 - Update new last_updated_at - Finished processing (I=1, O=0, R=1, W=1, U=1, E=0)
INFO 10-08 10:13:37,272 - Update Dimensions - Starting entry [Release update dims lock]
INFO 10-08 10:13:37,273 - Release update dims lock - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/release_lock_by_name.ktr]
INFO 10-08 10:13:37,278 - release_lock_by_name - Dispatching started for transformation [release_lock_by_name]
INFO 10-08 10:13:37,291 - get_update_dims_lock_name - Dispatching started for transformation [get_update_dims_lock_name]
INFO 10-08 10:13:37,300 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:37,302 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,305 - Set lock name = dims_lock - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,306 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,306 - Get lock name - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,323 - Set lock free state - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,324 - Get Now - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:37,327 - Free lock - Finished processing (I=1, O=0, R=1, W=1, U=1, E=0)
INFO 10-08 10:13:37,329 - Update Dimensions - Starting entry [Success 1]
INFO 10-08 10:13:37,330 - Update Dimensions - Finished job entry [Success 1] (result=[true])
INFO 10-08 10:13:37,330 - Update Dimensions - Finished job entry [Release update dims lock] (result=[true])
INFO 10-08 10:13:37,330 - Update Dimensions - Finished job entry [set_last_updated_at] (result=[true])
INFO 10-08 10:13:37,330 - Update Dimensions - Finished job entry [Set new last updated as the operantional synced at] (result=[true])
INFO 10-08 10:13:37,330 - Update Dimensions - Finished job entry [Iterate sequences] (result=[true])
INFO 10-08 10:13:37,330 - Update Dimensions - Finished job entry [Get Sequence] (result=[true])
INFO 10-08 10:13:37,331 - Update Dimensions - Finished job entry [get_operational_replication_synced_at] (result=[true])
INFO 10-08 10:13:37,331 - Update Dimensions - Finished job entry [get_last_updated_at] (result=[true])
INFO 10-08 10:13:37,331 - Update Dimensions - Finished job entry [Seize update dims lock] (result=[true])
INFO 10-08 10:13:37,331 - Update Dimensions - Finished job entry [Check if daily_lock is seized] (result=[true])
INFO 10-08 10:13:37,331 - Update Dimensions - Job execution finished
INFO 10-08 10:13:37,424 - Kitchen - Finished!
INFO 10-08 10:13:37,424 - Kitchen - Start=2020/08/10 10:13:33.753, Stop=2020/08/10 10:13:37.424
INFO 10-08 10:13:37,425 - Kitchen - Processing ended after 3 seconds.

voneddie · August 10, 2020, 8:25am

retention policy

/opt/kaltura/pentaho/pdi/kitchen.sh: line 11: cd: /root: Permission denied
INFO 10-08 10:13:50,812 - Kitchen - Start of run.
INFO 10-08 10:13:50,990 - perform retention policy - Start of job execution
INFO 10-08 10:13:51,474 - perform retention policy - Starting entry [Seize retention lock]
INFO 10-08 10:13:51,475 - Seize retention lock - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/seize_lock_by_name.ktr]
INFO 10-08 10:13:51,543 - seize_lock_by_name - Dispatching started for transformation [seize_lock_by_name]
INFO 10-08 10:13:51,599 - get_retention_lock_name - Dispatching started for transformation [get_retention_lock_name]
INFO 10-08 10:13:51,629 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:51,635 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:51,637 - Set lock name = retention_lock - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:51,642 - Create lock states - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:51,643 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:51,644 - Get lock name - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:51,665 - Register lock - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:51,667 - Get free lock - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:51,668 - Lock is free - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:51,672 - Get Now - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:51,677 - Seize lock - Finished processing (I=1, O=0, R=1, W=1, U=1, E=0)
INFO 10-08 10:13:51,678 - perform retention policy - Starting entry [Move old partitions to archive]
ERROR 10-08 10:13:51,756 - Move old partitions to archive - An error occurred executing this job entry :
Couldn’t execute SQL: CALL move_innodb_to_archive()

Cannot remove all partitions, use DROP TABLE instead

INFO 10-08 10:13:51,757 - perform retention policy - Finished job entry [Move old partitions to archive] (result=[false])
INFO 10-08 10:13:51,757 - perform retention policy - Finished job entry [Seize retention lock] (result=[false])
INFO 10-08 10:13:51,757 - perform retention policy - Job execution finished
INFO 10-08 10:13:51,773 - Kitchen - Finished!
ERROR 10-08 10:13:51,774 - Kitchen - Finished with errors
INFO 10-08 10:13:51,774 - Kitchen - Start=2020/08/10 10:13:50.815, Stop=2020/08/10 10:13:51.774
INFO 10-08 10:13:51,774 - Kitchen - Processing ended after 0 seconds.

voneddie · August 10, 2020, 8:33am

etl daily

/opt/kaltura/pentaho/pdi/kitchen.sh: line 11: cd: /root: Permission denied
INFO 10-08 10:13:43,090 - Kitchen - Start of run.
INFO 10-08 10:13:43,284 - daily - Start of job execution
INFO 10-08 10:13:43,290 - daily - Starting entry [Check if update_dims_lock is seized]
INFO 10-08 10:13:43,686 - daily - Starting entry [Seize daily lock]
INFO 10-08 10:13:43,687 - Seize daily lock - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/seize_lock_by_name.ktr]
INFO 10-08 10:13:43,765 - seize_lock_by_name - Dispatching started for transformation [seize_lock_by_name]
INFO 10-08 10:13:43,854 - get_daily_lock_name - Dispatching started for transformation [get_daily_lock_name]
INFO 10-08 10:13:43,895 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:43,900 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:43,906 - Set lock name = daily_lock - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:43,910 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:43,911 - Get lock name - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:43,913 - Create lock states - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:43,935 - Get free lock - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:43,936 - Lock is free - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:43,939 - Register lock - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:43,940 - Get Now - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:43,949 - Seize lock - Finished processing (I=1, O=0, R=1, W=1, U=1, E=0)
INFO 10-08 10:13:43,954 - daily - Starting entry [Set jobs sequence file name]
INFO 10-08 10:13:43,955 - daily - Starting entry [Execute daily sequence]
INFO 10-08 10:13:43,989 - Sequence executioner - Starting entry [get jobs]
INFO 10-08 10:13:43,991 - get jobs - Loading transformation from XML file [file:///opt/kaltura/dwh/etlsource/execute/get_jobs.ktr]
INFO 10-08 10:13:44,047 - get_jobs - Dispatching started for transformation [get_jobs]
INFO 10-08 10:13:44,078 - get etls - Opening file: /opt/kaltura/dwh/etlsource/execute/daily_etl_execution_sequence.txt
INFO 10-08 10:13:44,083 - get etls - Finished processing (I=5, O=0, R=0, W=5, U=1, E=0)
INFO 10-08 10:13:44,088 - Filter # - Finished processing (I=0, O=0, R=5, W=5, U=0, E=0)
INFO 10-08 10:13:44,111 - Copy rows to result - Finished processing (I=0, O=0, R=5, W=5, U=0, E=0)
INFO 10-08 10:13:44,113 - Sequence executioner - Starting entry [execute kettle-job file]
INFO 10-08 10:13:44,151 - execute kjb file - Starting entry [set kjb and log path into vars]
INFO 10-08 10:13:44,152 - set kjb and log path into vars - Loading transformation from XML file [file:///opt/kaltura/dwh/etlsource/execute/set_file_var.ktr]
INFO 10-08 10:13:44,176 - set_file_var - Dispatching started for transformation [set_file_var]
INFO 10-08 10:13:44,188 - Get rows from result - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:44,224 - jobName from fileName - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:44,488 - Set Variables - Setting environment variables…
INFO 10-08 10:13:44,488 - Set Variables - Set variable fileVar to value [storage_usage/calc_storage_usage.kjb]
INFO 10-08 10:13:44,488 - Set Variables - Set variable jobVar to value [storage_usage_calc_storage_usage]
INFO 10-08 10:13:44,490 - jobName from fileName - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:44,492 - Set Variables - Finished after 1 rows.
INFO 10-08 10:13:44,492 - Set Variables - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:44,493 - execute kjb file - Starting entry [Write To Log - start job]
INFO 10-08 10:13:44,501 - Status - Executing storage_usage/calc_storage_usage.kjb
Etl base path: /opt/kaltura/dwh/etlsource/
Writing to log [/opt/kaltura/dwh/logs/log_storage_usage_calc_storage_usage]

INFO 10-08 10:13:44,502 - execute kjb file - Starting entry [Run kjbVar job]
INFO 10-08 10:13:44,517 - calc_storage_usage - Starting entry [Add Storage Usage Missing Days]
INFO 10-08 10:13:44,517 - Add Storage Usage Missing Days - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/aggregation/add_aggr_missing_days.ktr]
INFO 10-08 10:13:44,544 - add_aggr_missing_days - Dispatching started for transformation [add_aggr_missing_days]
INFO 10-08 10:13:44,583 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:44,605 - calc_storage_usage - Starting entry [Calc storage usage]
INFO 10-08 10:13:44,605 - Calc storage usage - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/storage_usage/calc_storage_usage.ktr]
INFO 10-08 10:13:44,618 - calc_storage_usage - Dispatching started for transformation [calc_storage_usage]
INFO 10-08 10:13:44,636 - Table input - Finished reading query, closing connection.
INFO 10-08 10:13:44,637 - Table input - Finished processing (I=2, O=0, R=0, W=2, U=0, E=0)
ERROR 10-08 10:13:44,902 - Execute SQL script - Unexpected error
ERROR 10-08 10:13:44,904 - Execute SQL script - org.pentaho.di.core.exception.KettleStepException:
Error while running this step!

Couldn’t execute SQL: Call kalturadw.calc_entries_sizes(20200804)

Table has no partition for value 20200804

at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:208)
at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
at java.lang.Thread.run(Thread.java:748)

Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Couldn’t execute SQL: Call kalturadw.calc_entries_sizes(20200804)

Table has no partition for value 20200804

at org.pentaho.di.core.database.Database.execStatement(Database.java:1650)
at org.pentaho.di.core.database.Database.execStatement(Database.java:1595)
at org.pentaho.di.core.database.Database.execStatements(Database.java:1800)
at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:182)
... 2 more

Caused by: java.sql.SQLException: Table has no partition for value 20200804
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2531)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2489)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742)
at org.pentaho.di.core.database.Database.execStatement(Database.java:1618)
… 5 more

INFO 10-08 10:13:44,909 - Execute SQL script - Finished reading query, closing connection.
INFO 10-08 10:13:44,909 - Execute SQL script - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
ERROR 10-08 10:13:44,910 - calc_storage_usage - Errors detected!
ERROR 10-08 10:13:44,923 - calc_storage_usage - Errors detected!
INFO 10-08 10:13:44,923 - calc_storage_usage - calc_storage_usage
INFO 10-08 10:13:44,924 - calc_storage_usage - calc_storage_usage
INFO 10-08 10:13:44,988 - calc_storage_usage - Finished job entry [Calc storage usage] (result=[false])
INFO 10-08 10:13:44,988 - calc_storage_usage - Finished job entry [Add Storage Usage Missing Days] (result=[false])
INFO 10-08 10:13:44,994 - execute kjb file - Starting entry [Write To Log on error]
INFO 10-08 10:13:45,000 - Status - Failed storage_usage/calc_storage_usage.kjb

INFO 10-08 10:13:45,001 - execute kjb file - Starting entry [Abort job 1]
ERROR 10-08 10:13:45,002 - Abort job 1 - Aborting job.
INFO 10-08 10:13:45,003 - execute kjb file - Finished job entry [Abort job 1] (result=[false])
INFO 10-08 10:13:45,003 - execute kjb file - Finished job entry [Write To Log on error] (result=[false])
INFO 10-08 10:13:45,003 - execute kjb file - Finished job entry [Run kjbVar job] (result=[false])
INFO 10-08 10:13:45,003 - execute kjb file - Finished job entry [Write To Log - start job] (result=[false])
INFO 10-08 10:13:45,004 - execute kjb file - Finished job entry [set kjb and log path into vars] (result=[false])
INFO 10-08 10:13:45,008 - Sequence executioner - Finished job entry [execute kettle-job file] (result=[false])
INFO 10-08 10:13:45,009 - Sequence executioner - Finished job entry [get jobs] (result=[false])
INFO 10-08 10:13:45,014 - daily - Starting entry [Release daily lock]
INFO 10-08 10:13:45,015 - Release daily lock - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/release_lock_by_name.ktr]
INFO 10-08 10:13:45,033 - release_lock_by_name - Dispatching started for transformation [release_lock_by_name]
INFO 10-08 10:13:45,059 - get_daily_lock_name - Dispatching started for transformation [get_daily_lock_name]
INFO 10-08 10:13:45,089 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:45,099 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:45,101 - Set lock name = daily_lock - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:45,104 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:45,105 - Get lock name - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:45,106 - Set lock free state - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:45,108 - Get Now - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:45,112 - Free lock - Finished processing (I=1, O=0, R=1, W=1, U=1, E=0)
INFO 10-08 10:13:45,114 - daily - Starting entry [Success 1]
INFO 10-08 10:13:45,115 - daily - Finished job entry [Success 1] (result=[true])
INFO 10-08 10:13:45,115 - daily - Finished job entry [Release daily lock] (result=[true])
INFO 10-08 10:13:45,115 - daily - Finished job entry [Execute daily sequence] (result=[true])
INFO 10-08 10:13:45,115 - daily - Finished job entry [Set jobs sequence file name] (result=[true])
INFO 10-08 10:13:45,116 - daily - Finished job entry [Seize daily lock] (result=[true])
INFO 10-08 10:13:45,116 - daily - Finished job entry [Check if update_dims_lock is seized] (result=[true])
INFO 10-08 10:13:45,116 - daily - Job execution finished
INFO 10-08 10:13:45,122 - Kitchen - Finished!
INFO 10-08 10:13:45,122 - Kitchen - Start=2020/08/10 10:13:43.093, Stop=2020/08/10 10:13:45.122
INFO 10-08 10:13:45,123 - Kitchen - Processing ended after 2 seconds.

voneddie · August 10, 2020, 8:35am

log storage

2020/08/10 10:13:44 - calc_storage_usage - Starting entry [Add Storage Usage Missing Days]
2020/08/10 10:13:44 - Add Storage Usage Missing Days - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/aggregation/add_aggr_missing_days.ktr]
2020/08/10 10:13:44 - add_aggr_missing_days - Dispatching started for transformation [add_aggr_missing_days]
2020/08/10 10:13:44 - Table input.0 - Finished reading query, closing connection.
2020/08/10 10:13:44 - calc_storage_usage - Starting entry [Calc storage usage]
2020/08/10 10:13:44 - Calc storage usage - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/storage_usage/calc_storage_usage.ktr]
2020/08/10 10:13:44 - calc_storage_usage - Dispatching started for transformation [calc_storage_usage]
2020/08/10 10:13:44 - Table input.0 - Finished reading query, closing connection.
2020/08/10 10:13:44 - Table input.0 - Finished processing (I=2, O=0, R=0, W=2, U=0, E=0)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Unexpected error
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : org.pentaho.di.core.exception.KettleStepException:
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Error while running this step!
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Couldn’t execute SQL: Call kalturadw.calc_entries_sizes(20200804)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Table has no partition for value 20200804
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:208)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at java.lang.Thread.run(Thread.java:748)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Couldn’t execute SQL: Call kalturadw.calc_entries_sizes(20200804)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Table has no partition for value 20200804
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.core.database.Database.execStatement(Database.java:1650)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.core.database.Database.execStatement(Database.java:1595)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.core.database.Database.execStatements(Database.java:1800)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:182)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : … 2 more
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Caused by: java.sql.SQLException: Table has no partition for value 20200804
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2531)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2489)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : at org.pentaho.di.core.database.Database.execStatement(Database.java:1618)
2020/08/10 10:13:44 - Execute SQL script.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : … 5 more
2020/08/10 10:13:44 - Execute SQL script.0 - Finished reading query, closing connection.
2020/08/10 10:13:44 - Execute SQL script.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
2020/08/10 10:13:44 - calc_storage_usage - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Errors detected!
2020/08/10 10:13:44 - calc_storage_usage - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Errors detected!
2020/08/10 10:13:44 - calc_storage_usage - calc_storage_usage
2020/08/10 10:13:44 - calc_storage_usage - calc_storage_usage
2020/08/10 10:13:44 - calc_storage_usage - Finished job entry [Calc storage usage] (result=[false])
2020/08/10 10:13:44 - calc_storage_usage - Finished job entry [Add Storage Usage Missing Days] (result=[false])

voneddie · August 10, 2020, 8:39am

etl hourly

/opt/kaltura/pentaho/pdi/kitchen.sh: line 11: cd: /root: Permission denied
INFO 10-08 10:13:23,985 - Kitchen - Start of run.
INFO 10-08 10:13:24,116 - hourly - Start of job execution
INFO 10-08 10:13:24,120 - hourly - Starting entry [Register etl_server]
INFO 10-08 10:13:24,121 - Register etl_server - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/register_etl_server.ktr]
INFO 10-08 10:13:24,182 - register_etl_server - Dispatching started for transformation [register_etl_server]
INFO 10-08 10:13:24,292 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
INFO 10-08 10:13:24,829 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:24,833 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:24,869 - Set hostname command - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:25,081 - Set hostname command - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,284 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,295 - Select hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,297 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,298 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,343 - Copy rows to result - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,345 - Register server - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,349 - hourly - Starting entry [Seize hourly etl server lock]
INFO 10-08 10:13:25,351 - Seize hourly etl server lock - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/seize_lock_by_name.ktr]
INFO 10-08 10:13:25,408 - seize_lock_by_name - Dispatching started for transformation [seize_lock_by_name]
INFO 10-08 10:13:25,482 - get_etl_server_hourly_lock_name - Dispatching started for transformation [get_etl_server_hourly_lock_name]
INFO 10-08 10:13:25,530 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
INFO 10-08 10:13:25,544 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:25,549 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,549 - Set hostname command - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:25,553 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,557 - Set hostname command - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,566 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,568 - Select hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,574 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,575 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,577 - Add hourly lock prefix - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,581 - Create hourly lock name - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,583 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,583 - Get lock name - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,585 - Create lock states - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,587 - Register lock - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,589 - Get free lock - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,590 - Lock is free - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,592 - Get Now - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,595 - Seize lock - Finished processing (I=1, O=0, R=1, W=1, U=1, E=0)
INFO 10-08 10:13:25,596 - hourly - Starting entry [Set jobs sequence file name]
INFO 10-08 10:13:25,597 - hourly - Starting entry [Execute hourly sequence]
INFO 10-08 10:13:25,616 - Sequence executioner - Starting entry [get jobs]
INFO 10-08 10:13:25,617 - get jobs - Loading transformation from XML file [file:///opt/kaltura/dwh/etlsource/execute/get_jobs.ktr]
INFO 10-08 10:13:25,635 - get_jobs - Dispatching started for transformation [get_jobs]
INFO 10-08 10:13:25,647 - get etls - Opening file: /opt/kaltura/dwh/etlsource/execute/hourly_etl_execution_sequence.txt
INFO 10-08 10:13:25,648 - get etls - Finished processing (I=1, O=0, R=0, W=1, U=1, E=0)
INFO 10-08 10:13:25,651 - Filter # - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,654 - Copy rows to result - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,656 - Sequence executioner - Starting entry [execute kettle-job file]
INFO 10-08 10:13:25,681 - execute kjb file - Starting entry [set kjb and log path into vars]
INFO 10-08 10:13:25,682 - set kjb and log path into vars - Loading transformation from XML file [file:///opt/kaltura/dwh/etlsource/execute/set_file_var.ktr]
INFO 10-08 10:13:25,693 - set_file_var - Dispatching started for transformation [set_file_var]
INFO 10-08 10:13:25,700 - Get rows from result - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,700 - jobName from fileName - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:25,712 - jobName from fileName - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,731 - Set Variables - Setting environment variables…
INFO 10-08 10:13:25,732 - Set Variables - Set variable fileVar to value [events/events.kjb]
INFO 10-08 10:13:25,732 - Set Variables - Set variable jobVar to value [events_events]
INFO 10-08 10:13:25,736 - Set Variables - Finished after 1 rows.
INFO 10-08 10:13:25,736 - Set Variables - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,738 - execute kjb file - Starting entry [Write To Log - start job]
INFO 10-08 10:13:25,739 - Status - Executing events/events.kjb
Etl base path: /opt/kaltura/dwh/etlsource/
Writing to log [/opt/kaltura/dwh/logs/log_events_events]

INFO 10-08 10:13:25,740 - execute kjb file - Starting entry [Run kjbVar job]
INFO 10-08 10:13:25,802 - events cycle process - Starting entry [Set Main Variables]
INFO 10-08 10:13:25,802 - events cycle process - Starting entry [generate_cycle]
INFO 10-08 10:13:25,821 - Get files and generate cycle - Starting entry [Validate process exists in DB]
INFO 10-08 10:13:25,822 - Validate process exists in DB - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/cycles/validate_process_in_DB.ktr]
INFO 10-08 10:13:25,833 - Validate process in DB - Dispatching started for transformation [Validate process in DB]
INFO 10-08 10:13:25,851 - Get Variables - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,858 - Set Variables - Setting environment variables…
INFO 10-08 10:13:25,859 - Set Variables - Set variable ProcessName to value [events]
INFO 10-08 10:13:25,859 - Find process in DB - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,861 - Filter rows - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,863 - Set Variables - Finished after 1 rows.
INFO 10-08 10:13:25,863 - Set Variables - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:25,866 - Get files and generate cycle - Starting entry [List files from dir (Dynamic ktr load)]
INFO 10-08 10:13:25,866 - List files from dir (Dynamic ktr load) - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/list_files_from_local_dir.ktr]
INFO 10-08 10:13:25,876 - list_files_from_local_dir - Dispatching started for transformation [list_files_from_local_dir]
INFO 10-08 10:13:25,883 - Get File Names - Finished processing (I=0, O=0, R=0, W=22, U=0, E=0)
INFO 10-08 10:13:25,886 - Copy rows to result - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
INFO 10-08 10:13:25,888 - Get files and generate cycle - Starting entry [FetchMethod = “remote”]
INFO 10-08 10:13:25,890 - Get files and generate cycle - Starting entry [Register files]
INFO 10-08 10:13:25,891 - Register files - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//cycles/register_files.ktr]
INFO 10-08 10:13:25,906 - register_files - Dispatching started for transformation [register_files]
INFO 10-08 10:13:25,943 - Get rows from result - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
INFO 10-08 10:13:25,946 - Prep file attributes (compression_suffix, file_size, sub_dir) - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:25,964 - Get input folder - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
INFO 10-08 10:13:25,983 - Prep file attributes (compression_suffix, file_size, sub_dir) - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
INFO 10-08 10:13:26,011 - Register file - Finished reading query, closing connection.
INFO 10-08 10:13:26,013 - Register file - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
INFO 10-08 10:13:26,018 - Get files and generate cycle - Starting entry [Get registered cycles]
INFO 10-08 10:13:26,018 - Get registered cycles - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//cycles/read_assigned_cycles_by_status.ktr]
INFO 10-08 10:13:26,032 - Read assigned cycles by status - Dispatching started for transformation [Read assigned cycles by status]
INFO 10-08 10:13:26,052 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
INFO 10-08 10:13:26,076 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:26,081 - Set hostname command - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:26,092 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,095 - Set hostname command - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,106 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,109 - Select hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,110 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,111 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,265 - Get cycle id - Finished reading query, closing connection.
INFO 10-08 10:13:26,267 - Get cycle id - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
INFO 10-08 10:13:26,272 - Get files and generate cycle - Starting entry [Get files for registered cycles]
INFO 10-08 10:13:26,272 - Get files for registered cycles - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//cycles/get_files_for_registered_cycles.ktr]
INFO 10-08 10:13:26,294 - Get files and generate cycle - Finished job entry [Get files for registered cycles] (result=[true])
INFO 10-08 10:13:26,295 - Get files and generate cycle - Finished job entry [Get registered cycles] (result=[true])
INFO 10-08 10:13:26,295 - Get files and generate cycle - Finished job entry [Register files] (result=[true])
INFO 10-08 10:13:26,295 - Get files and generate cycle - Finished job entry [FetchMethod = “remote”] (result=[true])
INFO 10-08 10:13:26,295 - Get files and generate cycle - Finished job entry [List files from dir (Dynamic ktr load)] (result=[true])
INFO 10-08 10:13:26,295 - Get files and generate cycle - Finished job entry [Validate process exists in DB] (result=[true])
INFO 10-08 10:13:26,301 - events cycle process - Starting entry [process cycle]
INFO 10-08 10:13:26,314 - Process cycle - Starting entry [Get generated cycles]
INFO 10-08 10:13:26,315 - Get generated cycles - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/cycles/read_assigned_cycles_by_status.ktr]
INFO 10-08 10:13:26,328 - Read assigned cycles by status - Dispatching started for transformation [Read assigned cycles by status]
INFO 10-08 10:13:26,346 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
INFO 10-08 10:13:26,374 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:26,379 - Set hostname command - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:26,393 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,398 - Set hostname command - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,401 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,404 - Select hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,407 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,407 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,553 - Get cycle id - Finished reading query, closing connection.
INFO 10-08 10:13:26,555 - Get cycle id - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
INFO 10-08 10:13:26,560 - Process cycle - Starting entry [Process single cycle]
INFO 10-08 10:13:26,575 - Process cycle - Starting entry [Get processed cycles]
INFO 10-08 10:13:26,575 - Get generated cycles - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/cycles/read_assigned_cycles_by_status.ktr]
INFO 10-08 10:13:26,590 - Read assigned cycles by status - Dispatching started for transformation [Read assigned cycles by status]
INFO 10-08 10:13:26,616 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
INFO 10-08 10:13:26,666 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:26,673 - Set hostname command - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:26,688 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,693 - Set hostname command - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,700 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,703 - Select hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,712 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,713 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,855 - Get cycle id - Finished reading query, closing connection.
INFO 10-08 10:13:26,856 - Get cycle id - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
INFO 10-08 10:13:26,860 - Process cycle - Starting entry [load single cycle]
INFO 10-08 10:13:26,865 - Process cycle - Finished job entry [load single cycle] (result=[true])
INFO 10-08 10:13:26,866 - Process cycle - Finished job entry [Get processed cycles] (result=[true])
INFO 10-08 10:13:26,866 - Process cycle - Finished job entry [Process single cycle] (result=[true])
INFO 10-08 10:13:26,866 - Process cycle - Finished job entry [Get generated cycles] (result=[true])
INFO 10-08 10:13:26,867 - events cycle process - Starting entry [transfer cycle]
INFO 10-08 10:13:26,876 - transfer cycle - Starting entry [Get processed cycle]
INFO 10-08 10:13:26,877 - Get processed cycle - Loading transformation from XML file [file:////opt/kaltura/dwh/etlsource/cycles/read_assigned_cycles_by_status.ktr]
INFO 10-08 10:13:26,883 - Read assigned cycles by status - Dispatching started for transformation [Read assigned cycles by status]
INFO 10-08 10:13:26,896 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
INFO 10-08 10:13:26,920 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:26,923 - Set hostname command - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:26,930 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,931 - Set hostname command - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,934 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,936 - Select hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,938 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:26,938 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,083 - Get cycle id - Finished reading query, closing connection.
INFO 10-08 10:13:27,085 - Get cycle id - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
INFO 10-08 10:13:27,091 - transfer cycle - Starting entry [transfer cycle]
INFO 10-08 10:13:27,092 - transfer cycle - Loading transformation from XML file [file:////opt/kaltura/dwh/etlsource/cycles/transfer_cycle.ktr]
INFO 10-08 10:13:27,111 - transfer cycle - Finished job entry [transfer cycle] (result=[true])
INFO 10-08 10:13:27,111 - transfer cycle - Finished job entry [Get processed cycle] (result=[true])
INFO 10-08 10:13:27,113 - events cycle process - Finished job entry [transfer cycle] (result=[true])
INFO 10-08 10:13:27,113 - events cycle process - Finished job entry [process cycle] (result=[true])
INFO 10-08 10:13:27,113 - events cycle process - Finished job entry [generate_cycle] (result=[true])
INFO 10-08 10:13:27,114 - events cycle process - Finished job entry [Set Main Variables] (result=[true])
INFO 10-08 10:13:27,150 - execute kjb file - Starting entry [Write To Log on success]
INFO 10-08 10:13:27,151 - Status - Success events/events.kjb

INFO 10-08 10:13:27,151 - execute kjb file - Finished job entry [Write To Log on success] (result=[true])
INFO 10-08 10:13:27,152 - execute kjb file - Finished job entry [Run kjbVar job] (result=[true])
INFO 10-08 10:13:27,152 - execute kjb file - Finished job entry [Write To Log - start job] (result=[true])
INFO 10-08 10:13:27,152 - execute kjb file - Finished job entry [set kjb and log path into vars] (result=[true])
INFO 10-08 10:13:27,161 - Sequence executioner - Finished job entry [execute kettle-job file] (result=[true])
INFO 10-08 10:13:27,161 - Sequence executioner - Finished job entry [get jobs] (result=[true])
INFO 10-08 10:13:27,171 - hourly - Starting entry [Release hourly etl server lock ]
INFO 10-08 10:13:27,171 - Release hourly etl server lock - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/release_lock_by_name.ktr]
INFO 10-08 10:13:27,185 - release_lock_by_name - Dispatching started for transformation [release_lock_by_name]
INFO 10-08 10:13:27,208 - get_etl_server_hourly_lock_name - Dispatching started for transformation [get_etl_server_hourly_lock_name]
INFO 10-08 10:13:27,235 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
INFO 10-08 10:13:27,244 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 10-08 10:13:27,248 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,251 - Set hostname command - Optimization level not specified. Using default of 9.
INFO 10-08 10:13:27,259 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,262 - Set hostname command - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,268 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,270 - Select hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,273 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,273 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,277 - Add hourly lock prefix - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,278 - Create hourly lock name - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,280 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,280 - Get lock name - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,281 - Set lock free state - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,283 - Get Now - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 10-08 10:13:27,285 - Free lock - Finished processing (I=1, O=0, R=1, W=1, U=1, E=0)
INFO 10-08 10:13:27,288 - hourly - Starting entry [Success 1]
INFO 10-08 10:13:27,288 - hourly - Finished job entry [Success 1] (result=[true])
INFO 10-08 10:13:27,288 - hourly - Finished job entry [Release hourly etl server lock ] (result=[true])
INFO 10-08 10:13:27,288 - hourly - Finished job entry [Execute hourly sequence] (result=[true])
INFO 10-08 10:13:27,288 - hourly - Finished job entry [Set jobs sequence file name] (result=[true])
INFO 10-08 10:13:27,289 - hourly - Finished job entry [Seize hourly etl server lock] (result=[true])
INFO 10-08 10:13:27,289 - hourly - Finished job entry [Register etl_server] (result=[true])
INFO 10-08 10:13:27,289 - hourly - Job execution finished
INFO 10-08 10:13:27,289 - Kitchen - Finished!
INFO 10-08 10:13:27,289 - Kitchen - Start=2020/08/10 10:13:23.987, Stop=2020/08/10 10:13:27.289
INFO 10-08 10:13:27,289 - Kitchen - Processing ended after 3 seconds.

voneddie · August 10, 2020, 8:41am

events

2020/08/10 10:13:25 - events cycle process - Starting entry [Set Main Variables]
2020/08/10 10:13:25 - events cycle process - Starting entry [generate_cycle]
2020/08/10 10:13:25 - Get files and generate cycle - Starting entry [Validate process exists in DB]
2020/08/10 10:13:25 - Validate process exists in DB - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/cycles/validate_process_in_DB.ktr]
2020/08/10 10:13:25 - Validate process in DB - Dispatching started for transformation [Validate process in DB]
2020/08/10 10:13:25 - Get Variables.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:25 - Set Variables.0 - Setting environment variables…
2020/08/10 10:13:25 - Set Variables.0 - Set variable ProcessName to value [events]
2020/08/10 10:13:25 - Find process in DB.0 - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:25 - Filter rows.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:25 - Set Variables.0 - Finished after 1 rows.
2020/08/10 10:13:25 - Set Variables.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:25 - Get files and generate cycle - Starting entry [List files from dir (Dynamic ktr load)]
2020/08/10 10:13:25 - List files from dir (Dynamic ktr load) - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/list_files_from_local_dir.ktr]
2020/08/10 10:13:25 - list_files_from_local_dir - Dispatching started for transformation [list_files_from_local_dir]
2020/08/10 10:13:25 - Get File Names.0 - Finished processing (I=0, O=0, R=0, W=22, U=0, E=0)
2020/08/10 10:13:25 - Copy rows to result.0 - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
2020/08/10 10:13:25 - Get files and generate cycle - Starting entry [FetchMethod = “remote”]
2020/08/10 10:13:25 - Get files and generate cycle - Starting entry [Register files]
2020/08/10 10:13:25 - Register files - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//cycles/register_files.ktr]
2020/08/10 10:13:25 - register_files - Dispatching started for transformation [register_files]
2020/08/10 10:13:25 - Get rows from result.0 - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
2020/08/10 10:13:25 - Prep file attributes (compression_suffix, file_size, sub_dir).0 - Optimization level not specified. Using default of 9.
2020/08/10 10:13:25 - Get input folder.0 - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
2020/08/10 10:13:25 - Prep file attributes (compression_suffix, file_size, sub_dir).0 - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
2020/08/10 10:13:26 - Register file.0 - Finished reading query, closing connection.
2020/08/10 10:13:26 - Register file.0 - Finished processing (I=0, O=0, R=22, W=22, U=0, E=0)
2020/08/10 10:13:26 - Get files and generate cycle - Starting entry [Get registered cycles]
2020/08/10 10:13:26 - Get registered cycles - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//cycles/read_assigned_cycles_by_status.ktr]
2020/08/10 10:13:26 - Read assigned cycles by status - Dispatching started for transformation [Read assigned cycles by status]
2020/08/10 10:13:26 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
2020/08/10 10:13:26 - Generate Row.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
2020/08/10 10:13:26 - Set hostname command.0 - Optimization level not specified. Using default of 9.
2020/08/10 10:13:26 - Mapping input specification.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Set hostname command.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Select hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Mapping output specification.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get cycle id.0 - Finished reading query, closing connection.
2020/08/10 10:13:26 - Get cycle id.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
2020/08/10 10:13:26 - Get files and generate cycle - Starting entry [Get files for registered cycles]
2020/08/10 10:13:26 - Get files for registered cycles - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//cycles/get_files_for_registered_cycles.ktr]
2020/08/10 10:13:26 - Get files and generate cycle - Finished job entry [Get files for registered cycles] (result=[true])
2020/08/10 10:13:26 - Get files and generate cycle - Finished job entry [Get registered cycles] (result=[true])
2020/08/10 10:13:26 - Get files and generate cycle - Finished job entry [Register files] (result=[true])
2020/08/10 10:13:26 - Get files and generate cycle - Finished job entry [FetchMethod = “remote”] (result=[true])
2020/08/10 10:13:26 - Get files and generate cycle - Finished job entry [List files from dir (Dynamic ktr load)] (result=[true])
2020/08/10 10:13:26 - Get files and generate cycle - Finished job entry [Validate process exists in DB] (result=[true])
2020/08/10 10:13:26 - events cycle process - Starting entry [process cycle]
2020/08/10 10:13:26 - Process cycle - Starting entry [Get generated cycles]
2020/08/10 10:13:26 - Get generated cycles - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/cycles/read_assigned_cycles_by_status.ktr]
2020/08/10 10:13:26 - Read assigned cycles by status - Dispatching started for transformation [Read assigned cycles by status]
2020/08/10 10:13:26 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
2020/08/10 10:13:26 - Generate Row.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
2020/08/10 10:13:26 - Set hostname command.0 - Optimization level not specified. Using default of 9.
2020/08/10 10:13:26 - Mapping input specification.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Set hostname command.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Select hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Mapping output specification.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get cycle id.0 - Finished reading query, closing connection.
2020/08/10 10:13:26 - Get cycle id.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
2020/08/10 10:13:26 - Process cycle - Starting entry [Process single cycle]
2020/08/10 10:13:26 - Process cycle - Starting entry [Get processed cycles]
2020/08/10 10:13:26 - Get generated cycles - Loading transformation from XML file [/opt/kaltura/dwh/etlsource/cycles/read_assigned_cycles_by_status.ktr]
2020/08/10 10:13:26 - Read assigned cycles by status - Dispatching started for transformation [Read assigned cycles by status]
2020/08/10 10:13:26 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
2020/08/10 10:13:26 - Generate Row.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
2020/08/10 10:13:26 - Set hostname command.0 - Optimization level not specified. Using default of 9.
2020/08/10 10:13:26 - Mapping input specification.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Set hostname command.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Select hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Mapping output specification.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get cycle id.0 - Finished reading query, closing connection.
2020/08/10 10:13:26 - Get cycle id.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
2020/08/10 10:13:26 - Process cycle - Starting entry [load single cycle]
2020/08/10 10:13:26 - Process cycle - Finished job entry [load single cycle] (result=[true])
2020/08/10 10:13:26 - Process cycle - Finished job entry [Get processed cycles] (result=[true])
2020/08/10 10:13:26 - Process cycle - Finished job entry [Process single cycle] (result=[true])
2020/08/10 10:13:26 - Process cycle - Finished job entry [Get generated cycles] (result=[true])
2020/08/10 10:13:26 - events cycle process - Starting entry [transfer cycle]
2020/08/10 10:13:26 - transfer cycle - Starting entry [Get processed cycle]
2020/08/10 10:13:26 - Get processed cycle - Loading transformation from XML file [file:////opt/kaltura/dwh/etlsource/cycles/read_assigned_cycles_by_status.ktr]
2020/08/10 10:13:26 - Read assigned cycles by status - Dispatching started for transformation [Read assigned cycles by status]
2020/08/10 10:13:26 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
2020/08/10 10:13:26 - Generate Row.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
2020/08/10 10:13:26 - Set hostname command.0 - Optimization level not specified. Using default of 9.
2020/08/10 10:13:26 - Mapping input specification.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Set hostname command.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Select hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Mapping output specification.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:26 - Get hostname.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2020/08/10 10:13:27 - Get cycle id.0 - Finished reading query, closing connection.
2020/08/10 10:13:27 - Get cycle id.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
2020/08/10 10:13:27 - transfer cycle - Starting entry [transfer cycle]
2020/08/10 10:13:27 - transfer cycle - Loading transformation from XML file [file:////opt/kaltura/dwh/etlsource/cycles/transfer_cycle.ktr]
2020/08/10 10:13:27 - transfer cycle - Finished job entry [transfer cycle] (result=[true])
2020/08/10 10:13:27 - transfer cycle - Finished job entry [Get processed cycle] (result=[true])
2020/08/10 10:13:27 - events cycle process - Finished job entry [transfer cycle] (result=[true])
2020/08/10 10:13:27 - events cycle process - Finished job entry [process cycle] (result=[true])
2020/08/10 10:13:27 - events cycle process - Finished job entry [generate_cycle] (result=[true])
2020/08/10 10:13:27 - events cycle process - Finished job entry [Set Main Variables] (result=[true])

jess · August 10, 2020, 9:16am

Hello @voneddie,

Reviewing your log, I see that several stored procedures failed:

Couldn’t execute SQL: Call kalturadw.calc_entries_sizes()
Couldn’t execute SQL: CALL move_innodb_to_archive()

See DWH Table has no partition Error, kalturadw.calc_entries_sizes

voneddie · August 10, 2020, 12:33pm

Hi
I tried
call kalturadw.add_partitions();

ERROR 1499 (HY000): Too many partitions (including subpartitions) were defined

voneddie · August 12, 2020, 6:38am

Hi

It seems sugested solution is not working for me. Can i do something else?