Analytics Tasks hanging

Hi Community,

we have the problem that analytics cycles don’t finish:

MySQL [kalturadw_ds]> select * from cycles;
±---------±-----------±------------±--------------------±--------------------±--------------±-----------±-------------------+
| cycle_id | status | prev_status | insert_time | run_time | transfer_time | process_id | assigned_server_id |
±---------±-----------±------------±--------------------±--------------------±--------------±-----------±-------------------+
| 1 | GENERATING | REGISTERED | 2019-11-05 10:00:14 | NULL | NULL | 1 | 1 |
| 3 | RUNNING | GENERATED | 2019-11-05 13:48:18 | 2019-11-05 13:50:06 | NULL | 1 | 1 |
±---------±-----------±------------±--------------------±--------------------±--------------±-----------±-------------------+
2 rows in set (0.00 sec)

They are “generating” und “running” but will never complete.

No errors in logs.

Anyone any idea?
Kaltura CE 15.9.0

Hi @esnecca,

What’s the output of:

mysql> select * from kalturadw_ds.locks;

?

MySQL [kalturadw_ds]> select * from kalturadw_ds.locks ;
±--------±--------------------±--------------------±-----------+
| lock_id | lock_name | lock_time | lock_state |
±--------±--------------------±--------------------±-----------+
| 1 | daily_lock | 2019-11-06 15:13:45 | 0 |
| 2 | retention_lock | 2019-11-06 15:13:49 | 0 |
| 3 | update_dims_lock | 2019-11-06 15:13:39 | 0 |
| 4 | hourly_f39f90832ee2 | 2019-11-06 14:38:48 | 0 |
±--------±--------------------±--------------------±-----------+

After kaltura-drop-dwh.sh, kaltura-dwh-config.sh and kaltura-run-dwh.sh
and one day later i see:

MySQL [kalturadw_ds]> select * from cycles;
±---------±--------±------------±--------------------±--------------------±--------------±-----------±-------------------+
| cycle_id | status | prev_status | insert_time | run_time | transfer_time | process_id | assigned_server_id |
±---------±--------±------------±--------------------±--------------------±--------------±-----------±-------------------+
| 1 | RUNNING | GENERATED | 2019-11-06 15:45:26 | 2019-11-06 15:56:38 | NULL | 1 | 1 |
| 2 | RUNNING | GENERATED | 2019-11-07 04:00:05 | 2019-11-07 04:09:34 | NULL | 1 | 1 |
±---------±--------±------------±--------------------±--------------------±--------------±-----------±-------------------+

MySQL [kalturadw_ds]> select * from kalturadw_ds.locks ;
±--------±--------------------±--------------------±-----------+
| lock_id | lock_name | lock_time | lock_state |
±--------±--------------------±--------------------±-----------+
| 1 | daily_lock | 2019-11-07 09:03:18 | 0 |
| 2 | retention_lock | 2019-11-07 09:03:23 | 0 |
| 3 | hourly_f39f90832ee2 | 2019-11-07 04:00:05 | 1 |
| 4 | update_dims_lock | 2019-11-07 09:03:13 | 0 |
±--------±--------------------±--------------------±-----------+

and in the logs:
grep ERROR /opt/kaltura/dwh/logs/*
/opt/kaltura/dwh/logs/etl_hourly-20191107-09.log:ERROR 07-11 09:03:04,025 - Lock is already seized - Row nr 1 causing abort : [f39f90832ee2], [hourly_], [hourly_f39f90832ee2], [1], [0], [null]
/opt/kaltura/dwh/logs/etl_hourly-20191107-09.log:ERROR 07-11 09:03:04,025 - Lock is already seized - Aborting after having seen 1 rows.
/opt/kaltura/dwh/logs/etl_hourly-20191107-09.log:ERROR 07-11 09:03:04,026 - seize_lock_by_name - Errors detected!
/opt/kaltura/dwh/logs/etl_hourly-20191107-09.log:ERROR 07-11 09:03:04,026 - seize_lock_by_name - Errors detected!

Then i tried
update kalturadw_ds.locks set lock_state = 0;
and
kaltura-run-dwh.sh

but nothing happens…

We still have not data in the analytics-view.

after spending a lot of memory to the sql-server (amazon) it works…