Installation broken after upgrade from 15.6 to 16.4 with non-ssl

t-saito · January 4, 2021, 11:36am

I think the Kaltura system consists of the following server features:

front node (apache, kaltura-front, etc.)
sphinx node (kaltrua-sphinx, kaltura-populate, kaltura-elasticsearch, etc.)
streaming server (kaltura-nginx, etc.)
bath server (kaltura-batch, etc.)
database server (mysql or mariadb, etc.)
data-warehouse (kaltura-dwh, kaltura-pentaho, etc.)
administrator server (kaltura-front, etc.)

An all-in-one server contains all the elements.
On the other hand, you can use a separate server for each element.

Load of video uploads and transcoding reduces delivery performance.
As a simple cluster, separate the front node from the other elements.
One server has the front node feature, another server has rest features.

If you want to further improve the delivery performance, create two front nodes.
At this point, Kaltura cluster consists of three servers.
However, a proxy server (load balancer, or CDN host) is required when installing multiple front nodes.
In such cases, the Kaltura cluster consists of four servers (proxy server, two front nodes, backend server).

Multiple Sphinx nodes and batch servers can be installed.
And, Multiple streaming servers can be installed (require proxy server).
Therefore, it is possible to build a cluster corresponding to the server resources available to you.

Note that, if you build the Kaltura cluster, “/opt/kaltura/web” directory should be shared between servers.
Therefore, NFS server is also required.

Regards,

Sakurai · January 6, 2021, 6:31am

Hello @t-saito

DWH analytics problem(ERROR 1499(HY000))

I wrote the program to fix the sql file in C language.
I was able to fix it correctly, but I can’t figure out how to clean the DWH before I reconfigure it as described here.
Do you know how to do this?
I’m assuming it cleans the MariaDB database.
What does “clean” mean?

delete table data
drop table and create table
drop database and create database
other

Am I misunderstanding?

t-saito · January 6, 2021, 8:03am

Hello @Sakurai,

The “clean” means “drop database and create database”.
In order to drop databases about DWH, you can use “/opt/kaltura/bin/kaltura-drop-dwh.sh”.

In order to create databases about DWH, you can use “/opt/kaltura/bin/kaltura-dwh-config.sh”.
Before create databases, you should fix SQL files.

Regards,

Sakurai · January 7, 2021, 4:47am

Hello @t-saito

Thank you for all your advice.
I have tried to do the following

Modify the sql file (date)
Execute /opt/kaltura/bin/kaltura-drop-dwh.sh
Execute /opt/kaltura/bin/kaltura-dwh-config.sh again.
As a result, unfortunately, “analytics” did not work correctly.

Image2770×456 23.8 KB

I think the problem is as follows.

An error was shown on the screen when running 2.
[root@C0Z17001L dpisveri]# /opt/kaltura/bin/kaltura-drop-dwh.sh ********
This will remove all data from the DWH [analytics] DBs and cannot be reverted.
Are you sure you want to continue? [N/y]
y
ERROR 1051 (42S02) at line 1: Unknown table ‘dwh_aggr_events_entry_partitions’
ERROR 1051 (42S02) at line 1: Unknown table ‘dwh_dim_countries’
ERROR 1051 (42S02) at line 1: Unknown table ‘dwh_dim_countries_states’
ERROR 1051 (42S02) at line 1: Unknown table ‘dwh_dim_entries_v’
…
ERROR 1305 (42000) at line 1: PROCEDURE kalturadw.transfer_cycle_partition does not exist
ERROR 1305 (42000) at line 1: PROCEDURE kalturadw.unify_incomplete_api_calls does not exist
ERROR 1305 (42000) at line 1: FUNCTION kalturadw.get_error_code does not exist
ERROR 1305 (42000) at line 1: FUNCTION kalturadw.get_ip_country_location does not exist
The mariadb problem that you told me about before
“kaltura-db-config.sh” script created database users, “kaltura@%” and “etl @%”

I am aware of the “Year 2021 analytics program”.
What do you think I need to fix first?

Regards,

t-saito · January 7, 2021, 6:24am

Hello @Sakurai,

I think that the error about step 2 (/opt/kaltura/bin/kaltura-drop-dwh.sh) is harmless.
When you executed the step 3 (/opt/kaltura/bin/kaltura-dwh-config.sh), no error occurred?

Sakurai · January 7, 2021, 6:44am

Hello @t-saito

No particular error was found.

Can you tell me what it is?

One other thing I’m wondering is how to modify the sql file.
PARTITION p_201207 VALUES LESS THAN (20120801) ENGINE = INNODB)
In the case of
PARTITION p_20201231 VALUES LESS THAN (20210101) ENGINE = INNODB)
and the first p_201207 didn’t have the date 31 in it, but the fix adds the date 31.
I don’t think it matters.

t-saito · January 7, 2021, 9:40am

Hello @Sakurai,

When you execute “kaltura-dwh-config.sh”, system prints message, “The Kaltura DWH DB seems to already be installed”, and “DB creation be skipped”.
These message means that the DWH DB was not created.

If you allow Kaltura’s database to be deleted, use “kaltura-drop-db.sh” and “kaltura-db-config.sh”.

Regards,

Sakurai · January 12, 2021, 12:13am

Hello @t-saito

I tried different patterns.
I’ve tried many patterns, but not all of them have worked.
First of all, I ran “kaltura-drop-db.sh” and “kaltura-db-config.sh”.
As a result, I got an error in kaltura-db-config.sh.

I had no choice but to redo kaltura-config-all.sh from the beginning. (I reverted to the previous version with snapshot and ran it after modifying the sql file.)
As a result, “Error - bailing out!” came up again.

I think I fixed the sql file correctly. (I left the original file with the .org appended to the name.)

I’ve tried stopping the backups, losing the day of the year, month, and day of the date as well as the original file, but with the same result.

Am I doing something wrong?

Regards,

t-saito · January 12, 2021, 11:44am

Hello @Sakurai,

After error message “Error - bailing out!” was printed, please see a log file “/opt/kaltura/dwh/setup/installation_log.log”.

And, I have some concerns in your error reports.

There exist the following statement in your fixed SQL file.

(PARTITION p_20201231 VALUES LESS THAN (20210101) ENGINE = INNODB);

I seem that the correct statement is the following:

(PARTITION p_202012 VALUES LESS THAN (20210101) ENGINE = INNODB);

When modifying the “p _…” part in the SQL files, the 6-digit number should be modified as “202012” and the 8-digit number should be modified as “20201231”.

Next, I guess that the current MariaDB package for the CentOS 7.x has a bug about user authentication.
Even if users “kaltura@%” and “etl@%” are registered, the connection from localhost will fail.
Therefore, the “API request” fails.
So I fixed “/opt/kaltura/bin/kaltura-db-config.sh”, in order to create “kaltura@localhost” and “etl@localhost” instead of “kaltura@%” and “etl@%”.
After all configurations were finished, I created users “kaltura@%” and “etl@%”, and gave them required permissions.

Regards,

Sakurai · January 13, 2021, 7:07am

Hello @t-saito

Thank you for everything.
One of the patterns I’ve tried a few times is the following.
(Some of the same things you pointed out.)
1.(PARTITION p_202012 VALUES LESS THAN (20210101) ENGINE = INNODB);
2. “kaltura@%” and “etl@%”

The following is what happens when I run “kaltura-config-all.sh”.

In this case, “/opt/kaltura/dwh/setup/installation_log.log” shows the strange part.

Normally, if I do nothing
…
now executing /opt/kaltura/dwh/ddl//ddw//maintenance/populate_table_partitions.sql
ERROR 1499 (HY000) at line 1: Too many partitions (including subpartitions) were defined
1
Error - bailing out!

The following error occurs.
Am I doing something wrong?

Regards,

t-saito · January 13, 2021, 8:49am

Hello @Sakurai,

At first, “kaltura@%” and “etl@%” have been registered in database.
Were these users registered after execution of “kaltura-config-all.sh”?

Next, I seem that a syntax error was occurred in SQL file.
Did you check the syntax of “dwh_fact_entries_sizes.sql”?

Regards,

Sakurai · January 14, 2021, 4:02am

Hello @t-saito

Thank you for everything.
I had failed to modify the sql file.
I noticed it and modified the sql file again and ran “kaltura-config-all.sh”.
The result was “Error - balling out !” and “/opt/kaltura/dwh/setup/installation_log.log” has no more errors.

Here is the answer to your previous question.
“At first, “kaltura@%” and “etl@%” have been registered in database.
Were these users registered after execution of “kaltura-config-all.sh”?”

No, I just ran “kaltura-config-all.sh” after modifying “kaltura-db-config.sh”.

In the current state, I tried to play the video by accessing the address created in “Standalone Preview”, but the count in “Analytics” was still 0.

Am I making a mistake in the way I created and configured the MariaDB user?

Regards,

t-saito · January 14, 2021, 4:45am

Hello @Sakurai,

When you running “kaltura-config-all.sh” or “kaltura-db-config.sh”, the users “kaltura @%” and “etl @%” should not exist.
These users cause errors about “API request fails” and “bailing out” during configurations.

If video can be played successfully, then your database has been created successfully.
Playback count of video is not updated immediately.
Log-rotation daemon copies log-files of the Apache and nginx to “/opt/kaltura/web/logs” directory.
At the CentOS, the log-rotation daemon runs between 3:00 AM and 4:00 AM.
Then, cron task (etl_hourly.sh) inserts records into the database according to log-files.
Finally, “dwh_plays_views_sync.sh” updates playback count of each video at 10:00 AM.

If the playback count is not updated, please see log-files in “/opt/kaltura/dwh/logs” directory.

Regards,

Sakurai · January 14, 2021, 5:38am

Hello @t-saito

I don’t know why, but I make sure that “kaltura@%” and “elt@%” and “kaltura@localhost” and “etl@localhost” are created immediately after running “kaltura-config-all.sh” as follows.

I have made two modifications before running “kaltura-config-all.sh”.
One, in “kaltura-db-config.sh”, I changed “kaltura@’#’” and “etl@’#’” to “kaltura@localhost” and “etl@localhost”.
Another change is to move “kaltura-es-config.sh” to run after “kaltura-db-config.sh”.
This is because “kaltura-es-config.sh” causes an error if the user kaltura and etl are not present when executing it.

I figured out from your advice that the count does not change immediately.
If you can tell me, is there any way to test and confirm that the count changes immediately?
Also, do you know if there is a way to check who has accessed the site and for how long in Kaltura?

Regards,

t-saito · January 14, 2021, 6:16am

Hello @Sakurai,

The “kaltura-config-all.sh” and “kaltura-db-config.sh” do not delete existing users.
So, I think that the previously created “kaltura@%” and “etl@%” remain in the database.
If all tables of database are created successfully, these users will not cause any troubles in the future.

In order to check the Kaltura DWH, please execute “/opt/kaltura/bin/kaltura-run-dwh.sh”.
Note that, this script moves apache and nginx log-files into “/opt/kaltura/web/log” directory.
So that, after execution of this script, apache and nginx log-files become empty.
And, today’s log-files in “/opt/kaltura/web/logs” are overwritten.

Regards,

Sakurai · January 14, 2021, 7:28am

Hello @t-saito

I’m sorry if I’m rambling.
This is the first time I ran “kaltura-config-all.sh”.( I ran “kaltura-config-all.sh” for the first time, and I reverted to the original snapshot and ran it from the beginning.)
Just before running “kaltura-config-all.sh”, I did the following.

As for “/opt/kaltura/web/logs”, I have confirmed that the file is created.
Apart from that, can’t I check the access immediately in KMC?
I want to see who played the video and for how long in KMC’s “User Analytics”.
Is this not possible?

Regards,

t-saito · January 14, 2021, 7:51am

Hello @Sakurai,

If execution of “/opt/kaltura-run-dwh.sh” is finished successfully, playback counts of each video are updated immediately.
Updates to analytics data will be reflected immediately in the KMC.
If playback counts are not updated, process of the “/opt/kaltura-run-dwh.sh” failed.
So, please check log-files in “/opt/kaltura/dwh/logs” directory.
When an error occurred, there exists a log-file contains “ERROR” or “Error” statement.

Regards,

Sakurai · January 14, 2021, 8:18am

Hello @t-saito

I found that the error is written in two files under “/opt/kaltura/dwh/logs”.

They are “etl_hourly-20210114-16.log” and “log_events_events.log”.

In “etl_hourly-20210114-16.log”, the following is written as an example.
…
ERROR 14-01 16:18:50,816 - parse bandwidth lines - Unexpected error
ERROR 14-01 16:18:50,816 - parse bandwidth lines - org.pentaho.di.core.exception.KettleValueException:
Javascript error:
Could not apply the given format dd/MMM/yyyy:HH:mm:ss on the string for 14/Jan/2021:12:34:23 : Format.parseObject(String) failed (script#15)

at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.addValues(ScriptValuesMod.java:457)
at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.processRow(ScriptValuesMod.java:688)
at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
at java.lang.Thread.run(Thread.java:748)

…

“log_events_events.log” contains the following as an example.
…
2021/01/14 16:18:50 - parse bandwidth lines.1 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Unexpected error
2021/01/14 16:18:50 - parse bandwidth lines.1 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : org.pentaho.di.core.exception.KettleValueException:
2021/01/14 16:18:50 - parse bandwidth lines.1 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Javascript error:
2021/01/14 16:18:50 - parse bandwidth lines.1 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) : Could not apply the given format dd/MMM/yyyy:HH:mm:ss on the string for 14/Jan/2021:12:34:23 : Format.parseObject(String) failed (script#15)
…

Can you think of anything?

Regards,

t-saito · January 14, 2021, 11:08am

Hello @Sakurai,

Sorry!
I forgot an important issue.
The Kaltura DWH retrieves log statements contains date-time string based on “dd/MMM/yyyy:HH:mm:ss” format.
Under the CentOS and Japanese locale environments, date-time format in log-files is “dd/MMM/yyyy:HH:mm:ssZ”.
Therefore, the Kaltura DWH does not retrieves playback events.
You should fixed some scripts.
Please see the following post.

This post is created based on the Kaltura 14.1.0.
In the current version, the lines to be modified in each file are different.
But, the things to fix are the same.

Regards,

Sakurai · January 15, 2021, 5:39am

Hello @t-saito

Thank you again and again.
This stopped the error I was getting, but I did see another error.

“etl_daily-20210115.log”.
“etl_hourly-20210115-14.log”
“log_aggregation_perform_aggregations.log”
“log_events_events.log”

In the first one, “etl_daily-20210115.log”, I have something like this
…
INFO 15-01 14:17:45,617 - Table input - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
INFO 15-01 14:17:45,618 - Write to log - before calc - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
ERROR 15-01 14:17:45,634 - aggregate plays - Unexpected error
ERROR 15-01 14:17:45,634 - aggregate plays - org.pentaho.di.core.exception.KettleStepException:
Error while running this step!

Couldn’t execute SQL: call kalturadw.calc_aggr_day_play(date(‘20210115’),14,‘plays_entry’)

Table ‘kalturadw.dwh_fact_plays’ doesn’t exist
…

Do you understand what “Table ‘kalturadw.dwh_fact_plays’ doesn’t exist” means?
First of all, I’d like to fix this in the future.

Regards,