Hello
We recently found ourselves locked out of the management and admin consoles even though we know the passwords we had set. After going through several articles proffering solutions to the problem inclduing running this commad…
/opt/kaltura/bin/kaltura-config-all.sh
We have now found ourselves in this situation where the error returned is the following:
Starting monit: Starting Monit 5.19.0 daemon with http interface at [*]:2812
[ OK ]
Running Batch config...
kaltura-batch-12.5.0-1.noarch
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Install Process
Loading mirror speeds from cached hostfile
* base: ftp.is.co.za
* epel: mirror.wbs.co.za
* extras: ftp.is.co.za
* remi: fr2.rpmfind.net
* remi-php56: fr2.rpmfind.net
* remi-safe: fr2.rpmfind.net
* updates: ftp.is.co.za
Package php-pecl-zendopcache-7.0.5-3.el6.remi.5.4.x86_64 is obsoleted by php-opcache-5.6.28-1.el6.remi.x86_64 which is already installed
Nothing to do
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
# rm /opt/kaltura/app/base-config.lock
# /opt/kaltura/bin/kaltura-base-config.sh
ERROR 1146 (42S02) at line 1: Table 'kaltura.partner' doesn't exist
kaltura-batch-config.sh FAILED with: 1 on line 66
Archving logs to /opt/kaltura/log/log_20_03_17_14_44.tar.gz...
ERROR: /opt/kaltura/bin/kaltura-batch-config.sh failed:( You can re-run it when the issue is fixed.
Can this table be recreated? What happens to the associated video entries?
The partner table is of course critical as all partner info is stored in it.
If you made periodic backups for the DB, you can of course restore it from there, otherwise, manually creating all needed records for it will be somewhat difficult. It does not only store your partners but also internally used partners:
mysql> select partner_name,id from partner where id<100;
±---------------------±—+
| partner_name | id |
±---------------------±—+
| Play-Server | -6 |
| Media | -5 |
| Monitor | -4 |
| Hosted Pages | -3 |
| Admin Console | -2 |
| Batch | -1 |
| Global | 0 |
| Template KMC account | 99 |
±---------------------±—+
I suppose you could install a fresh instance somewhere else, then export the partner table with all the internal partners, import it into the current ENV, then create your own original partners [from Admin Console or making the partner->register() API call] and then manually update the partner DB records, setting the ID column to the original IDs as they are referenced in the other tables [entry.partner_id to name one].
Also, you need to change the auto_increment counter after manually manipulating the ID column value so, say your highest partner ID is now 103:
mysql> ALTER TABLE tbl AUTO_INCREMENT = 104;
However, the fact the table just disappeared is very disconcerting and there may be additional corrupted or deleted tables.
The safer alternative would be to drop the entire DB, recreate it and then re-digest the source video entries.
Note however, that doing so will break any current player embed codes since the entry IDs in the newly populated DB will be different.
Thanks for the response @jess. I have just discovered that the problem is much worse than I earlier thought. Apparently all the tables in the kaltura db had disappeared leaving just kuser and one other! I have no idea what happened. I suddenly realised I could no longer log in to the KMC or the Admin console and that’s when the rabbit hole began. I’m in the process of backing up my content data which (unfortunately) also resides on the same machine. So if i succeed in setting up a new instance, any chance of mapping the videos to the original partner?
With a completely corrupted DB such as yours, I’m afraid you’ll have to re-ingest the videos… reconstructing all needed records in multiple tables based on the file names under /opt/kaltura/web/content/entry/data and the files’ metadata would be terribly difficult and far more consuming and error prone than just re-ingesting the source videos.
I do recommend that you set periodic DB backups from now on [you can call /opt/kaltura/bin/kaltura-export-db.sh from crontab] or else, setup a MySQL slave configuration.
@jess Hoping you can help me further. I decided to nuke the install and start afresh. However I am encountering this error which has has me stumped…
kaltura-front-12.12.0-1.noarch
Please select one of the following options [0]:
0. All web interfaces
1. Kaltura Management Console [KMC], Hosted Apps, HTML5 lib and ClipApp
2. KAC - Kaltura Admin Console
0
Enabling Apache config - apps.conf
Enabling Apache config - var.conf
Enabling Apache config - admin.conf
========================================================================================================================
Kaltura install answer file written to /tmp/kaltura_21_03_01_20.ans - Please save it!
This answers file can be used to silently-install re-install this machine or deploy other hosts in your cluster.
========================================================================================================================
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
Stopping memcached: [ OK ]
Starting memcached: [ OK ]
Shutting down monit: [FAILED]
Starting monit: Starting Monit 5.19.0 daemon with http interface at [*]:2812
[ OK ]
Failed to set locale, defaulting to C
Running Sphinx config...
kaltura-sphinx-2.2.1-20.x86_64
base-config completed successfully, if you ever want to re-configure your system (e.g. change DB hostname) run the following script:
# rm /opt/kaltura/app/base-config.lock
# /opt/kaltura/bin/kaltura-base-config.sh
Starting monit: Starting Monit 5.19.0 daemon with http interface at [*]:2812
[ OK ]
Configuring your Kaltura DB...
Checking MySQL version..
Ver 5.5.54 found compatible
CREATE USER kaltura;
CREATE USER etl;
CREATE DATABASE kaltura;
CREATE DATABASE kaltura_sphinx_log;
CREATE DATABASE kalturadw;
CREATE DATABASE kalturadw_ds;
CREATE DATABASE kalturadw_bisources;
CREATE DATABASE kalturalog;
Checking connectivity to needed daemons...
Connectivity test passed:)
ERROR: Couldn't make an API request to http://kaltura.yookos.com/api_v3/index.php?service=system&action=ping.
Please check your setup and then run /opt/kaltura/bin/kaltura-db-config.sh again.
Seems to me that some service could not start or not properly configured. Apache perhaps?
According to your output, Apache seems to be running.
Is kaltura.yookos.com resolvable from inside the server itself? if not, you must provide a service URL/endpoint that is resolvable. If it is resolvable and:
# curl kaltura.yookos.com
returns correctly, what’s the output for:
# apachectl -t -DDUMP_VHOSTS
?
You fell victim of a breach where hackers stole your data. You most probably had a “warning” table with a ransom message to pay bitcoins to get the DB dump.
Check if your machine is not further compromised.
I also encountered this issue yesterday on two of my machines. They were test servers so I simply reinstalled everything (OS and all). Rule of thumb, don’t use default ports.
@jess, I also encountered an issue using passwords with special characters.
We update the version every other week and these updates often include security fixes so it is very important to try to update as often as possible.
Using alternatives ports cannot hurt but security by obscurity is never enough. Anyone who knows even a little can run port scans quite easily using nmap and many other tools.
As with any system, it’s also important to limit communications to the absolute mandatory. For instance, your MySQL should only be accessible to the Kaltura nodes that need it [API, front, batch], never to 0.0.0.0.
What was the issue when using special chars and which passwd are we talking about? please provide more info and I’ll be happy to look into it.
Hi @jess
I figured out the problem and it was with Mysql strangely enough. I forgot to disable strict mode (found this out after scouring the web for similar issues). That solved it for me. I’m back up and running. I would like to attempt reingesting the videos but the issue I have is that they are user generated. I have stored in some other db links to the videos like these https://xxx.yyy.com/p/101/sp/10100/playManifest/entryId/0_ihq9jjgs/format/url/protocol/http
Based on the filename that I see in /web/content/entry/data/, its obvious that given the entryId i can somehow locate the corresponding file. However I see several files like this…
-rw-rw-r–. 1 kaltura kaltura 7659 Mar 21 06:24 0_ihq9jjgs_0_0bim5vkm_2.conv.log
-rw-rw-r–. 1 kaltura kaltura 2850616 Mar 21 06:24 0_ihq9jjgs_0_0bim5vkm_2.mp4
-rw-rw-r–. 1 kaltura kaltura 8263 Mar 21 06:24 0_ihq9jjgs_0_1tbzr9vz_2.conv.log
-rw-rw-r–. 1 kaltura kaltura 4665660 Mar 21 06:24 0_ihq9jjgs_0_1tbzr9vz_2.mp4
-rw-rw-r–. 1 kaltura kaltura 11943 Mar 21 06:24 0_ihq9jjgs_0_544yeqmu_2.conv.log
-rw-rw-r–. 1 kaltura kaltura 2365142 Mar 21 06:24 0_ihq9jjgs_0_544yeqmu_2.webm
-rw-rw-r–. 1 kaltura kaltura 11080 Mar 21 06:24 0_ihq9jjgs_0_9qu02rn2_2.conv.log
-rw-rw-r–. 1 kaltura kaltura 8146811 Mar 21 06:24 0_ihq9jjgs_0_9qu02rn2_2.mp4
-rw-rw-r–. 1 kaltura kaltura 991950 Mar 21 06:25 0_ihq9jjgs_0_bd09xy9l_2.3gp
-rw-rw-r–. 1 kaltura kaltura 4256 Mar 21 06:25 0_ihq9jjgs_0_bd09xy9l_2.conv.log
-rw-rw-r–. 1 kaltura kaltura 7088 Mar 21 06:23 0_ihq9jjgs_0_c1ffqwgc_2.conv.log
-rw-rw-r–. 1 kaltura kaltura 1926191 Mar 21 06:23 0_ihq9jjgs_0_c1ffqwgc_2.mp4
-rw-rw-r–. 1 kaltura kaltura 13691 Mar 21 06:25 0_ihq9jjgs_0_c630xcbn_2.conv.log
-rw-rw-r–. 1 kaltura kaltura 13107798 Mar 21 06:25 0_ihq9jjgs_0_c630xcbn_2.mp4
-rw-rw-r–. 1 kaltura kaltura 255020 Mar 21 06:23 0_ihq9jjgs_0_i82zanjl_12.jpg
-rw-rw-r–. 1 kaltura kaltura 7087 Mar 21 06:23 0_ihq9jjgs_0_kcxrjjch_2.conv.log
-rw-rw-r–. 1 kaltura kaltura 1290782 Mar 21 06:23 0_ihq9jjgs_0_kcxrjjch_2.mp4
-rw-r-----. 1 apache kaltura 43245471 Mar 21 06:22 0_ihq9jjgs_0_y834b1wj_2.mp4
Its obvious to me that these are transcoded versions of the original. My question is how do I know the original video?
Update: I’m guessing that in this instance the last entry is the orignal video? Given that the owner is the apache user?
The one owned by apache.kaltura is most probably the source entry. The batch daemon, responsible for the flavour conversion runs as the ‘kaltura’ user so the files owned by kaltura.kaltura were placed there by it.
The naming schema is $ENTRYID_$FLAVOURID_$VERSION.$SUFFIX, since the source file is represented as yet another flavour [the first one created, with the ‘source’ set for flavor_asset.tags]. Because you no longer have the table data, it will be very hard to say for sure my for the reasons I explained before, my bet is the source file is:
-rw-r-----. 1 apache kaltura 43245471 Mar 21 06:22 0_ihq9jjgs_0_y834b1wj_2.mp4