Kaltura "Internal database error" while uploading a content or deleting a content

abhimanyu_singh · January 12, 2016, 1:36pm

I am using kaltura version 11.20 on centos 6.7 machine and mysql 5.1.73 . It was running fine since last one month. Today , I ran out of disk issue and I have deleted kaltura_api_v3.log and kaltura_populate.log log files.

Now after this , I am getting internal database error in uploading and deleting the content.

Kaltlog is below

2016-01-11 23:58:59 [0.000121] [1120303733] [11] [BATCH] [KalturaClientBase->doQueue] INFO: execution time for [https://kaltura.babyflix.net/api_v3/service/dropfolder_dropfolder/action/list]: [0.16753101348877]
2016-01-11 23:58:59 [0.000153] [1120303733] [12] [BATCH] [KAsyncDropFolderWatcher->getDropFoldersList] ERR: exception ‘Exception’ with message ‘Cannot get drop folder list - Internal database error’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/plugins/drop_folder/batch/DropFolderWatcher/KAsyncDropFolderWatcher.class.php(108): KalturaLog::err(‘Cannot get drop…’)

thrown in /opt/kaltura/app/batch/client/KalturaClientBase.php on line 362
PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘couldn’t connect to host’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(1131): KalturaClientBase->doQueue()

thrown in /opt/kaltura/app/batch/client/KalturaClientBase.php on line 894
PHP Fatal error: Uncaught exception ‘KalturaException’ with message ‘Internal database error’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:894
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(634): KalturaClientBase->throwExceptionIfError(Array)

thrown in /opt/kaltura/app/batch/client/KalturaClientBase.php on line 362
PHP Fatal error: Uncaught exception ‘KalturaClientException’ with message ‘couldn’t connect to host’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:362
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(1116): KalturaClientBase->doQueue()

==> /opt/kaltura/log/batch/synchronizedistribution-0-2016-01-12.err.log <==
PHP Fatal error: Uncaught exception ‘KalturaException’ with message ‘Internal database error’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:894
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaPlugins/KalturaContentDistributionClientPlugin.php(2804): KalturaClientBase->throwExceptionIfError(Array)

thrown in /opt/kaltura/app/batch/client/KalturaClientBase.php on line 894
PHP Fatal error: Uncaught exception ‘KalturaException’ with message ‘Internal server error occurred’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:894
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(4124): KalturaClientBase->throwExceptionIfError(Array)

2016-01-12 06:21:06 [0.000652] [::1] [75185419] [15] [API] [KalturaStatement->execute] DEBUG: /* kaltura.babyflix.net[75185419][propel] */ INSERT INTO scheduler_status (`ID`,`CREATED_AT`,`UPDATED_AT`,`SCHEDULER_ID`,`SCHEDULER_CONFIGURED_ID`,`TYPE`,`VALUE`) VALUES (NULL,‘2016-01-12 06:21:06’,‘2016-01-12 06:21:06’,‘1’,‘99524’,‘7’,‘1’)
2016-01-12 06:21:06 [0.000380] [::1] [75185419] [16] [API] [BasePeer::doInsert] ERR: exception ‘Exception’ with message ‘SQLSTATE[HY000]: General error: 1364 Field ‘worker_configured_id’ doesn’t have a default value’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/vendor/propel/Propel.php(408): KalturaLog::err(‘SQLSTATE[HY000]…’)

2016-01-12 06:21:06 [0.000470] [::1] [75185419] [17] [API] [KalturaFrontController->getExceptionObject] ALERT: exception ‘PropelException’ with message ‘Unable to execute INSERT statement. [wrapped: SQLSTATE[HY000]: General error: 1364 Field ‘worker_configured_id’ doesn’t have a default value]’ in /opt/kaltura/app/vendor/propel/util/BasePeer.php:299
Stack trace:
#0 /opt/kaltura/app/alpha/lib/model/om/BaseSchedulerStatusPeer.php(766): BasePeer::doInsert(Object(Criteria), Object(KalturaPDO))

==> /opt/kaltura/log/batch/kschedulehelper-0-2016-01-12.err.log <==
PHP Fatal error: Uncaught exception ‘KalturaException’ with message ‘Internal database error’ in /opt/kaltura/app/batch/client/KalturaClientBase.php:894
Stack trace:
#0 /opt/kaltura/app/batch/client/KalturaClient.php(634): KalturaClientBase->throwExceptionIfError(Array)

Kaltura_api_v3.log

When I am trying to upload a content , I am getting following error in kaltura_api_v3.log

2016-01-12 05:32:21 [0.003574] [::1] [2018493301] [8] [API] [KalturaPDO->__construct] DEBUG: conn took - 0.00036001205444336 seconds to mysql:host=127.0.0.1;port=3306;dbname=kaltura;
2016-01-12 05:32:21 [0.000381] [::1] [2018493301] [9] [API] [Propel::initConnection] NOTICE: total conn took 0.00080704689025879 mysql:host=127.0.0.1;port=3306;dbname=kaltura;
2016-01-12 05:32:21 [0.000133] [::1] [2018493301] [10] [API] [DbManager::connectFallbackLogic] DEBUG: connected to propel3
2016-01-12 05:32:21 [0.002660] [::1] [2018493301] [11] [API] [KalturaStatement->execute] DEBUG: /* kaltura.babyflix.net[2018493301][propel3] */ SELECT scheduler.ID, scheduler.CREATED_AT, scheduler.CREATED_BY, scheduler.UPDATED_AT, scheduler.UPDATED_BY, scheduler.CONFIGURED_ID, scheduler.NAME, scheduler.DESCRIPTION, scheduler.STATUSES, scheduler.LAST_STATUS, scheduler.HOST FROM scheduler WHERE scheduler.CONFIGURED_ID=‘99524’ LIMIT 1
2016-01-12 05:32:21 [0.000497] [::1] [2018493301] [12] [API] [KalturaStatement->execute] DEBUG: Sql took - 0.00037002563476562 seconds
2016-01-12 05:32:21 [0.005051] [::1] [2018493301] [13] [API] [KalturaPDO->__construct] DEBUG: conn took - 0.00023317337036133 seconds to mysql:host=127.0.0.1;port=3306;dbname=kaltura;
2016-01-12 05:32:21 [0.000145] [::1] [2018493301] [14] [API] [Propel::initConnection] NOTICE: total conn took 0.00043892860412598 mysql:host=127.0.0.1;port=3306;dbname=kaltura;
2016-01-12 05:32:21 [0.000586] [::1] [2018493301] [15] [API] [KalturaStatement->execute] DEBUG: / kaltura.babyflix.net[2018493301][propel] / INSERT INTO scheduler_status (ID,CREATED_AT,UPDATED_AT,SCHEDULER_ID,SCHEDULER_CONFIGURED_ID,TYPE,VALUE) VALUES (NULL,‘2016-01-12 05:32:21’,‘2016-01-12 05:32:21’,‘1’,‘99524’,‘7’,‘1’)
2016-01-12 05:32:21 [0.000350] [::1] [2018493301] [16] [API] [BasePeer::doInsert] ERR: exception ‘Exception’ with message ‘SQLSTATE[HY000]: General error: 1364 Field ‘worker_configured_id’ doesn’t have a default value’ in /opt/kaltura/app/infra/log/KalturaLog.php:83
Stack trace:
#0 /opt/kaltura/app/vendor/propel/Propel.php(408): KalturaLog::err(‘SQLSTATE[HY000]…’)

I have gone through the table scheduler_config and it has values of worker_configured_id for all the records when the kaltura was working properly

oferc1 · January 12, 2016, 4:49pm

Maybe you DB is damaged, because of no disk space?
Have you tried to check your DB? http://dev.mysql.com/doc/refman/5.7/en/mysqlcheck.html

abhimanyu_singh · January 12, 2016, 4:53pm

Yes I am able to logged in into the databases using kaltura and etl user and it seems fine to me

abhimanyu_singh · January 12, 2016, 9:26pm

I have done the mysqlcheck and it seems fine to me. I am thinking the error is related to the kaltura batch process scheduler as worker_configured_id is null error is thrown on console for scheduler_config table in insert statement whenever i try to upload a file. I have also truncate dscheduler, scheduler_config, scheduler_status , scheduler_worker tables ( after taking the backup) and run the kaltura_batch_config.sh as well as restart the kaltura-batch process but it neither generate the table nor the restart thrown any error. both kaltura_batch_config.sh and kaltura-batch service run successfully. But the error remains the same. I have also found that earlier I had no category available in kmc but when I run shpinx reindex it gives me all the category start appearing in KMC as well as there was metadata entry for all the unsuccessfull upload. When I delete these metadata then status appeared to deleted but they were still in kmc. I think it has an issue with database entry of scheduler.