Errors when kaltura server under load

Hi,

I’m running Kaltura server 11.7 with nfs server (for/opt/kaltura/web), and separate db,sphinx machines
and I’m getting few types of errors when the server is under load, any hint to what should be fixed would be nice:

  1. Memory cache allocation:
    PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 1800631551 bytes) in /opt/kaltura/app/alpha/apps/kaltura/lib/cache/kApiCache.php on line 4,

Or

PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 1800631551 bytes) in /opt/kaltura/web/html5/html5lib/v2.37.4/LocalSettings.php on line 20

  1. File opening:
    [client 127.0.0.1] PHP Fatal error: require_once(): Failed opening required ‘1’ (include_path=’/opt/kaltura/app/vendor/ZendFramework/library:.:/usr/share/pear:/usr/share/php:/opt/kaltura/app:/opt/kaltura/app/vendor/symfony:/opt/kaltura/app/vendor/symfony/vendor:/opt/kaltura/app/vendor/ZendFramework/library:/opt/kaltura/app/alpha:/opt/kaltura/app/alpha/lib:/opt/kaltura/app/alpha/apps/kaltura/lib’) in /opt/kaltura/app/vendor/ZendFramework/library/Zend/Log/Formatter/Simple.php on line 24

and all kind of strange errors like that (for files that doesn’t exist)

Ofer

Hi Ofer,

The error is pretty obvious: PHP utilised all the memory it is allowed to allocate… how much RAM does the server have and what is the value of memory_limit?

Also, make sure that the cache dirs have the correct permissions, allowing the apache user to write to them.

The issue with the missing ZF files in admin console is known and should be fixed but it is not what’s causing the problem you mentioned.

Hi Jess,
The machine has ~ 8GB memory
memory_limit is 256 MB.
What is your recommendation?

About the error with missing files, it happens only after some time of proper working, so I guessed it is also some kind of resource exhausted, and I though maybe you have an idea.

256M for a very loaded server can be exhausted. Seeing how you have 8G, I think you can increase it to 512M.
Regardless however, check what I wrote about cache dir permissions and also, try to understand what sort of requests cause the issue. From the error string above, it seems it is related to HTML5lib only? I really do think it is related to a failure in writing the cache.
Also, make sure you have /opt/kaltura/web/html5/html5lib/v2.*/cache/
and that it too, in addition to /opt/kaltura/app/cache, is writeable for the apache user. Note that /opt/kaltura/app/cache should be writeable for both the ‘apache’ and ‘kaltura’ users whereas the HTML5 cache is only written to by apache.

Ok,
You may be right about the v2.*/cache I see that there is one which doesn’t have that folder.

Another question:
If I get it right the dwh is working on gz logs in the /opt/kaltura/web/logs
I work in aws environment with auto scaling, meaning the server can go down at any time because on lack of usage.
Is there any kaltura builtin solution to prevent “log lose”?

You need to make sure you move the access log to /opt/kaltura/web/logs which is on NFS at relatively small intervals and that, before the machine goes down for being unneeded [i,e, when the load is down], this happens.
Since /opt/kaltura/web/logs resides on the NFS, if you move the access logs to it before your algorithm brings the machine down, you should have no issues.

Yes, guessed so. Though you may have some builtin action for that.
Thanks.