Timeouts and considerations on a new solution

Hi,

I’ve recently put together a .NET solution (with great help from these forums) for uploading videos from our Classic ASP system to Kaltura. Generally things were working ok until we started pushing the size of the files being uploaded.

Details of our setup and how we are expecting it to work are as follows:

  • Windows server running our software using our own Linux server running Kaltura.
  • Video file is uploaded from user’s local drive to our .NET server. File is “chunked up” and uploaded via a buffer.
  • We are expecting the following interactions with the Kaltura API:
  1. Create session passing in supplied ID, admin_secret and secret strings.
  2. Create UploadTokenService upload token
  3. Call upload method on API
  4. Create MediaResource object
  5. Create MediaEntry object
  6. Save details to MediaEntry object
  • Poll Kaltura getting the status of the upload.
  • When the uploaded video is “converted” save details to our local database – duration, thumbnail and video id
  • The upload from local drive to .NET drive and then the import into Kaltura is all done in the same page.

When we first started getting the problem it appeared that any file above approximately 120MB was failing to upload. Upon investigation this was fixed by increasing the execution timeout but the import into Kaltura is still failing with a timeout error. The line of code generating the error was the one where we call the UploadTokenService.Upload method.

We are now considering a couple of alternative options going forwards:

  1. Keep the existing upload but terminate the page at that point. Then have another process that runs in the background to perform the import to Kaltura
  2. Upload the video file directly to the Kaltura server and do the import in one move. This option in particular we need advice on considering as it isn’t at all obvious from the API documentation if this is even possible.

What we are asking for are any constructive comments on our approach so far, whether anyone has hit this timeout issue in the past and thoughts on our proposed solutions. In particular the second solution as we are struggling to find if there is anything to achieve this in the API.

I’ve not included any code in the post yet as it is long enough already but I’m quite happy to post it in another post if it would prove useful.

Many thanks in anticipation for any help and advice.

Steve

Hi Steve,

It would be good to see the full code and also the stack trace that it generates.
Also, are you working against our SaaS or your own CE/OnPrem instance? if the latter, what version?

Another option you might want to consider for large files, BTW, is the bulk upload method which enables you to upload a CSV or an XML to the server, with the URLs for the files as well as the metadata and then the server will actually download these files and process them or you can use a dropfolder, in which case you simply define a profile where you provide access to this directory via SFTP for example and the server will automatically pull from it and process the files in much the same way.

Thanks,

Hi Jess,

Thanks for your reply.

We are running our own CE instance of Kaltura and the version number is v10.21.0.

Here is the code for the page where we select the file to upload and then trigger the upload: (apologies I can’t make the code formatting work properly on the forum)

<%@ Page Language=“C#” AutoEventWireup=“true” CodeBehind=“AddVideo.aspx.cs” Inherits=“TWMKaltura.AddVideo” %>

<%----%>
<link href="CSS/StyleSheet.css" rel="stylesheet" type="text/css" />
<link href="CSS/pager.css" rel="stylesheet" type="text/css" />
    <div class="subHeader">
    <div class="subHeaderTextContainer">

        <!--Sub header content start-->
        <h1>Video Manager - Video Import</h1>
        <p>This page allows you to upload videos, to start please click the 'Select file' button below and browse to your file.</p>
        <!--Sub header content finish-->

    </div>
</div>

<script type="text/javascript">
    var currFile = 0;
    var totalFileCount = 0;
    function sendFile(file) {
        //debugger;
        document.getElementById('progressSpinner').style.display = "block";
        

        var objectid = document.getElementById("hdnObjectID").value;
        var objectTypeid = document.getElementById("hdnObjectTypeID").value;
        var prefix = document.getElementById("hdnPrefix").value;
        var title = document.getElementById("hdnTitle").value;
        var kas = document.getElementById("hdnkas").value;
        var kus = document.getElementById("hdnkus").value;
        



        $.ajax({
            url: 'FileUpload.aspx?objectid=' + objectid + '&objecttypeid=' + objectTypeid + '&prefix=' + prefix + '&title=' + title + '&FileName=' + file.name + "&kas=" + kas + "&kus=" + kus, //server script to process data
            type: 'POST',
            xhr: function () {
                myXhr = $.ajaxSettings.xhr();
                if (myXhr.upload) {
                    myXhr.upload.addEventListener('progress', progressHandlingFunction, false);
                }
                return myXhr;
            },
            success: function (result) {
                document.getElementById('progressSpinner').style.display = "none";
                //On success if you want to perform some tasks.
                switch (result) {
                    case 'Success':// Import Success
                        document.getElementById("dvProgess").text = "Import succesfull";
                        document.getElementById('fileUploaded').style.display = "block";
                        document.getElementById('uploadModule').style.display = "none";
                        window.location.replace("Completed.aspx");
                    break;
                    case 'Fail':// Import Failed check import log in DB
                        alert("fail");
                        document.getElementById("progress" + currFile).hide
                        document.getElementById("dvProgess").text = "Import failed - Please try again or contact technical support";
                        break;
                    default:
                        alert(result);
                        break;


                }
            },
            data: file,
            cache: false,
            contentType: false,
            processData: false
        });
        function progressHandlingFunction(e) {
            if (e.lengthComputable) {
                var s = parseInt((e.loaded / e.total) * 100);
                $("#progress" + currFile).text(s + "%");
                $("#progbarWidth" + currFile).width(s + "%");
                if (s == 100) {
                    triggerNextFileUpload();
                }
            }
        }
    }

    function triggerNextFileUpload() {
        if (currFile < totalFileCount - 1) {
            currFile = currFile + 1;
            sendFile($("#fileInput")[0].files[currFile]);
        }
        else {
            $("#fileInput").replaceWith($("#fileInput").clone());
        }
    }

    function FileSelected() {
        document.getElementById('uploadModule').style.display = "block";
        document.getElementById('fileUploaded').style.display = "none";
        $("#dvProgess").html('');
        totalFileCount = $("#fileInput")[0].files.length;
        for (j = 0; j < totalFileCount; j++) {
            var progControl = $("#dvProgess").append($("#fileInput")[0].files[j].name + "<div id='progress" + j + "'></div><div class='progress progress-striped' id='ProgressComp" + j + "' style='width: 400px; height: 15px; '> <div class='bar' id='progbarWidth" + j + "' style='width: 0%; height: 15px;'>&nbsp;</div></div>");
        }
    }

    function uploadFile() {
        currFile = 0;
        sendFile($("#fileInput")[0].files[0]);
    }
</script>

<div class="formContainer">
    <div class="formInputContainer">
        <h3>Step 1. Select your video.</h3>
        <div class="formButtonLink"><a class="" onclick="document.getElementById('fileInput').click();" role="button">Select File</a></div>
    </div>
</div>

<div class="formContainer" id="uploadModule" style="padding-top: 0px; display: none;">
    <div class="formInputContainer">
        <h3>Step 2. Upload your file</h3>
        <div class="formButtonLink"><a class="" onclick="javascript:uploadFile();" role="button">Upload file</a></div>

        <br /><br />
        <h3 style="margin: 0px;">File upload progress</h3>

        <input id="fileInput" multiple="multiple" onchange="FileSelected();" style="visibility: hidden; width: 1px; height: 1px" type="file" />
        <!--<div id="progress">&nbsp;</div>-->
        <div id="dvProgess">&nbsp;</div>

        <div id="progressSpinner" style="display: none;"><img src="img/ajax_spinner.gif" width="50px;"/></div>

    </div>
</div>

<div class="formContainer" id="fileUploaded" style="padding-top: 0px; display: none;">
       <div class="formInputContainer">
            <h3>Success!</h3>
            <div>Your file has been uploaded. If you would like to upload another please select another file above.  To manage your modules please <a href="CourseManagerHome.aspx">click here.</a></div>
        </div>
</div>

<input type="hidden" id="hdnObjectID" name="hdnObjectID" value=<%=Request.QueryString["objectId"] %> />
<input type="hidden" id="hdnObjectTypeID" name="hdnObjectTypeID" value=<%=Request.QueryString["objectTypeId"] %> />
<input type="hidden" id="hdnPrefix" name="hdnPrefix" value=<%=Request.QueryString["prefix"] %> />
<input type="hidden" id="hdnTitle" name="hdnTitle" value=<%=Request.QueryString["Title"] %> />
<input type="hidden" id="hdnkas" name="hdnkas" value=<%=Request.QueryString["kas"] %> />
<input type="hidden" id="hdnkus" name="hdnkus" value=<%=Request.QueryString["kus"] %> />

</div>
</form>

Here is the code from the FileUpload.aspx which is called from the code above:

using System;
using System.Web;
using System.Text.RegularExpressions;
using System.Configuration;
using System.Data;
using System.Data.SqlClient;
using System.IO;
using System.Diagnostics;
using Kaltura;

namespace TWMKaltura
{
public partial class FileUpload : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
Page.Server.ScriptTimeout = 1000;
FileUploader fu = new FileUploader();
fu.ProcessRequest(HttpContext.Current);
}

    //Routine to check filename for invalid chars
    private static readonly Regex InvalidFileRegex = new Regex(string.Format("[{0}]", Regex.Escape(@"<>:""/\|?*")));

    public static string SanitizeFileName(string fileName)
    {
        return InvalidFileRegex.Replace(fileName, string.Empty);
    }


    public class FileUploader : IHttpHandler
    {
        //private const int PARTNER_ID = 101; //enter your partner id
        static int PARTNER_ID; //enter your partner id

        //private const string SECRET = "d2b73ff16b6e090319299e61918d6992";
        //private const string ADMIN_SECRET = "8ad9971e1ab7dcb6d5266bb21a4ce166"; //enter your admin secret
        static string SECRET = "";
        static string ADMIN_SECRET = "";

        //static string SERVICE_URL = "https://twmmediaserver.com";
        //static string USER_ID = "testUser";

        static string SERVICE_URL = "";
        static string USER_ID = "";

        static KalturaConfiguration GetConfig()
        {
            KalturaConfiguration config = new KalturaConfiguration(PARTNER_ID);
            SERVICE_URL = ConfigurationManager.AppSettings["SERVICE_URL"].ToString();
            USER_ID = ConfigurationManager.AppSettings["USER_ID"].ToString();
            config.ServiceUrl = SERVICE_URL;
            return config;
        }

        private void setupKaltura(HttpContext context)
        {
            SECRET = context.Request.QueryString["kus"].ToString();
            ADMIN_SECRET = context.Request.QueryString["kas"].ToString();
            PARTNER_ID = Convert.ToInt16(ConfigurationManager.AppSettings["PARTNER_ID"]);

        }

        //this function checks if a given flavor system name exist in the account.
        static int? CheckIfFlavorExist(String name)
        {
            KalturaClient client = new KalturaClient(GetConfig());
            string ks = client.GenerateSession(ADMIN_SECRET, USER_ID, KalturaSessionType.ADMIN, PARTNER_ID, 86400, "");
            client.KS = ks;

            //verify that the account we're testing has the new iPad flavor enabled on the default conversion profile
            KalturaConversionProfile defaultProfile = client.ConversionProfileService.GetDefault();
            KalturaConversionProfileAssetParamsFilter flavorsListFilter = new KalturaConversionProfileAssetParamsFilter();
            flavorsListFilter.SystemNameEqual = name;
            flavorsListFilter.ConversionProfileIdEqual = defaultProfile.Id;

            KalturaConversionProfileAssetParamsListResponse list = client.ConversionProfileAssetParamsService.List(flavorsListFilter);
            if (list.TotalCount > 0)
                return list.Objects[0].AssetParamsId;
            else
                return null;
        }

        public void ProcessRequest(HttpContext context)
        {

            setupKaltura(context);

            try
            {

                string fileName = HttpContext.Current.Request.QueryString["FileName"].ToString();
                fileName = SanitizeFileName(fileName);
                
                using (FileStream fs = File.Create(context.Server.MapPath("~")  + @"\" + fileName))
                {
                    Byte[] buffer = new Byte[32 * 1024];
                    int read = context.Request.GetBufferlessInputStream().Read(buffer, 0, buffer.Length);
                    while (read > 0)
                    {
                        fs.Write(buffer, 0, read);
                        read = context.Request.GetBufferlessInputStream().Read(buffer, 0, buffer.Length);
                    }
                }

                //context.Response.Write("File uploaded from local drive");

                string objectid = HttpContext.Current.Request.QueryString["objectid"];
                string objectTypeid = HttpContext.Current.Request.QueryString["objectTypeid"];
                string prefix = HttpContext.Current.Request.QueryString["prefix"];

                SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["Database"].ConnectionString);
                

                // Get details saved already for this video which are then used
                // when importing the video
                SqlDataAdapter da = new SqlDataAdapter("Select * from tbl_D2_Object where objectid=" + objectid + " and objecttypeid=" + objectTypeid, conn);
                DataSet ds = new DataSet();
                da.Fill(ds);

                if (ds.Tables[0].Rows.Count > 0)
                {

                    FileStream fileStream = new FileStream(context.Server.MapPath("~") + @"\" + fileName, FileMode.Open, FileAccess.Read);
                    //context.Response.Write("File stream created");

                    KalturaClient client = new KalturaClient(GetConfig());
                    string ks = client.GenerateSession(ADMIN_SECRET, USER_ID, KalturaSessionType.ADMIN, PARTNER_ID, 86400, "");
                    client.KS = ks;
                    
                    //context.Response.Write("Session created");

                    KalturaUploadToken uploadToken = client.UploadTokenService.Add();

                    //context.Response.Write("Upload start");
                    
                    client.UploadTokenService.Upload(uploadToken.Id, fileStream);

                    //context.Response.Write("Upload end");

                    KalturaUploadedFileTokenResource mediaResource = new KalturaUploadedFileTokenResource();
                    mediaResource.Token = uploadToken.Id;
                    KalturaMediaEntry mediaEntry = new KalturaMediaEntry();

                    //context.Response.Write("Media entry created");

                    mediaEntry.Name = ds.Tables[0].Rows[0]["Synopsis"].ToString();
                    mediaEntry.ReferenceId = prefix + "-" + objectid;
                    mediaEntry.MediaType = KalturaMediaType.VIDEO;
                    mediaEntry = client.MediaService.Add(mediaEntry);
                    mediaEntry = client.MediaService.AddContent(mediaEntry.Id, mediaResource);

                    //context.Response.Write("Media entry details set");


                    string videoid = mediaEntry.Id;
                    string thumbnail = mediaEntry.ThumbnailUrl;

                    string status = mediaEntry.Status.ToString();
                    int duration = mediaEntry.Duration;
                    
                    //int duration = 0;

                    while (status != "1")
                    {
                        client.StartMultiRequest();
                        client.SessionService.Start(ADMIN_SECRET, "", KalturaSessionType.ADMIN, PARTNER_ID, 86400);
                        KalturaMediaEntry mediaEntry2 = client.MediaService.Get(videoid);

                        status = mediaEntry2.Status.ToString();
                        duration = mediaEntry2.Duration;
                    }


                    //context.Response.Write("Start db updates");

                    conn.Open();

                    SqlDataAdapter da2 = new SqlDataAdapter("Select * from tbl_D2_ObjectExtras where objectid=" + objectid + " and objecttypeid=" + objectTypeid + " and Prefix='" + prefix + "'", conn);
                    DataSet ds2 = new DataSet();
                    da2.Fill(ds2);


                    if (ds2.Tables[0].Rows.Count > 0)
                    {
                        SqlCommand CMD = new SqlCommand("UPDATE tbl_D2_ObjectExtras SET Prefix='" + prefix + "', ObjectId =" + objectid + ", objectTypeId =" + objectTypeid + ", Filename = '" + videoid + "', Thumbnail = '" + thumbnail + "', Duration = " + duration + " WHERE Prefix = '" + prefix + "' AND ObjectID =" + objectid + " AND objectTypeId =" + objectTypeid, conn);
                        CMD.ExecuteNonQuery();
                    }
                    else
                    {
                        //string sql = "INSERT INTO tbl_D2_ObjectExtras (Prefix, ObjectID, PageNo, ObjectTypeID, Filename, Thumbnail, Duration) VALUES ('" + prefix + "'," + objectid + ",1," + objectTypeid + ",'" + videoid + "','" + thumbnail + "'," + duration + ")";
                        //context.Response.Write(sql);
                        SqlCommand cmd = new SqlCommand("INSERT INTO tbl_D2_ObjectExtras (Prefix, ObjectID, PageNo, ObjectTypeID, Filename, Thumbnail, Duration) VALUES ('" + prefix + "'," + objectid + ",1," + objectTypeid + ",'" + videoid + "','" + thumbnail + "'," + duration + ")", conn);
                        cmd.ExecuteNonQuery();
                    }

                    // Set the status on the object table to offline
                    SqlCommand cmd2 = new SqlCommand("UPDATE tbl_D2_Object SET Status = 1 WHERE ObjectID = " + objectid + " AND ObjectTypeID = 16", conn);
                    cmd2.ExecuteNonQuery();

                    //context.Response.Write("End db updates");

                    context.Response.Write("Success");
                    
                }
            }
            catch(Exception ex)
            {
                //context.Response.Write(ex.Message);
                context.Response.Write("Fail");
            }
        }

        public bool IsReusable
        {
            get
            {
                return false;
            }
        }
    }

}

}

Many thanks,

Steve

Steve, we are experiencing the exact problem. Our front end is Drupal and we wrote a service to upload in chunks and then that service calls the upload raw service that sends the file to Kaltura CE. We are getting timeouts only when we load video files over 15 megs or so. We are choosing to write an upload service in Kaltura and that service will call a Drupal service (to be written) to send the Kaltura meta data. We have been spinning on this issue for 6 weeks and finally just decided to move in this direction.