Metadata Errors

Hello,

Created a custom metadata schema, and started using it on my content. After wanting to go back and edit the custom data, I’m receiving an error when I go to save saying “Metadata profile is currently transforming”

I see a batch job for this, but it never seems to complete. I can cancel the batch job, but it looks like the database is still marked as being locked/active (Metadata_profile table, row with my custom metadata has a status of “3”) Using kaltlog on my batch servers, I’m seeing a lot of PHP warnings regarding the metadata transform process:

PHP Warning: Invalid argument supplied for foreach() in /opt/kaltura/app/plugins/metadata/batch/Transform/KAsyncTransformMetadata.class.php on line 61

Could this be related?

Hi @siactive,

So, the exception happens here:
/opt/kaltura/app/plugins/metadata/services/MetadataProfileService.php

function updateAction($id, KalturaMetadataProfile $metadataProfile, $xsdData = null, $viewsData = null)
{
        $dbMetadataProfile = MetadataProfilePeer::retrieveByPK($id);

        if(!$dbMetadataProfile)
                throw new KalturaAPIException(MetadataErrors::METADATA_PROFILE_NOT_FOUND, $id);

        if($dbMetadataProfile->getStatus() != MetadataProfile::STATUS_ACTIVE)
                throw new KalturaAPIException(MetadataErrors::METADATA_TRANSFORMING); 

What was the API call you made to update the custom data?

The foreach warning comes from invalidateFailedMetadatas() and is definitely related.

Do you see any other errors in /opt/kaltura/log/kaltura_api_v3.log prior to the exception? we need to trace the flow and see what led to the data being invalid…

Hi Jess,

Just trying to update the metadata fields via the KMC at this point, so this is happening from the web interface. This error is thrown when I hit “save changes” directly from the Webserver:

2015-10-27 07:53:20 [0.000875] [] [1612074029] [27] [API] [KalturaStatement->execute] DEBUG: Sql took - 0.00063490867614746 seconds
2015-10-27 07:53:20 [0.002469] [] [1612074029] [28] [API] [KalturaFrontController->getExceptionObject] ERR: exception ‘KalturaAPIException’ with message ‘Metadata profile is currently transforming’ in /opt/kaltura/app/plugins/metadata/services/MetadataProfileService.php:158
Stack trace:
#0 [internal function]: MetadataProfileService->updateAction(2, Object(KalturaMetadataProfile), ‘<xsd:schema xml…’, ‘’)

One recurring thing I am seeing in older api_v3 logs is the following:

[API] [KalturaFrontController->errorHandler] NOTICE: /opt/kaltura/app/api_v3/lib/KalturaFrontController.php line 221 - Undefined offset: 1

Hey @siactive,

Does this reproduce with any other data schemas you have? can you try to manually set the status of the relevant schema to to 1 in the metadata_profile table and try to edit it again, while looking closely at the logs?
It may have been a one time issue and it just got stuck in that state or else, something in the values you are inputting is cause it to behave that way… would help if you can provide your schema’s XSD which you can download from the KMC and also the data you’re trying to input in your update call. That way, I can try to reproduce it.

Thanks,

Jess,

I’ll update the database directly, edit the schema and see if this resolves it. If it doesn’t, I’ll drop the XSD we are using here.

Great, thanks.
If it does happen again, please also specify the values you’re trying to update, along with the XSD, so I can try to reproduce.

Jess,

After setting the row to a status of 1 I was able to update the metadata via the web interface (Removed a System Name) and save it successfully. After updating it, I checked the database row for that metadata profile and it is stuck back in a status of 3, and I can’t update it it unless I set it back to 1. Changing the position of the System names, and saving it also gets the status stuck at 3 in the DB.

One batch server is still throwing the foreach() errors. Do you want me to dump the full output of the XML file here, or would you rather I send it to a dropoff somewhere?

Hi,

You can paste the XML here, thanks.
Can you explain what you mean by “Changing the position of the System names”? just trying to understand the exact action you’re taking.

Just using the arrows on the side of the dialogue box to reposition the fields, sorry if I was a bit vague.

XSD Starts Here:

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <xsd:element name="metadata">
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element id="md_B1CCCC9B-14C3-5277-B06F-8FA95EB7A74F" name="PublishDate" minOccurs="0" maxOccurs="1" type="dateType">
          <xsd:annotation>
            <xsd:documentation>Date the item was published</xsd:documentation>
            <xsd:appinfo>
              <label>Publish Date</label>
              <key>Publish Date</key>
              <searchable>true</searchable>
              <timeControl>false</timeControl>
              <description>Publish Date</description>
            </xsd:appinfo>
          </xsd:annotation>
        </xsd:element>
        <xsd:element id="md_4BFB2E5A-C3E8-0E7E-B372-8FB3D47B943E" name="School" minOccurs="0" maxOccurs="1">
          <xsd:annotation>
            <xsd:documentation>School</xsd:documentation>
            <xsd:appinfo>
              <label>School</label>
              <key>School</key>
              <searchable>true</searchable>
              <timeControl>false</timeControl>
              <description>School</description>
            </xsd:appinfo>
          </xsd:annotation>
          <xsd:simpleType>
            <xsd:restriction base="listType">
              <xsd:enumeration value="Alternative Schools"/>
              <xsd:enumeration value="Avalon Elementary"/>
              <xsd:enumeration value="Barron Collier High"/>
              <xsd:enumeration value="Big Cypress Elementary"/>
              <xsd:enumeration value="Calusa Park Elementary"/>
              <xsd:enumeration value="Corkscrew Elementary"/>
              <xsd:enumeration value="Corkscrew Middle"/>
              <xsd:enumeration value="Cypress Palm Middle"/>
              <xsd:enumeration value="East Naples Middle"/>
              <xsd:enumeration value="Eden Park Elementary"/>
              <xsd:enumeration value="Estates Elementary"/>
              <xsd:enumeration value="Everglades City"/>
              <xsd:enumeration value="Golden Gate Elementary"/>
              <xsd:enumeration value="Golden Gate High"/>
              <xsd:enumeration value="Golden Gate Middle"/>
              <xsd:enumeration value="Golden Terrace Elementary"/>
              <xsd:enumeration value="Gulf Coast High"/>
              <xsd:enumeration value="Gulfview Middle"/>
              <xsd:enumeration value="Highlands Elementary"/>
              <xsd:enumeration value="Immokalee High"/>
              <xsd:enumeration value="Immokalee Middle"/>
              <xsd:enumeration value="Lake Park Elementary"/>
              <xsd:enumeration value="Lake Trafford Elementary"/>
              <xsd:enumeration value="Laurel Oak Elementary"/>
              <xsd:enumeration value="Lely Elementary"/>
              <xsd:enumeration value="Lely High"/>
              <xsd:enumeration value="Lorenzo Walker High"/>
              <xsd:enumeration value="Manatee Elementary"/>
              <xsd:enumeration value="Manatee Middle"/>
              <xsd:enumeration value="Mike Davis Elementary"/>
              <xsd:enumeration value="Naples High"/>
              <xsd:enumeration value="Naples Park Elementary"/>
              <xsd:enumeration value="North Naples Middle"/>
              <xsd:enumeration value="Oakridge Middle"/>
              <xsd:enumeration value="Osceola Elementary"/>
              <xsd:enumeration value="Palmetto Elementary"/>
              <xsd:enumeration value="Palmetto Ridge High"/>
              <xsd:enumeration value="Parkside Elementary"/>
              <xsd:enumeration value="Pelican Marsh Elementary"/>
              <xsd:enumeration value="Pine Ridge Middle"/>
              <xsd:enumeration value="Pinecrest Elementary"/>
              <xsd:enumeration value="Poinciana Elementary"/>
              <xsd:enumeration value="Sabal Palm Elementary"/>
              <xsd:enumeration value="Sea Gate Elementary"/>
              <xsd:enumeration value="Shadowlawn Elementary"/>
              <xsd:enumeration value="Tommie Barfield Elementary"/>
              <xsd:enumeration value="Veterans Memorial Elementary"/>
              <xsd:enumeration value="Village Oaks Elementary"/>
              <xsd:enumeration value="Vineyards Elementary"/>
            </xsd:restriction>
          </xsd:simpleType>
        </xsd:element>
        <xsd:element id="md_76D56F14-81C4-9134-50D2-8FB485C470C6" name="Department" minOccurs="0" maxOccurs="1">
          <xsd:annotation>
            <xsd:documentation>Department</xsd:documentation>
            <xsd:appinfo>
              <label>Department</label>
              <key>Department</key>
              <searchable>true</searchable>
              <timeControl>false</timeControl>
              <description>Department</description>
            </xsd:appinfo>
          </xsd:annotation>
          <xsd:simpleType>
            <xsd:restriction base="listType">
              <xsd:enumeration value="Communications and Community Engagement"/>
              <xsd:enumeration value="Accountability and Data Management"/>
              <xsd:enumeration value="Accounting"/>
              <xsd:enumeration value="ADA"/>
              <xsd:enumeration value="Adult and Community Education"/>
              <xsd:enumeration value="Advanced Studies and Gifted Learners"/>
              <xsd:enumeration value="Budget"/>
              <xsd:enumeration value="Career and Technical Education"/>
              <xsd:enumeration value="Continuous Improvement Initiatives"/>
              <xsd:enumeration value="Counseling"/>
              <xsd:enumeration value="Curriculum and Instruction"/>
              <xsd:enumeration value="District General Counsel"/>
              <xsd:enumeration value="English Language Learner Programs"/>
              <xsd:enumeration value="Exceptional Education and Student Support"/>
              <xsd:enumeration value="Exceptional Student Education"/>
              <xsd:enumeration value="Facilities Management"/>
              <xsd:enumeration value="Federal/State and Competitive Grants"/>
              <xsd:enumeration value="Financial Services"/>
              <xsd:enumeration value="Fine Arts"/>
              <xsd:enumeration value="Head Start"/>
              <xsd:enumeration value="Health Education"/>
              <xsd:enumeration value="Human Resources"/>
              <xsd:enumeration value="Interscholastic Athletics"/>
              <xsd:enumeration value="Literacy"/>
              <xsd:enumeration value="Math"/>
              <xsd:enumeration value="Nutrition Services"/>
              <xsd:enumeration value="Payroll"/>
              <xsd:enumeration value="Physical Education"/>
              <xsd:enumeration value="PreK"/>
              <xsd:enumeration value="Purchasing"/>
              <xsd:enumeration value="School Board"/>
              <xsd:enumeration value="School Operations"/>
              <xsd:enumeration value="Science and Environment"/>
              <xsd:enumeration value="Security and Environmental"/>
              <xsd:enumeration value="Social Studies"/>
              <xsd:enumeration value="STEM Resources and Instructional Technology"/>
              <xsd:enumeration value="Student and Staff Projections"/>
              <xsd:enumeration value="Student Relations"/>
              <xsd:enumeration value="Superintendent Office"/>
              <xsd:enumeration value="Support Services"/>
              <xsd:enumeration value="Technology Services"/>
              <xsd:enumeration value="Transportation"/>
            </xsd:restriction>
          </xsd:simpleType>
        </xsd:element>
        <xsd:element id="md_9A7A05BD-701A-E553-2642-8FAB66AD3E81" name="Audience" minOccurs="0" maxOccurs="unbounded">
          <xsd:annotation>
            <xsd:documentation>Audience age for item.</xsd:documentation>
            <xsd:appinfo>
              <label>Audience</label>
              <key>Audience</key>
              <searchable>true</searchable>
              <timeControl>false</timeControl>
              <description>Audience age for item.</description>
            </xsd:appinfo>
          </xsd:annotation>
          <xsd:simpleType>
            <xsd:restriction base="listType">
              <xsd:enumeration value="Grade K-5"/>
              <xsd:enumeration value="Grade 6-8"/>
              <xsd:enumeration value="Grade 9-12"/>
              <xsd:enumeration value="Adult"/>
            </xsd:restriction>
          </xsd:simpleType>
        </xsd:element>
        <xsd:element id="md_6BD30C6C-86AB-0860-4E27-8FB78589E739" name="RelatedLinks" minOccurs="0" maxOccurs="unbounded" type="textType">
          <xsd:annotation>
            <xsd:documentation>Related Links</xsd:documentation>
            <xsd:appinfo>
              <label>Related Links</label>
              <key>Related Links</key>
              <searchable>true</searchable>
              <timeControl>false</timeControl>
              <description>Related Links</description>
            </xsd:appinfo>
          </xsd:annotation>
        </xsd:element>
      </xsd:sequence>
    </xsd:complexType>
  </xsd:element>
  <xsd:complexType name="textType">
    <xsd:simpleContent>
      <xsd:extension base="xsd:string"/>
    </xsd:simpleContent>
  </xsd:complexType>
  <xsd:complexType name="dateType">
    <xsd:simpleContent>
      <xsd:extension base="xsd:long"/>
    </xsd:simpleContent>
  </xsd:complexType>
  <xsd:complexType name="objectType">
    <xsd:simpleContent>
      <xsd:extension base="xsd:string"/>
    </xsd:simpleContent>
  </xsd:complexType>
  <xsd:simpleType name="listType">
    <xsd:restriction base="xsd:string"/>
  </xsd:simpleType>
</xsd:schema>

Hi @siactive,

So, I was unable to reproduce it using version 11.0.0 which is the current stable.
What I did is:

  • use /opt/kaltura/bin/create_metadata.php passing the XSD you provided as argument, which created the profile.
  • click on the position up and down buttons randomly
  • hit “Save Changes”

I got this alert box:
"Note: If you use XMLs for bulk upload,Reordering schema fields affects the validation of custom data elements in existing XMLs. If you reorder schema fields, you also need to update your bulk upload XML structure to reflect the new field order. Do you want to save your schema change"
and hit “Yes”

All went well enough.

What version are you using? also, can you check whether creating the schema the way I did makes any difference? it should not, I just used my script because I hate manual labour but maybe for some reason it causes a different behaviour…
/opt/kaltura/bin/create_metadata.php is part of the kaltura-postinst package and is used during the sanity check so you should have it available.

Jess,
I’ll give the script a shot. Should I delete the schema before creating it via the script?

EDIT

Also, currently running release 10.21 CE

Hi @siactive,

Cool, please let me know.
Also, I recommend you upgrade to latest, regardless.
Thanks,

Jess,

Quick update. Ran the script this morning to populate the metadata via the XSD file, and changes to the schema don’t seem to be causing status locks in the database. I’ll keep an eye on it, thanks for letting me know about that script. Saved me a lot of time :smile:

I’ll also look into 11.0. Thanks again.

Hi @siactive,

I am glad to hear it works well this way but it is still a bug:) if it reproduces for you with 11.0.0, can you kindly post an issue at github.com/kaltura/server? you can ref this post there.

Thanks,

Also, as far as learning how to work with our API, there are a lot of useful scripts under /opt/kaltura/bin for you to look at, they are all called as part of the kaltura-sanity.sh.
And of course, there is also developer.kaltura.org which is another good resource. To use, you need to open a free account with our SaaS, it a simple enough form to fill:)

Jess,
I was unaware of the developer site, thanks for that. Will definitely be taking a look at it.

Surething, enjoy.
Happy Kalturing,