Posts Tagged 'Transcoding'

September 12, 2012

How Can I Use SoftLayer Message Queue?

One of the biggest challenges developers run into when coding large, scalable systems is automating batch processes and distributing workloads to optimize compute resource usage. More simply, intra-application and inter-system communications tend to become a bottleneck that affect the user experience, and there is no easy way to get around it. Well ... There *was* no easy way around it.

Meet SoftLayer Message Queue.

As the name would suggest, Message Queue allows you to create one or more "queues" or containers which contain "messages" — strings of text that you can assign attributes to. The queues pass along messages in first-in-first-out order, and in doing so, they allow for parallel processing of high-volume workflows.

That all sounds pretty complex and "out there," but you might be surprised to learn that you're probably using a form of message queuing right now. Message queuing allows for discrete threads or applications to share information with one another without needing to be directly integrated or even operating concurrently. That functionality is at the heart of many of the most common operating systems and applications on the market.

What does it mean in a cloud computing context? Well, Message Queue facilitates more efficient interaction between different pieces of your application or independent software systems. The easiest way demonstrate how that happens is by sharing a quick example:

Creating a Video-Sharing Site

Let's say we have a mobile application providing the ability to upload video content to your website: sharevideoswith.phil. The problem we have is that our webserver and CMS can only share videos in a specific format from a specific location on a CDN. Transcoding the videos on the mobile device before it uploads proves to be far too taxing, what with all of the games left to complete from the last Humble Bundle release. Having the videos transcoded on our webserver would require a lot of time/funds/patience/knowledge, and we don't want to add infrastructure to our deployment for transcoding app servers, so we're faced with a conundrum. A conundrum that's pretty easily answered with Message Queue and SoftLayer's (free) video transcoding service.

What We Need

  • Our Video Site
  • The SoftLayer API Transcoding Service
  • SoftLayer Object Storage
    • A "New Videos" Container
    • A "Transcoded Videos" Container with CDN Enabled
  • SoftLayer Message Queue
    • "New Videos" Queue
    • "Transcoding Jobs" Queue

The Process

  1. Your user uploads the video to sharevideoswith.phil. Your web app creates a page for the video and populates the content with a "processing" message.
  2. The web application saves the video file into the "New Vidoes" container on object storage.
  3. When the video is saved into that container, it creates a new message in the "New Videos" message queue with the video file name as the body.
  4. From here, we have two worker functions. These workers work independently of each other and can be run at any comfortable interval via cron or any scheduling agent:
Worker One: Looks for messages in the "New Videos" message queue. If a message is found, Worker One transfers the video file to the SoftLayer Transcoding Service, starts the transcoding process and creates a message in the "Transcoding Jobs" message queue with the Job ID of the newly created transcoding job. Worker One then deletes the originating message from the "New Videos" message queue to prevent the process from happening again the next time Worker One runs.

Worker Two: Looks for messages in the "Transcoding Jobs" queue. If a message is found, Worker Two checks if the transcoding job is complete. If not, it does nothing with the message, and that message is be placed back into the queue for the next Worker Two to pick up and check. When Worker Two finds a completed job, the newly-transcoded video is pushed to the "Transcoded Videos" container on object storage, and Worker Two updates the page our web app created for the video to display an embedded media player using the CDN location for our transcoded video on object storage.

Each step in the process is handled by an independent component. This allows us to scale or substitute each piece as necessary without needing to refactor the other portions. As long as each piece receives and sends the expected message, its colleague components will keep doing their jobs.

Video transcoding is a simple use-case that shows some of the capabilities of Message Queue. If you check out the Message Queue page on our website, you can see a few other examples — from online banking to real-time stock, score and weather services.

Message Queue leverages Cloudant as the highly scalable low latency data layer for storing and distributing messages, and SoftLayer customers get their first 100,000 messages free every month (with additional messages priced at $0.01 for every 10,000).

What are you waiting for? Go get started with Message Queue!

-Phil (@SoftLayerDevs)

April 7, 2011

Thou Shalt Transcode

Deep in the depths of an ancient tomb of the great Abswalli, you and your team accidentally awaken the Terbshianaki ghost army. You’re disconnected from the supply caravan with the valuable resources that could not only sustain your journey but also save your team. As Zeliagh the Protesiann hunter fires his last arrow, you come to the sudden realization that continuing your quest is now hopeless. Alas, true terror was unknown before this moment as you come to the most surprising realization: The one thing you truly can't live without is your trusty server that converts one type of media into another.

Fear not great adventurer, for I, Phil of the SLAPI, have come, and I bear the gifts of empowerment, automation and integration. Freedom from the horror of your epiphany can be found in our complementary media transcoding service.
Before we can begin, some preparation is required. First, you must venture to our customer portal and create a transcoding user: Private Network->Transcoding. As you know from the use of your other SoftLayer spoils, you shan't be obligated to access this functionality from your web browser. You can summon the API wizardry bequeathed to you by coders of old in the the SLDN scroll: SoftLayer_Network_Media_Transcode_Account::createTranscodeAccount.*

*For the sake of this blog, we'll abbreviate "SoftLayer_Network_Media_Transcode_Account" as "SNMTA" from here forward ... Shortening it helps with blog formatting.

You must then construct an object to represent a SoftLayer Network Media Transcode Job, like our SoftLayer Network Media Transcode Job template object. This template object will be built with a number of properties. Your pursuit in relieving your aforementioned horror only necessitates the use of the required properties.

You will need to decide in which format the final treasure will take form. You may find this information with the SNMTA::getPresets method.

$client = SoftLayer_SoapClient::getClient('SoftLayer_Network_Media_Transcode_Account', $trandcodeAccountId, $apiUsername, $apiKey);
$transcodePresets = $client->getPresets();
print_r($transcodePresets);
Array
(
    [0] => stdClass Object
        (
            [GUID] => {9C3716B9-C931-4873-9FD1-03A17B0D3350}
            [category] => Application Specific
            [description] => MPEG2, Roku playback, 1920 x 1080, Interlaced, 29.97fps, 25Mbps, used with Component/VGA connection.
            [name] => MPEG2 - Roku - 1080i
        )
 
    [1] => stdClass Object
        (
            [GUID] => {03E81152-2A74-4FF3-BAD9-D1FF29973032}
            [category] => Application Specific
            [description] => MPEG2, Roku playback, 720 x 480, 29.97fps, 6.9Mbps, used with Component/S-Video connection.
            [name] => MPEG2 - Roku - 480i
        )
 
    [2] => stdClass Object
        (
            [GUID] => {75A264DB-7FBD-4976-A422-14FBB7950BD1}
            [category] => Application Specific
            [description] => MPEG2, Roku playback, 720 x 480, Progressive, 29.97fps, 6.9Mbps, used with Component/VGA connection.
            [name] => MPEG2 - Roku - 480p
        )
.....

The freedom to use this power (the more you know!) is yours, in this instance, I scrolled through let my intuition find the option which just felt right:

stdClass Object
(
            [GUID] => {21A33980-5D78-4010-B4EB-6EF15F5CD69F}
            [category] => Web\Flash
            [description] =>
            [name] => FLV 1296kbps 640x480 4x3 29.97fps
        )

To decipher this language we must know the following:

  1. The GUID is the unique identifier which we will use to reference our champion
  2. The category section is used to group like presets together, this will be useful for those who's journey leads down the path of GUI creation
  3. A description of the preset, if one is available, will be listed under description
  4. name is simply a human-readable name for our preset

You are nearly ready to restore your yearned for transcoding service as the ghostly horde presses the defensive perimeter. We have but one more task of preparation: We must provide the transcoding service a file! Using your Wand of File Transference +3, or your favorite FTP client, you enter the details for your transcode FTP account found on the Transcoding page of the IMS (or of course SNMTA::getFtpAttributes) and choose the "in" directory as the destination for your source file. Lacking no other option, you may call upon Sheshura, a fairy sprite, specializing in arcane documents for a source video file: Epic Battle

The battle rages around you, as the Wahwatarian mercenaries protect your flank. The clicking of your laptop keys twist and merge in the air around your ears only to transcend into a booming chorus of "The Flight of the Valkyries" as you near transcoding Utopia. You strike:

<?php
//  Create a transcoding client
$client = SoftLayer_SoapClient::getClient('SoftLayer_Network_Media_Transcode_Job', null, $apiUsername, $apiKey);
 
// Define our preset GUID and filename
$presetGUID = '{95861D24-9DF5-405E-A130-A40C6637332D}';
$inputFile = 'video.mov';
 
/*
 * The transcoding service will append the new file extension to the output file
 * so we strip the extension here.
 */
$outputFile = substr($inputFile, 0, strrpos($inputFile, '.'));
 
try {
    // Create a SoftLayer_Network_Media_Transcode_Job template object with the required properties
    $transcodeJob = new stdClass();
    $transcodeJob->transcodePresetGuid = $presetGUID;
    $transcodeJob->inputFile = "/in/$inputFile";
    $transcodeJob->outputFile = "/out/$outputFile";
 
    // Call createObject() with our template object as a parameter
    $result = $client->createObject($transcodeJob);
    // $result will contain a SoftLayer_Network_Media_Transcode_Job object
    print_r($result);
} catch ( Exception $e) {
    die( $e->getMessage());
}

If your will did not waver nor did your focus break in the face of ever-closing ghouls pounding your resolve, your treasure will be waiting. Brandish your Wand of File Transference +3, or utilize your favorite FTP client to retrieve your reward: "out/video.flv"

If the gods be with thee, your resulting file should look like this: Epic Battle (in .flv)

With your victory fresh upon the tablets of history, you can now encode to any of our supported formats. Try using the process above to convert the video to .mp4 format so your resulting file output is Epic Battle (in .mp4)!

-Phil

P.S. If you're going to take off your training wheels, the second example uses "[description] => MPEG4 file, 320x240, 15fps, 256kbps for download" for the bandwidth impaired.

Subscribe to transcoding