Contents

MovieEditor

The steps for working with a movie editor are:

Tests associated with using the MovieEditor object:
Ruby test script for working with the MovieEditor object
Swift 1.2 tests of working with the MovieEditor object

Ruby and JSON example of using the Movie Editor:
The ruby used to generate JSON using MovieEditor for Zukini Demo app.
The generated JSON for the movie editor used in the Zukini Demo app

Closing the movie editor object does not result in an export. Since there is no way to undo adding tracks, adding segments, adding compositing instructions and that all the work of compositing happens at export time having export happen coincidentally with closing the object is expensive for possibly unintended behaviour. This is different behaviour of the video frames writer object where much of the work of saving the file happens as frames are added and the finalise command is relatively low cost.

Creating and closing a MovieEditor

The command module method to create a movie importer:

createMovieEditorCommand = CommandModule.make_createmovieeditor(
                            name: 'documentation.movieeditor')
puts JSON.pretty_generate(createMovieEditorCommand.commandhash)

The generated JSON is:

{
  "command": "create",
  "objecttype": "movieeditor",
  "objectname": "documentation.movieeditor"
}

You can also create a movie editor using make_createmovieeditor method of a SmigCommands object. With default arguments a name for the object will be automatically generated and a close command for the object is automatically added to the list of cleanup commands.

theCommands = SmigCommands.new
movieEditor = theCommands.make_createmovieeditor()
puts JSON.pretty_generate(theCommands.commandshash)

The generated JSON is:

{
  "commands": [
    {
      "command": "create",
      "objecttype": "movieeditor",
      "objectname": "5d0a941f-fa31-4267-a36a-adcb6cf45ccf"
    }
  ],
  "cleanupcommands": [
    {
      "command": "close",
      "receiverobject": {
        "objecttype": "movieeditor",
        "objectname": "5d0a941f-fa31-4267-a36a-adcb6cf45ccf"
      }
    }
  ]
}

Adding tracks

You can add audio tracks as well as video tracks. The mediatype for an audio track is :soun.

Assuming we have a movieEditor identifier and that a SmigCommands object theCommands has been created. To add a track to the movie editor you use the 'createtrack' command. In ruby:

addVideoTrackCommand = CommandModule.make_createtrackcommand(
                                                movieEditor,
                                     mediatype: :vide)
puts JSON.pretty_generate(addVideoTrackCommand.commandhash)

The generated JSON is:

{
  "command": "createtrack",
  "receiverobject": {
    "objecttype": "movieeditor",
    "objectname": "5d0a941f-fa31-4267-a36a-adcb6cf45ccf"
  },
  "mediatype": "vide"
}

If you want to add more than one track you can add the create track command to the list of commands once for each track you want to add.

theCommands.add_command(addVideoTrackCommand)
theCommands.add_command(addVideoTrackCommand)
puts JSON.pretty_generate(theCommands.commandshash)

The generated JSON is:

{
  "commands": [
    {
      "command": "create",
      "objecttype": "movieeditor",
      "objectname": "5d0a941f-fa31-4267-a36a-adcb6cf45ccf"
    },
    {
      "command": "createtrack",
      "receiverobject": {
        "objecttype": "movieeditor",
        "objectname": "5d0a941f-fa31-4267-a36a-adcb6cf45ccf"
      },
      "mediatype": "vide"
    },
    {
      "command": "createtrack",
      "receiverobject": {
        "objecttype": "movieeditor",
        "objectname": "5d0a941f-fa31-4267-a36a-adcb6cf45ccf"
      },
      "mediatype": "vide"
    }
  ],
  "cleanupcommands": [
    {
      "command": "close",
      "receiverobject": {
        "objecttype": "movieeditor",
        "objectname": "5d0a941f-fa31-4267-a36a-adcb6cf45ccf"
      }
    }
  ]
}

Adding segments to tracks

It is probably best to add segments in time order. It keeps things simplest that way. If you insert a segment prior to the end of another segment in the same track it is not always clear what will happen. If you insert a segment with the same start time as another segment on the same track the other segment will be pushed to start after the inserted segments end time. It is possible to get information about inserted segments using the 'getproperty' command. In ruby test file test018 (see links above), test02 obtains the track segment mappings before and after the insertion of an empty segment that moves another segment.

You can also add an image map of the current state of the video composition showing tracks, segments and composition instructions to the image collection. See section below for how to do this.

Video tracks in the movie editor do not have to have segments with content through their full range. Content in a time range in a track is needed where a layer instruction refers to that track and time range.

Adding a segment with content

Ruby for inserting a track segment from a movie importer object into a track of the movie editor. To add sound content to an audio track you need to make sure the source is an audio track and vice versa for video content.

movieImporter = SmigIDHash.make_objectid(objecttype: :movieimporter,
                            objectname: 'documentation.movieimporter')
movieEditor = SmigIDHash.make_objectid(objecttype: :movieeditor,
                            objectname: 'documentation.movieeditor')

track0 = MovieTrackIdentifier.make_movietrackid_from_mediatype(
                                        mediatype: :vide, trackindex: 0)
sourceTrack = MovieTrackIdentifier.make_movietrackid_from_mediatype(
                                        mediatype: :vide, trackindex: 0)

timeZero = MovieTime.make_movietime(timevalue: 0, timescale: 6000)
duration = MovieTime.make_movietime(timevalue: 12000, timescale: 6000)
sourceTimeRange = MovieTime.make_movie_timerange(start: timeZero,
                                              duration: duration)
insertionTime = MovieTime.make_movietime(timevalue: 0, timescale: 6000)
insertTrackSegmentCommand = CommandModule.make_inserttracksegment(
                            movieEditor, 
                     track: track0,
             source_object: movieImporter,
              source_track: sourceTrack,
             insertiontime: timeZero, 
          source_timerange: sourceTimeRange)

puts JSON.pretty_generate(insertTrackSegmentCommand.commandhash)

The generated JSON:

{
  "command": "inserttracksegment",
  "receiverobject": {
    "objecttype": "movieeditor",
    "objectname": "documentation.movieeditor"
  },
  "track": {
    "trackindex": 0,
    "mediatype": "vide"
  },
  "sourceobject": {
    "objecttype": "movieimporter",
    "objectname": "documentation.movieimporter"
  },
  "sourcetrack": {
    "trackindex": 0,
    "mediatype": "vide"
  },
  "insertiontime": {
    "value": 0,
    "timescale": 6000,
    "flags": 1,
    "epoch": 0
  },
  "sourcetimerange": {
    "start": {
      "value": 0,
      "timescale": 6000,
      "flags": 1,
      "epoch": 0
    },
    "duration": {
      "value": 12000,
      "timescale": 6000,
      "flags": 1,
      "epoch": 0
    }
  }
}

The above describes taking 2 seconds of video content from video track 0 of a movie importer and inserts it at the beginning of track 0 of the movie editor.

Adding an empty segment

You can also add empty segments into tracks. Ruby for adding an empty track segment:

movieEditor = SmigIDHash.make_objectid(objecttype: :movieeditor,
                            objectname: 'documentation.movieeditor')
targetTID = MovieTrackIdentifier.make_movietrackid_from_mediatype(
                                        mediatype: :vide, trackindex: 0)
timeZero = MovieTime.make_movietime(timevalue: 0, timescale: 6000)
duration = MovieTime.make_movietime(timevalue: 12000, timescale: 6000)
emptySegmentTimeRange = MovieTime.make_movie_timerange(
                                start: timeZero,
                             duration: duration)
insertEmptySegmentCommand = CommandModule.make_insertemptysegment(
                                movieEditor,
                         track: targetTID,
            insertiontimerange: emptySegmentTimeRange)
puts JSON.pretty_generate(insertEmptySegmentCommand.commandhash)

The generated JSON is:

{
  "command": "insertemptytracksegment",
  "receiverobject": {
    "objecttype": "movieeditor",
    "objectname": "documentation.movieeditor"
  },
  "track": {
    "trackindex": 0,
    "mediatype": "vide"
  },
  "timerange": {
    "start": {
      "value": 0,
      "timescale": 6000,
      "flags": 1,
      "epoch": 0
    },
    "duration": {
      "value": 12000,
      "timescale": 6000,
      "flags": 1,
      "epoch": 0
    }
  }
}

Adding composition instructions

Composition instructions define how 1 or more video tracks are combined to compose the final output video. Each composition instruction is created from a list of layer instructions. Each layer instruction refers to one track. If you don't add the composition instructions in temporal order the output video will fail to be composed. Each layer instruction applies only within the time period of the composition instruction but the time specification is not defined by the time range of the composition instruction but of the time frame of the video.

The order of the layer instructions is important. The first layer instruction in the list of layer instructions is at the front, and the last layer instruction is at the back.

If the first layer instruction was a pass thru layer instruction and the video track applied was the same size as the output video composition then it would cover over all other instructions. If you just want one a specific video track as output for a time range of the output video then you would create composition instruction with a single layer instruction. That single layer instruction would be a passthru instruction.

The different layer instructions

All the layer instructions have a start and duration times. For the non ramp layer instructions then the instruction applies unchanging for the duration of the layer instruction. For the ramp layer instructions the instruction varies continuously from the start ramp value at the beginning to the end ramp value at the end of the duration.

Currently not implemented is the ability to have more than one type of instruction in a single layer instruction, so for example you can not have a transform with opacity ramp applied in one layer instruction.

The example I use here is to have the first couple of seconds have track 0 be the source for the output, then followed by a dissolve ramp that lasts for two seconds which slowly dissolves out track 0 and replaces it with track 1 and then 2 seconds where the output is just track 1. Each subsection below implements one instruction with one or more layer instructions.

We will assume that the tracks have been setup, and that segments with content have been added. test03 of ruby test file test018 covers the complete process.

Adding a passthru instruction

Track 0 has a passthru layer instruction applied which means it is composited to the output video without modification while track 1 is ignored.

movieEditor = SmigIDHash.make_objectid(objecttype: :movieeditor,
                                objectname: 'documentation.movieeditor')
track0 = MovieTrackIdentifier.make_movietrackid_from_mediatype(
                                        mediatype: :vide, trackindex: 0)
passThru0 = VideoLayerInstructions.new
passThru0.add_passthrulayerinstruction(track: track0)
timeZero = MovieTime.make_movietime(timevalue: 0, timescale: 6000)
duration = MovieTime.make_movietime(timevalue: 12000, timescale: 6000)
passThru0TimeRange = MovieTime.make_movie_timerange(start: timeZero,
                                                duration: duration)
passThru1Command = CommandModule.make_addvideoinstruction(movieEditor,
                                             timerange: passThru0TimeRange,
                                     layerinstructions: passThru0)
puts JSON.pretty_generate(passThru1Command.commandhash)

The generated JSON is:

{
  "command": "addmovieinstruction",
  "receiverobject": {
    "objecttype": "movieeditor",
    "objectname": "documentation.movieeditor"
  },
  "timerange": {
    "start": {
      "value": 0,
      "timescale": 6000,
      "flags": 1,
      "epoch": 0
    },
    "duration": {
      "value": 12000,
      "timescale": 6000,
      "flags": 1,
      "epoch": 0
    }
  },
  "layerinstructions": [
    {
      "layerinstructiontype": "passthruinstruction",
      "track": {
        "trackindex": 0,
        "mediatype": "vide"
      }
    }
  ]
}

Adding an opacity ramp layer instruction

Actually two layer instructions will be applied. An opacity ramp layer instruction and a pass thru layer instruction. The order of the instructions is relevant.

Notice that the pass thru instruction which is applied to track 1 is the second layer instruction and the first layer instruction is the opacity instruction and is applied to track 0. The opacity instruction is the front instruction and as the opacity value reduces to 0 the pass thru layer below appears.

firstTansitionStartTime = MovieTime.make_movietime(timevalue:2, timescale:1)
track1 = MovieTrackIdentifier.make_movietrackid_from_mediatype(
                                        mediatype: :vide, trackindex: 1)
dissolveTimeRange = MovieTime.make_movie_timerange(
                                               start: firstTansitionStartTime,
                                            duration: duration)
dissolveRamp = VideoLayerInstructions.new
dissolveRamp.add_opacityramplayerinstruction(track: track0, 
                                 startopacityvalue: 1.0,
                                   endopacityvalue: 0.0,
                                         timerange: dissolveTimeRange)
dissolveRamp.add_passthrulayerinstruction(track: track1)
dissolveRampCommand = CommandModule.make_addvideoinstruction(
                                                  movieEditor,
                                       timerange: dissolveTimeRange,
                               layerinstructions: dissolveRamp)
puts JSON.pretty_generate(dissolveRampCommand.commandhash)

The generated JSON is:

{
  "command": "addmovieinstruction",
  "receiverobject": {
    "objecttype": "movieeditor",
    "objectname": "documentation.movieeditor"
  },
  "timerange": {
    "start": {
      "value": 2,
      "timescale": 1,
      "flags": 1,
      "epoch": 0
    },
    "duration": {
      "value": 12000,
      "timescale": 6000,
      "flags": 1,
      "epoch": 0
    }
  },
  "layerinstructions": [
    {
      "layerinstructiontype": "opacityramp",
      "startrampvalue": 1.0,
      "endrampvalue": 0.0,
      "track": {
        "trackindex": 0,
        "mediatype": "vide"
      },
      "timerange": {
        "start": {
          "value": 2,
          "timescale": 1,
          "flags": 1,
          "epoch": 0
        },
        "duration": {
          "value": 12000,
          "timescale": 6000,
          "flags": 1,
          "epoch": 0
        }
      }
    },
    {
      "layerinstructiontype": "passthruinstruction",
      "track": {
        "trackindex": 1,
        "mediatype": "vide"
      }
    }
  ]
}

At the end of this section I'm combining all the parts into one single script and JSON output. Here I'll be leaving out the bits where I am just doing further passthru only instructions as the only things that change are the track and the time range.

Adding a transform ramp layer instruction

Once again two layer instructions will be applied. The transform ramp instruction and a pass thru instruction. Since we are now transferring from track 1 to track 0 rather than the other way round with the opacity ramp instruction the transform ramp instruction is the first layer instruction and is applied to track 1.

secondTansStartTime = MovieTime.make_movietime(timevalue:6, timescale:1)
transformRampTimeRange = MovieTime.make_movie_timerange(
                                          start: secondTansStartTime,
                                       duration: duration)

startTransform = MITransformations.make_affinetransform() # identity.
endTransform = MITransformations.make_contexttransformation()
scaleXY = MIShapes.make_point(0.0, 1.0)
MITransformations.add_scaletransform(endTransform, scaleXY)

transformRamp = VideoLayerInstructions.new
transformRamp.add_transformramplayerinstruction(track: track1,
                            starttransformvalue: startTransform,
                              endtransformvalue: endTransform,
                                      timerange: transformRampTimeRange)
transformRamp.add_passthrulayerinstruction(track: track0)
transformRampCommand = CommandModule.make_addvideoinstruction(
                                                 movieEditor,
                                      timerange: transformRampTimeRange,
                              layerinstructions: transformRamp)
puts JSON.pretty_generate(transformRampCommand.commandhash)

The generated JSON looks like:

{
  "command": "addmovieinstruction",
  "receiverobject": {
    "objecttype": "movieeditor",
    "objectname": "documentation.movieeditor"
  },
  "timerange": {
    "start": {
      "value": 6,
      "timescale": 1,
      "flags": 1,
      "epoch": 0
    },
    "duration": {
      "value": 12000,
      "timescale": 6000,
      "flags": 1,
      "epoch": 0
    }
  },
  "layerinstructions": [
    {
      "layerinstructiontype": "transformramp",
      "startrampvalue": {
        "m11": 1.0,
        "m12": 0.0,
        "m21": 0.0,
        "m22": 1.0,
        "tX": 0.0,
        "tY": 0.0
      },
      "endrampvalue": [
        {
          "transformationtype": "scale",
          "scale": {
            "x": 0.0,
            "y": 1.0
          }
        }
      ],
      "track": {
        "trackindex": 1,
        "mediatype": "vide"
      },
      "timerange": {
        "start": {
          "value": 6,
          "timescale": 1,
          "flags": 1,
          "epoch": 0
        },
        "duration": {
          "value": 12000,
          "timescale": 6000,
          "flags": 1,
          "epoch": 0
        }
      }
    },
    {
      "layerinstructiontype": "passthruinstruction",
      "track": {
        "trackindex": 0,
        "mediatype": "vide"
      }
    }
  ]
}

The various passthru and transition layer instructions have been demonstrated in this ruby script.

The generated JSON from running that script.

The script also generates a video composition diagram image file. If for some reason your video composition doesn't render and save then looking at the video composition diagram can sometimes be helpful when working out why it doesn't work.

Adding Audio mix instructions

Since you can create sound tracks, and add audio content to those tracks, it is necessary to be able to mix the audio content of the different tracks. This is done using audio mix instructions which allows you to specify the track volume from a particular time, or set a volume ramp, where the volume level is gradually changed to the new desired level.

By default each audio track starts with the maximum volume level of 1.0. The following example sets the start volume of the second audio track to 0.0. At 1 second in, two volume ramps that last for 2 seconds are setup, the first audio track has the volume ramped down 1.0 to 0.0, whilst the second has volume ramped up from 0.0 to 1.0

theCommands = SmigCommands.new
movieEditor = SmigIDHash.make_objectid(objecttype: :movieeditor,
    objectname: 'documentation.movieeditor.audio')
audioTrack1 = MovieTrackIdentifier.make_movietrackid_from_mediatype(
    mediatype: :soun, trackindex: 0)
audioTrack2 = MovieTrackIdentifier.make_movietrackid_from_mediatype(
    mediatype: :soun, trackindex: 1)
startTime0 = MovieTime.make_movietime_fromseconds(0)

# Create the volume set instruction.
volumeInstruction = AudioInstruction.new(track: audioTrack2)
volumeInstruction.set_volume_instruction(time: startTime0, volume: 0.0)
addAudioInstruction = CommandModule.make_addaudioinstruction(movieEditor,
                                           audioinstruction: volumeInstruction)
theCommands.add_command(addAudioInstruction)

# Create the first volume ramp instruction.
rampDuration = MovieTime.make_movietime_fromseconds(2)
rampStart = MovieTime.make_movietime_fromseconds(1)
rampTimeRange = MovieTime.make_movie_timerange(start: rampStart,
                                            duration: rampDuration)
volumeRampInstruction1 = AudioInstruction.new(track: audioTrack1)
volumeRampInstruction1.set_volumeramp_instruction(timerange: rampTimeRange,
                                                startvolume: 1.0,
                                                  endvolume: 0.0)
addAudioVolumeRamp1 = CommandModule.make_addaudioinstruction(movieEditor,
                                           audioinstruction: volumeRampInstruction1)
theCommands.add_command(addAudioVolumeRamp1)

# Create the second volume ramp instruction.
volumeRampInstruction2 = AudioInstruction.new(track: audioTrack2)
volumeRampInstruction2.set_volumeramp_instruction(timerange: rampTimeRange,
                                                startvolume: 0.0,
                                                  endvolume: 1.0)
addAudioVolumeRamp2 = CommandModule.make_addaudioinstruction(movieEditor,
                                           audioinstruction: volumeRampInstruction2)
theCommands.add_command(addAudioVolumeRamp2)

The generated json looks like:

{
  "commands": [
    {
      "command": "addaudiomixinstruction",
      "receiverobject": {
        "objecttype": "movieeditor",
        "objectname": "documentation.movieeditor.audio"
      },
      "track": {
        "trackindex": 1,
        "mediatype": "soun"
      },
      "audioinstruction": "volumeinstruction",
      "time": {
        "timeinseconds": 0
      },
      "instructionvalue": 0.0
    },
    {
      "command": "addaudiomixinstruction",
      "receiverobject": {
        "objecttype": "movieeditor",
        "objectname": "documentation.movieeditor.audio"
      },
      "track": {
        "trackindex": 0,
        "mediatype": "soun"
      },
      "audioinstruction": "volumerampinstruction",
      "timerange": {
        "start": {
          "timeinseconds": 1
        },
        "duration": {
          "timeinseconds": 2
        }
      },
      "startrampvalue": 1.0,
      "endrampvalue": 0.0
    },
    {
      "command": "addaudiomixinstruction",
      "receiverobject": {
        "objecttype": "movieeditor",
        "objectname": "documentation.movieeditor.audio"
      },
      "track": {
        "trackindex": 1,
        "mediatype": "soun"
      },
      "audioinstruction": "volumerampinstruction",
      "timerange": {
        "start": {
          "timeinseconds": 1
        },
        "duration": {
          "timeinseconds": 2
        }
      },
      "startrampvalue": 0.0,
      "endrampvalue": 1.0
    }
  ]
}

Add composition diagram to the image collection

The composition diagram is an image that describes the state of the video composition when it is generated. It will show you the created tracks, the inserted segments, and the applied instructions. It can be really helpful in debugging your video compositions.

The ruby for creating the composition map diagram is:

theCommands = SmigCommands.new
imageExporter = SmigIDHash.make_objectid(objecttype: :imageexporter,
                                objectname: 'documentation.imageexporter')
imageIdentifier = SecureRandom.uuid
addCompositionImage = CommandModule.make_assignimage_tocollection(
                                            movieEditor,
                                identifier: imageIdentifier)
theCommands.add_command(addCompositionImage)
imageExporter = theCommands.make_createexporter("Dummypath.jpg",
                           pathsubstitutionkey: :compositionmapimage)
addImageToExporter = CommandModule.make_addimage_fromimagecollection(
                               imageExporter,
              imageidentifier: imageIdentifier)
theCommands.add_command(addImageToExporter)
exportCommand = CommandModule.make_export(imageExporter)
theCommands.add_command(exportCommand)
puts JSON.pretty_generate(theCommands.commandshash)

The generated JSON is:

{
  "commands": [
    {
      "command": "assignimagetocollection",
      "receiverobject": {
        "objecttype": "movieeditor",
        "objectname": "documentation.movieeditor"
      },
      "imageidentifier": "1d6f6afc-b1a2-4924-838c-0bad5d72d372"
    },
    {
      "command": "create",
      "objecttype": "imageexporter",
      "file": "Dummypath.jpg",
      "utifiletype": "public.jpeg",
      "objectname": "6bdb1d76-f941-4d70-bcea-9a1a4e84477c",
      "pathsubstitution": "compositionmapimage"
    },
    {
      "command": "addimage",
      "receiverobject": {
        "objecttype": "imageexporter",
        "objectname": "6bdb1d76-f941-4d70-bcea-9a1a4e84477c"
      },
      "imageidentifier": "1d6f6afc-b1a2-4924-838c-0bad5d72d372"
    },
    {
      "command": "export",
      "receiverobject": {
        "objecttype": "imageexporter",
        "objectname": "6bdb1d76-f941-4d70-bcea-9a1a4e84477c"
      }
    }
  ],
  "cleanupcommands": [
    {
      "command": "close",
      "receiverobject": {
        "objecttype": "imageexporter",
        "objectname": "6bdb1d76-f941-4d70-bcea-9a1a4e84477c"
      }
    }
  ]
}

Contents
© Zukini Ltd. 2015