[Bf-committers] New VSE proxies, feedback appreciated

Peter Schlaile peter at schlaile.de
Sun May 29 15:01:26 CEST 2011


Hi,

I'm currently coding a new Video Sequencer Proxy and indexing system on
gitorious:

https://gitorious.org/~schlaile/blenderprojects/blender-vse-schlaile

vse-proxies branch

Usage:

    git clone git://gitorious.org/~schlaile/blenderprojects/blender-vse-schlaile.git
    cd blender-vse-schlaile/blender
    git checkout vse-proxies

    scons / cmake as usual.

Before I do a merge into trunk, I'd like to have some feedback, if the 
code works like expected.

The problems:
* VSE proxies are still missing from Blender 2.5
* FFMPEG seeking heuristics works currently "most of the time" at best,
   sometimes it breaks down completely (depending on the mood of ffmpeg
   internals)

Those are not "a let's hack on the interface and finally we figure out
a way to do it" kind of problems. There are fundamental ones:

The heuristics are DTS based (in case you don't know, 
non-intra-formats code a "decoding timestamp" and a "presentation 
timestamp" (PTS) into every data packet. Decoding timestamps are 
(most of the time) monotonous, so a very good candidate for binary 
searches, but: they have very little in common with the frame, we are 
actually looking for, since that is determined by the PTS.

Fundamental flaws:
* there are (commonly used!) formats, like HDV, where the DTS
   doesn't have to be monotonous. So: seeking can (and will!) break down
   completely.
* the DTS is only a rough idea where the frame, we are actually
   looking for is probably located. So: we are working with rough
   preseeks in the hope, the right PTS will appear (which probably
   will never happen)
* track duration is a wild guess, since ffmpeg uses the average frame
   and bitrate of the beginning of the file, to deduce the right
   duration. That can (and will) fail completely on certain files for
   obvious reasons.
* it's rather tricky to determine the right offset between DTS and
   PTS, since start offsets are specified for DTS and the PTS has
   an unspecified time offset (which could be detected by decoding
   the first view frames): so, even if we got the right DTS and even
   if we are at the right place within the file, we are probably looking
   for the wrong PTS from the start...
* sometimes, the DTS/PTS pair written into the stream, isn't the
   timecode we were looking for in the first place. Camcorders
   can write seperate timecode tracks (and do so, even the most
   cheep ones). Sometimes, you can only deduce from the timecode
   track, that some dropouts happened. And: if you work with
   muliple cams, and your cam operators tend to start/stop a lot,
   you will go crazy syncing the timeline without timecode support.

The idea:
* let's optionally build seperate timecode indices, which specify the
   exact position (absolute file position, DTS to seek to, PTS to search
   for) for every possible frame in the file.
* let's build that index together with the proxies, since for them, we
   have to scan and decode the file anyways. And: a proxy only makes sense
   within the context of an index. Another good reason, why those should
   be build in the same pass. (Besides the fact, that we can do it a heck
   lot faster, when only using ffmpeg functions, since color conversion
   steps can be optimized away.)

The implementation:
* currently, proxy building (in background) and "Record run" indices are
   implemented, other time code indices following later.

   To use them: build the branch (see above), open a movie file, go to
   N-keys panel "Proxies", check "Use Proxy", by default 25% proxies are
   the only one build, but you can change that (or build no proxy at all
   and use timecode indices only).

   Do that for as much movie clips as you like.

   Select all of them.

   Go to "Strip -> Rebuild proxy"

   On the status bar, there should be a progress bar (which currently shows
   per file progress, total progress is TODO).

   Blender is building proxies in the background, so you can continue
   working.

   After the build process finishes, you can select different time codes
   (currently only "record run" is implemented)
   and use the Proxy preview (N-keys dialog in preview window) again.

* seeking works perfectly for the first time in blender (no stuttering,
   failing completely on broken files, etc.) since with indices, the
   system now works completely PTS based. And: the preseek heuristics isn't
   needed anymore, so blender will always seek to the right I-Frame, which
   means: faster feedback on CPU-heavy footage (like Sony EX3) e.g.!

* currently missing:
   * other timecode tracks than "record run".
     In case you don't know: "record run" means: use the frames as they
     are recorded, ignore gaps, if someone hits the stop button on the
     camcorder.

     "free run" means: use a global time counter (configurable on the
     camcorder), if someone hits stop, consider those frames, and display
     them as black gaps, frozen pictures.

     "free run (interpolated)" means: if your camcorder is too cheap,
     use record time/date to simulate a real "free run"-mode, usually only
     found in prosumer camcorders.

   * proper variable frame rate support (could be considered as some sort
     of "free run", since the idea has a lot of similarities.

   * audio code doesn't use the same seeking code path, leading to A/V sync
     problems.

If things don't work out for you, using blender with "blender -d" will 
print a lot of debugging output within the seeking code.

Please send me an email with the debugging output pasteall-ed, your 
ffmpeg version, if you want to report problems.

Otherwise: enjoy!

And: I'm eager to hear your feedback, also code / design suggestions.

Cheers,
Peter

----
Peter Schlaile



More information about the Bf-committers mailing list