[Bf-committers] New VSE proxies, feedback appreciated

[Brecht Van Lommel]

Hi Peter,

I took a look at the patch, it's quite a lot of code, reviewing all
the indexer code is a bit much, but I'll give some high level
comments.

Am I right that if you don't enable proxies, this patch basically has
no effect? In that case it should be pretty safe to commit before
2.58, but still think this could use some good testing on complex
files.

* Job name in header is "Seq Proxy", maybe name it "Building Proxies"
or something like that?
* On running rebuild proxy, it does not redraw the sequencer header
immediately, makes it unclear if the job has started.
* Do we need the option to disable building time codes for proxies?
All 3 are enabled by default, and it seems like there isn't much
reason not to build them, since you can still afterwards decide to use
them or not.

* Timecode enum: the items here could use descriptions. Also there is
no need to repeat "TC" in the item names, and generally abbreviations
like that should be avoided in the UI. Would just go with "Record
Run", "Free Run", ..
* If timecode options other than Record Run are not supported, I guess
they should not be exposed in the UI?
* These timecode indexes are not used by default, is there a reason for this?
* One thing I don't understand about this is how it can be a proxy
level setting. Doesn't this also affect non-proxy renders and the
length of the strip in the sequencer?

* What happens when you remove a strip while the proxy for it is being
built? Didn't find any checks for that.

Brecht.

On Sun, May 29, 2011 at 3:01 PM, Peter Schlaile <peter at schlaile.de> wrote:
> Hi,
>
> I'm currently coding a new Video Sequencer Proxy and indexing system on
> gitorious:
>
> https://gitorious.org/~schlaile/blenderprojects/blender-vse-schlaile
>
> vse-proxies branch
>
> Usage:
>
>    git clone git://gitorious.org/~schlaile/blenderprojects/blender-vse-schlaile.git
>    cd blender-vse-schlaile/blender
>    git checkout vse-proxies
>
>    scons / cmake as usual.
>
> Before I do a merge into trunk, I'd like to have some feedback, if the
> code works like expected.
>
> The problems:
> * VSE proxies are still missing from Blender 2.5
> * FFMPEG seeking heuristics works currently "most of the time" at best,
>   sometimes it breaks down completely (depending on the mood of ffmpeg
>   internals)
>
> Those are not "a let's hack on the interface and finally we figure out
> a way to do it" kind of problems. There are fundamental ones:
>
> The heuristics are DTS based (in case you don't know,
> non-intra-formats code a "decoding timestamp" and a "presentation
> timestamp" (PTS) into every data packet. Decoding timestamps are
> (most of the time) monotonous, so a very good candidate for binary
> searches, but: they have very little in common with the frame, we are
> actually looking for, since that is determined by the PTS.
>
> Fundamental flaws:
> * there are (commonly used!) formats, like HDV, where the DTS
>   doesn't have to be monotonous. So: seeking can (and will!) break down
>   completely.
> * the DTS is only a rough idea where the frame, we are actually
>   looking for is probably located. So: we are working with rough
>   preseeks in the hope, the right PTS will appear (which probably
>   will never happen)
> * track duration is a wild guess, since ffmpeg uses the average frame
>   and bitrate of the beginning of the file, to deduce the right
>   duration. That can (and will) fail completely on certain files for
>   obvious reasons.
> * it's rather tricky to determine the right offset between DTS and
>   PTS, since start offsets are specified for DTS and the PTS has
>   an unspecified time offset (which could be detected by decoding
>   the first view frames): so, even if we got the right DTS and even
>   if we are at the right place within the file, we are probably looking
>   for the wrong PTS from the start...
> * sometimes, the DTS/PTS pair written into the stream, isn't the
>   timecode we were looking for in the first place. Camcorders
>   can write seperate timecode tracks (and do so, even the most
>   cheep ones). Sometimes, you can only deduce from the timecode
>   track, that some dropouts happened. And: if you work with
>   muliple cams, and your cam operators tend to start/stop a lot,
>   you will go crazy syncing the timeline without timecode support.
>
> The idea:
> * let's optionally build seperate timecode indices, which specify the
>   exact position (absolute file position, DTS to seek to, PTS to search
>   for) for every possible frame in the file.
> * let's build that index together with the proxies, since for them, we
>   have to scan and decode the file anyways. And: a proxy only makes sense
>   within the context of an index. Another good reason, why those should
>   be build in the same pass. (Besides the fact, that we can do it a heck
>   lot faster, when only using ffmpeg functions, since color conversion
>   steps can be optimized away.)
>
> The implementation:
> * currently, proxy building (in background) and "Record run" indices are
>   implemented, other time code indices following later.
>
>   To use them: build the branch (see above), open a movie file, go to
>   N-keys panel "Proxies", check "Use Proxy", by default 25% proxies are
>   the only one build, but you can change that (or build no proxy at all
>   and use timecode indices only).
>
>   Do that for as much movie clips as you like.
>
>   Select all of them.
>
>   Go to "Strip -> Rebuild proxy"
>
>   On the status bar, there should be a progress bar (which currently shows
>   per file progress, total progress is TODO).
>
>   Blender is building proxies in the background, so you can continue
>   working.
>
>   After the build process finishes, you can select different time codes
>   (currently only "record run" is implemented)
>   and use the Proxy preview (N-keys dialog in preview window) again.
>
> * seeking works perfectly for the first time in blender (no stuttering,
>   failing completely on broken files, etc.) since with indices, the
>   system now works completely PTS based. And: the preseek heuristics isn't
>   needed anymore, so blender will always seek to the right I-Frame, which
>   means: faster feedback on CPU-heavy footage (like Sony EX3) e.g.!
>
> * currently missing:
>   * other timecode tracks than "record run".
>     In case you don't know: "record run" means: use the frames as they
>     are recorded, ignore gaps, if someone hits the stop button on the
>     camcorder.
>
>     "free run" means: use a global time counter (configurable on the
>     camcorder), if someone hits stop, consider those frames, and display
>     them as black gaps, frozen pictures.
>
>     "free run (interpolated)" means: if your camcorder is too cheap,
>     use record time/date to simulate a real "free run"-mode, usually only
>     found in prosumer camcorders.
>
>   * proper variable frame rate support (could be considered as some sort
>     of "free run", since the idea has a lot of similarities.
>
>   * audio code doesn't use the same seeking code path, leading to A/V sync
>     problems.
>
> If things don't work out for you, using blender with "blender -d" will
> print a lot of debugging output within the seeking code.
>
> Please send me an email with the debugging output pasteall-ed, your
> ffmpeg version, if you want to report problems.
>
> Otherwise: enjoy!
>
> And: I'm eager to hear your feedback, also code / design suggestions.
>
> Cheers,
> Peter
>
> ----
> Peter Schlaile
>
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>