[Bf-committers] Blender dependencies scripts in main repo

Brecht Van Lommel brechtvanlommel at pandora.be
Thu Jul 27 18:20:54 CEST 2017 

Here's an initial patch that moves the code into the main Blender repo with
some minor cleanups, which I'd like to get committed as a starting point.
https://developer.blender.org/D2753

With blender-dependencies we get a versions.cmake that has all the library
versions and download links. This could be moved into another file format
to be read by other scripts, not sure which file format would be best.

For integration with install_deps.sh, that does risk getting quite
complicated if we have to take into account distro paths, incompatible
versions or (c++11) build flags, .. the kind of issues I would like to
sidestep with building everything from source. I'll check if there's a good
way to do it, it's not clear to me currently how to integrate these bash
and cmake scripts without making a mess.

It's possible to rebuild individual dependencies, not currently integrated
into GNUmakefile but by going into ../build_linux/deps and typing "make
help" or "make external_opensubdiv".


On Tue, Jul 25, 2017 at 2:26 PM, Sergey Sharybin <sergey.vfx at gmail.com>
wrote:

> Perhaps what we really need is some dedicated file which lists exact
> library versions we need for a proper Blender release, which is used by
> whatever build/install/testing system is used on various platforms. This
> way:
>
> 1. We will have version of libraries stored in Git. Not sure history is
> really important, rather than just knowing latest versions needed, but
> having that doesn't harm either.
> 2. All the platform support / compile scripts will always be in sync.
> 3. It will be possible to have a test of release build which will compare
> versions from that file with what system-info reports.
> 4. It will be simple to verify our Wiki page is up to date.
>
> This does NOT cancel the fact that when library version update is needed
> there should be corresponding mail done to this list.
>
> As for replacement of install_deps.sh. What we can do is to remove all
> compile logic from it, make it responsible for only installing packages
> from main repository. And then invoke your scripts to compile libraries for
> the libraries which can not be installed from main repository. This way
> we'll de-duplicate all compilation logic across all platforms, but will not
> cause changes in how things behaves for users. At least not at this moment.
> Optionally, there might be an option to ignore packages install logic from
> install_deps and force everything to be compiled.
>
> The `WITH_*` logic i don't like. Ideally it should be `make opensubdiv` to
> just compile library i currently want to be compiled. Maybe this comes "for
> free" anyway. What i mean here, it's kind of nice/important to have ability
> to compile individual library.
>
>
> On Tue, Jul 25, 2017 at 2:09 PM, Brecht Van Lommel <
> brechtvanlommel at pandora.be> wrote:
>
> > On Tue, Jul 25, 2017 at 11:38 AM, Sergey Sharybin <sergey.vfx at gmail.com>
> > wrote:
> >
> > > Ideally you need to tell the builder script which library you want to
> > > (re)build. Exposing all this to Blender's makefile/make.bat could cause
> > too
> > > much noise in Blender's sources.
> >
> >
> > It could use the WITH_* flags to determine which libraries need to be
> > built. For rebuilds, blender-dependencies automatically figures out which
> > libraries changed and rebuilds only those and their dependencies.
> >
> >
> > > We do not _REQUIRE_ latest versions of ALL dependencies. The policy is
> > > still: 10 years old hardware, 5 years old software. I am not aware of
> > > anyone cancelling this policy. Surely, sometimes we break this rule,
> but
> > > there usually a good reason to. Python is kind of exception here, but
> it
> > > doesn't mean it makes me happy either.
> >
> >
> >
> > As for time: i wouldn't want to wait for Boost and LLVM to be compiled on
> > > my laptop. This will take HOURS, while apt-get install will fetch those
> > in
> > > few moments (together with OIIO,. OCIO, FFmpeg and lots of more
> > packages).
> >
> >
> > We may not strictly require them, but install_deps.sh builds a lot of
> > libraries anyway. The c++11 switch also would have been easier.
> >
> > Every day there are users in #blendercoders with install_deps.sh build
> > errors, I suspect mostly due to conflicts in versions with system
> packages.
> > That also leads to many hours wasted time. Building all libraries from
> > scratch on my dual core mac takes 55min.
> >
> > Anyway, I don't want to push for a replacement for install_deps.sh if
> > others don't care about it.
> >
> >
> > > It is always difficult to build those if you want same level of
> > staticness
> > > than official releases.
> > >
> > > For a proper static build you need to build extra libraries, make sure
> > they
> > > are compiled with -fPIC, and patch some build scripts from various
> > packages
> > > to force their dependencies to be linked against static libraries.
> >
> >
> > I understand, we do a lot of that stuff in blender-dependencies. Having
> all
> > those things in source code is helpful I think, even if only the platform
> > maintainer runs it. You've been maintaining Linux for a long time, but
> for
> > other platforms there's been a lot changes and each time it's been a pain
> > to figure things out again. Having the history of changes in a git repo
> is
> > more helpful than a Docker image I think.
> >
> > I don't see why submitting a patch to differencial is important. It is
> much
> > > more important to mail this mailing list when you need something. This
> is
> > > the only reliable way to guarantee all platform maintainers and
> > maintainers
> > > of our build instructions on Wiki will pick things up and  sign under
> > that
> > > they made a build.
> > >
> >
> > Of course there is still coordination between maintainers needed, but why
> > not have it part of the main repo, what is the downside? We already have
> > install_deps.sh there.
> >
> > It's just easier if it works with differential, as a single branch
> > containing both the library update and corresponding code changes on the
> > Blender side, if there's a commit that can be linked to a bug report and
> > found with git log, if I can rebuild an old version without having to
> > manually match repository versions, etc.
> >
> > If we think "dependencies" is more appropriate name, we should rename
> > > existing "build_environment". We definitely should not make yet another
> > > obscure folder with similar sounding name.
> > >
> >
> > Moving install_deps.sh to a different location is not worth it, so might
> as
> > well use build_environment then.
> >
> >
> >
> >
> > > > On Tue, Jul 25, 2017 at 8:26 AM, Sergey Sharybin <
> sergey.vfx at gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > You can't easily replace install_deps.sh easily with those CMake
> > based
> > > > > scripts. It is a requirement to use as much as possible packages
> from
> > > > main
> > > > > repository.
> > > > >
> > > > > As for where to put the files, i don't see a reason to make it
> > > additional
> > > > > folder in build_files, this is what build_environment folder there
> > was
> > > > > supposed to be used for.
> > > > >
> > > > > But then, i don't think regular users should ever bother with those
> > > > > precompiled libraries. From this point of view, adding those
> scripts
> > > to a
> > > > > blender-dev-tools repo (which is a submodule to source/tools) seems
> > > more
> > > > > reasonable to me.
> > > > >
> > > > > On Tue, Jul 25, 2017 at 4:17 AM, Ray Molenkamp <ray at lazydodo.com>
> > > wrote:
> > > > >
> > > > > > I have no objections, will this replace install_deps.sh? I have
> > > > suggested
> > > > > > merging all platforms
> > > > > > before but got push back due to install_deps wanting to use as
> much
> > > > > > apt-get as possible to
> > > > > > keep the time needed to fetch the dependencies down.
> > > > > >
> > > > > > I'm not sure making it easily available though make has my
> > > > preference,  I
> > > > > > can support other
> > > > > > platform maintainers building libs on windows, but I would lack
> the
> > > > time
> > > > > > to do this for a wider
> > > > > > user base.
> > > > > >
> > > > > > --Ray
> > > > > >
> > > > > > On 7/24/2017 5:59 PM, Brecht Van Lommel wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I would like to get the scripts used to build library
> > dependencies
> > > on
> > > > > > Windows and macOS committed to the main Blender repository.
> > Currently
> > > > > these
> > > > > > scripts are on GitHub:
> > > > > > > https://github.com/LazyDodo/blender_dependencies
> > > > > > > https://github.com/brechtvl/blender_dependencies/tree/apple
> > > > > > >
> > > > > > > They could go into build_files/dependencies, and be invoked
> > through
> > > > > > "make dependencies".
> > > > > > >
> > > > > > > There are some code style differences, and various cleanups
> could
> > > be
> > > > > > done as well. But I would like to know if others agree this is
> the
> > > > right
> > > > > > way to go, before spending time on making a branch / code review.
> > Any
> > > > > > objections/concerns about this?
> > > > > > >
> > > > > > > I've also ported the scripts to Linux, which wasn't a big step
> > from
> > > > > > macOS. I'd love to have a unified system for all platforms to
> > > generate
> > > > a
> > > > > > static release binary from scratch, but that's a more complicated
> > > > > > discussion.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Brecht.
> > > > > >
> > > > > > _______________________________________________
> > > > > > Bf-committers mailing list
> > > > > > Bf-committers at blender.org
> > > > > > https://lists.blender.org/mailman/listinfo/bf-committers
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > With best regards, Sergey Sharybin
> > > > > _______________________________________________
> > > > > Bf-committers mailing list
> > > > > Bf-committers at blender.org
> > > > > https://lists.blender.org/mailman/listinfo/bf-committers
> > > > >
> > > > _______________________________________________
> > > > Bf-committers mailing list
> > > > Bf-committers at blender.org
> > > > https://lists.blender.org/mailman/listinfo/bf-committers
> > > >
> > >
> > >
> > >
> > > --
> > > With best regards, Sergey Sharybin
> > > _______________________________________________
> > > Bf-committers mailing list
> > > Bf-committers at blender.org
> > > https://lists.blender.org/mailman/listinfo/bf-committers
> > >
> > _______________________________________________
> > Bf-committers mailing list
> > Bf-committers at blender.org
> > https://lists.blender.org/mailman/listinfo/bf-committers
> >
>
>
>
> --
> With best regards, Sergey Sharybin
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> https://lists.blender.org/mailman/listinfo/bf-committers
>

More information about the Bf-committers mailing list