[Bf-committers] Blender dependencies scripts in main repo

Tue Jul 25 11:38:23 CEST 2017

Hi,

Answers are inlined.

On Tue, Jul 25, 2017 at 11:15 AM, Brecht Van Lommel <
brechtvanlommel at pandora.be> wrote:

> I don't think we want users to build their own libraries on Windows and
> macOS, so we wouldn't advertise this in the build docs. The option can be
> made a bit more hidden if needed.
>

Ideally you need to tell the builder script which library you want to
(re)build. Exposing all this to Blender's makefile/make.bat could cause too
much noise in Blender's sources.

>
> For install_deps.sh, in my personal opinion reusing as many packages from
> the system as possible is rather pointless. Since we depend on recent
> versions of a lot of packages we already require many to be built, and then
> reusing some other system libraries doesn't save much time.

We do not _REQUIRE_ latest versions of ALL dependencies. The policy is
still: 10 years old hardware, 5 years old software. I am not aware of
anyone cancelling this policy. Surely, sometimes we break this rule, but
there usually a good reason to. Python is kind of exception here, but it
doesn't mean it makes me happy either.

>From the studio experience, there is not that much packages which really
needs to be compiled from sources, on both older and ewer system installs.

As for time: i wouldn't want to wait for Boost and LLVM to be compiled on
my laptop. This will take HOURS, while apt-get install will fetch those in
few moments (together with OIIO,. OCIO, FFmpeg and lots of more packages).

> Building
> everything from source is less sensitive to distribution specific issues,
> which users seems to encounter quite often. It's also really difficult now
> to make static builds.
>

It is always difficult to build those if you want same level of staticness
than official releases.

For a proper static build you need to build extra libraries, make sure they
are compiled with -fPIC, and patch some build scripts from various packages
to force their dependencies to be linked against static libraries. The
extra libraries includes:

- bz2
- gpm
- libcrypt
- libnls
- lzma
- ncurses
- readline
- sqlite
- ssl
- tbb
- zlib

Most of the libraries are needed for python, and requirement for those
libraries to be static is granted by bug reports in the tracker about
failure of imports on various systems. libcrypt and libnls are coming from
libc, which means you need to wait whole libc to compile to get those
libraries.

Even putting compile time issues aside, those are the extra libraries to
maintain building, extra points of failure, and as a result: extra time
wasted on trying to support someone who've got compilation error on those
packages.

I do not consider requirement of proper static builds is something
important/doable for install_deps.sh. Stripping all possible sensitive
information from our release builder chroots and wrapping them into docker
seems much more sensitive and doable to me for those who needs a proper
release environment.

> Anyway, my intention is not to replace install_deps.sh if others are not
> interested in that. Maybe this could still replace what you are using to
> make libraries for releases / buildbot, not sure how that is done now.
>

I wouldn't want random people to make libraries used by releases. We've got
platform maintainers for that. Those are handful of people, who can totally
live with those scripts just to be moved to blender-dev-tools.

> I would like to have the files in the main Blender repo so that when adding
> or upgrading a library, a single commit documents all the changes, and can
> be submitted to differential. It also means we can reuse existing cmake
> code, which is more painful if we use a submodule.
>

I don't see why submitting a patch to differencial is important. It is much
more important to mail this mailing list when you need something. This is
the only reliable way to guarantee all platform maintainers and maintainers
of our build instructions on Wiki will pick things up and  sign under that
they made a build.

>
> We could put it in the build_environment folder. I think "dependencies" is
> a bit more clear about what it is since this isn't about a chroot anymore,
> but don't care too much.
>

If we think "dependencies" is more appropriate name, we should rename
existing "build_environment". We definitely should not make yet another
obscure folder with similar sounding name.

> On Tue, Jul 25, 2017 at 8:26 AM, Sergey Sharybin <sergey.vfx at gmail.com>
> wrote:
>
> > Hi,
> >
> > You can't easily replace install_deps.sh easily with those CMake based
> > scripts. It is a requirement to use as much as possible packages from
> main
> > repository.
> >
> > As for where to put the files, i don't see a reason to make it additional
> > folder in build_files, this is what build_environment folder there was
> > supposed to be used for.
> >
> > But then, i don't think regular users should ever bother with those
> > precompiled libraries. From this point of view, adding those scripts to a
> > blender-dev-tools repo (which is a submodule to source/tools) seems more
> > reasonable to me.
> >
> > On Tue, Jul 25, 2017 at 4:17 AM, Ray Molenkamp <ray at lazydodo.com> wrote:
> >
> > > I have no objections, will this replace install_deps.sh? I have
> suggested
> > > merging all platforms
> > > before but got push back due to install_deps wanting to use as much
> > > apt-get as possible to
> > > keep the time needed to fetch the dependencies down.
> > >
> > > I'm not sure making it easily available though make has my
> preference,  I
> > > can support other
> > > platform maintainers building libs on windows, but I would lack the
> time
> > > to do this for a wider
> > > user base.
> > >
> > > --Ray
> > >
> > > On 7/24/2017 5:59 PM, Brecht Van Lommel wrote:
> > > > Hi all,
> > > >
> > > > I would like to get the scripts used to build library dependencies on
> > > Windows and macOS committed to the main Blender repository. Currently
> > these
> > > scripts are on GitHub:
> > > > https://github.com/LazyDodo/blender_dependencies
> > > > https://github.com/brechtvl/blender_dependencies/tree/apple
> > > >
> > > > They could go into build_files/dependencies, and be invoked through
> > > "make dependencies".
> > > >
> > > > There are some code style differences, and various cleanups could be
> > > done as well. But I would like to know if others agree this is the
> right
> > > way to go, before spending time on making a branch / code review. Any
> > > objections/concerns about this?
> > > >
> > > > I've also ported the scripts to Linux, which wasn't a big step from
> > > macOS. I'd love to have a unified system for all platforms to generate
> a
> > > static release binary from scratch, but that's a more complicated
> > > discussion.
> > > >
> > > > Thanks,
> > > > Brecht.
> > >
> > > _______________________________________________
> > > Bf-committers mailing list
> > > Bf-committers at blender.org
> > > https://lists.blender.org/mailman/listinfo/bf-committers
> > >
> >
> >
> >
> > --
> > With best regards, Sergey Sharybin
> > _______________________________________________
> > Bf-committers mailing list
> > Bf-committers at blender.org
> > https://lists.blender.org/mailman/listinfo/bf-committers
> >
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> https://lists.blender.org/mailman/listinfo/bf-committers
>

-- 
With best regards, Sergey Sharybin