[Bf-committers] Libraries source code

Mon Dec 14 17:24:34 CET 2020

see inline replies

On 2020-12-14 4:21 a.m., Dalai Felinto via Bf-committers wrote:
> Hi Ray,
> Thanks for your reply.
>
> The "it is scattered all over the place, and is fragile" is my personal opinion, and what motivated me to write the email.
>
> The propose to self-host was indeed from Ton. But I didn't want to add the weight of his opinion on something that could first use some clarification.
Still feels like you had a political issue and came up with some bogus reason to justify it, but let the past be the past, perhaps we should have a new years resolution of calling a spade a spade for 2021.
>
> My proposal to move forward is for the Building module to write down in the wiki the current rules for:
>
> * When is a library hosted in `//extern`

I'm not aware of any hard rules, /extern existed before I joined, what I understood over the years it's a collection of libraries that fall in one or more of the following groups:

1) Are hard to find in the average linux package repo
2) we have significant patches on
3) We are maintaining since they have no other home (anymore?)

Sergey may have a more accurate definition.

personal opinion:

I not a huge fan of /extern, it is code that seemingly seldom changes and some of it contributes to a fair chunk of the total blender build time (looking at you, libmv) however previous attempts to evict some libs from there were met with fierce resistance, so it is what is, not a hornets nest I'm particularly looking forward to kicking.

> * When is a library maintained in `svn lib`
logically following the last question: when it's not in /extern and we still need it to build blender
> * When should the libraries be updated
I'm not aware of any hard rules there either, however personal opinion :

Ideal situation: Every lib is owned by one or more of the module teams, they ask for updates in bcon1,  occasionally newer versions are required for platform support, this can be discussed with the module(s) that own the dependency.

Actual situation: essentially no-one cares, i did a complete lib version bump a few versions ago, and i will literally never do it again, rarely have i seen that much indecisive apathy on something.  The question wasn't even hard, "hey your module depends on lib X,  would you like an update?" they didn't even have to do the work!

> * The differences between `make deps` and `install_deps.sh` and why we maintain both

As per my last email, "make deps" is for the platform devs to maintain the SVN libs, target audience +- 3 people, it covers all platforms we support and only those.

`install_deps.sh` is a linux only script that will get the libs from a distro's repo is possible and build from source other wise, this used to be the way the linux developers got the libs before we had SVN libs for that platform, we still maintain it since it has a passionate maintainer willing to do the work.

> * The current reasoning to not self-host the svn libraries sources
There has never been any reason to, In the last 5 yeas I maintained this script the "it's a bit fragile" has not once caused any issues, so I question the validity of that argument, I'm not sure we need to document the reasoning for not doing it any more than we need to document why not every developer on the animations team owns a cat named sparkles. They are just question that don't come up a whole lot.
>
> Having this clear would have also helped the recent "VFX Reference Platform" discussion.

I do not see how any of this discussion has any merit on "do we follow the VFX Reference Platform versions or not" given that once more is a political discussion, not a technical or organizational one. The choice to stay on python 3.7.x or update to 3.9.x is not depending on if the pyhon libs live in svn or /extern , why install_deps.sh exists or the reasoning why we don't keep the source of deps in svn, even who gets to ask for updates is irrelevant if we politically decide to stick with 3.7.x.

This is not a discussion for the platform team, this is between the python team and whomever makes the political calls, we (the platform team) have no horse in this race, building python 3.7.x is about as much work as building python 3.9.x, it really doesn't matter from a platform perspective.

> I can gladly help out with the writing if no else from the "Platforms, Builds & Tests" module can pick that up.

I don't think the lack of documentation is due to lack of people wanting to document it, but as you probably noticed by now getting a decisions out of people on these matters is like herding cats.

--Ray

> On 13-12-2020 18:49, Ray Molenkamp via Bf-committers wrote:
>> Seems like the reason has moved from "it's scattered all
>> over the place, that's a bit fragile" (technical reason,
>> which I will happily share/defend my views on) to
>> "because I want it for political reasons" (where not a
>> single technical argument will change your mind)
>>
>> In the future it's probably best to be upfront where a
>> desire comes from rather than having it masquerade  as a
>> technical issue and hope no-one calls you on it.
>>
>> --Ray
>>
>> On 2020-12-13 9:29 a.m., Ton Roosendaal via Bf-committers wrote:
>>> Hi,
>>>
>>> The reason is to protect software freedom in general. I don't like it that for building Blender you are forced to use commercial sites offering code. It would be different if we use established GNU approved platforms.
>>>
>>> https://www.gnu.org/software/repo-criteria-evaluation.html
>>>
>>> https://www.gnu.org/software/repo-criteria.en.html
>>>
>>> I would find it really a positive statement if we copy all external bundles to blender.org and build from there.
>>>
>>> Nothing urgent though, it's politics :)
>>>
>>> -Ton-
>>> ----------------------------------------------------------------------
>>> Ton Roosendaal - ton at blender.org - www.blender.org
>>> Chairman Blender Foundation, Director Blender Institute
>>> Buikslotermeerplein 161, 1025 ET Amsterdam, the Netherlands
>>>
>>>
>>> On 10/12/2020 16:02, Ray Molenkamp via Bf-committers wrote:
>>>> I'm unsure what this would achieve beyond making the lib update process more frustrating than it already is?
>>>>
>>>> The deps builder we have its singe purpose is to facilitate the building of our SVN libs nothing more nothing less, its target audience is essentially 3 people (the mac/linux/windows platform maintainers) we share the script with the world since that's the spirit of opensource, but we offer very little (if any) support on it. Developers are advised to use the SVN libs and most distro's have their own build infrastructure for dependencies already. If you want to build all deps using our script on your own, good on you, we certainly won't stop you, but the script is aimed at a very narrow build environment (ours) with a very narrow use-case (our svn libs) it *cannot* be and *will not* be the end all and be all build script for all possible environments and all possible distributions.
>>>>
>>>> Having the source to all deps on our server would bring very little (actually just an extra burden) to the party, keeping that context in mind, what is the problem you are trying to solve?
>>>>
>>>> --Ray
>>>> On 2020-12-09 8:14 a.m., Dalai Felinto via Bf-committers wrote:
>>>>> Hi,
>>>>>
>>>>> At the moment the source code to build the libraries required by Blender is scattered everywhere:
>>>>>
>>>>> * github
>>>>> * sourceforge
>>>>> * own projects sites
>>>>> * archived pages on the web (e.g., http.debian.net for the bzip)
>>>>>
>>>>> For the complete list see: `build_files/build_environment/cmake/versions.cmake`
>>>>>
>>>>>
>>>>> Is there a reason for Blender to not host a copy of the compressed source files? Given that we depend on almost 40 different libraries, it seems a bit fragile to count on them be online forever.
>>>>>
>>>>>
>>>>> The zip/tar.gz, ... packages could be stored in: https://svn.blender.org/svnroot/bf-blender/trunk/lib/source
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Dalai-
>>>>> --------------------------------------------------------------------
>>>>> Dalai Felinto - dalai at blender.org - www.blender.org
>>>>> Blender Development Coordinator
>>>>> Buikslotermeerplein 161, 1025 ET Amsterdam, the Netherlands
>>>>>
>>>>> _______________________________________________
>>>>> Bf-committers mailing list
>>>>> Bf-committers at blender.org
>>>>> https://lists.blender.org/mailman/listinfo/bf-committers
>>>> _______________________________________________
>>>> Bf-committers mailing list
>>>> Bf-committers at blender.org
>>>> https://lists.blender.org/mailman/listinfo/bf-committers
>>> _______________________________________________
>>> Bf-committers mailing list
>>> Bf-committers at blender.org
>>> https://lists.blender.org/mailman/listinfo/bf-committers
>> _______________________________________________
>> Bf-committers mailing list
>> Bf-committers at blender.org
>> https://lists.blender.org/mailman/listinfo/bf-committers
>