[Bf-committers] move prebuilt libs from svn to git?
Ankit
ankitjmeel at gmail.com
Sat Jun 18 09:51:18 CEST 2022
Zack
- To track library versions https://developer.blender.org/T88438
- To tag corresponding versions for each release
https://svn.blender.org/svnroot/bf-blender/tags/
- Some bugs show up due to deps also,
https://developer.blender.org/rBL62677 . VCS can point to the commit
that caused it.
Ankit
On 18/06/22 12:34, Zack Brown via Bf-committers wrote:
> Hi everyone,
>
> It's fascinating to see this discussion. It never occurred to me that
> anyone might continue to prefer svn once git came into existence, and yet
> here is a sample case, right under my nose! I guess svn is still relevant
> even after all these years!
>
> But actually I'd be curious to know why revision control is used at all in
> this particular case. We're talking about precompiled binaries, right?
> That's sort of the poster child for situations where revision control is
> *not* what one wants, exactly for the reason why git would be a bad choice.
>
> Please, could someone explain why the blender project doesn't simply use a
> dev-accessible shared folder somewhere? I love revision control, but it's
> only useful if it's useful.
>
> Be well,
> Zack
>
>
> On Sat, Jun 18, 2022 at 1:10 AM Ray Molenkamp via Bf-committers <
> bf-committers at blender.org> wrote:
>
>> Just an update with some new numbers, dan's changes
>> brought down the time for a full clone from my home
>> connection down to 5m48s (down from 27m39s previously)
>>
>> well done dan!
>>
>> --Ray
>>
>> On 2022-06-17 3:45 p.m., Dan McGrath wrote:
>>> Just a heads-up that Ray and I took a packet capture on both ends for me.
>>>
>>> Thanks to his capture, I noticed that the SVN server host was not
>> sending TCP window scale or SACK options. Ultimately this turned out to be
>> due to that host using the PF firewall "synproxy" state option, which
>> apparently was causing the TCP header options to be removed, which limited
>> the window size to 64k.
>>>
>>> I have corrected this change to match the www host, and we did observe
>> some improvements that were hidden from me before due to my and the studio
>> IPs special treatment that bypassed this feature.
>>>
>>> While it should help, I also tuned a few sysctl settings for send and
>> receive buffers. In addition, I noticed that the SVN server is also not
>> sending gzip/deflate accept headers for the files. No doubt this is going
>> to be a factor. However, there was a note from 2016 from myself explaining
>> that Brecht asked for the SVN compression to be disabled due to some client
>> problems. So, this will require some research, perhaps next week.
>>>
>>> Anyway, hopefully it helps a bit until I can review some more settings.
>>>
>>> Enjoy your weekend everyone!
>>>
>>>
>>> Dan
>>>
>>> On Fri, Jun 17, 2022, 2:22 PM Ray Molenkamp via Bf-committers <
>> bf-committers at blender.org> wrote:
>>>
>>> =8<==[Editors note]=8<==
>>> I'll be honest I'm 100% convinced mailinator will ruin
>>> the little formatting I have done, I put a copy of this
>>> email on https://developer.blender.org/P3014.
>>> =8<====8<====8<====8<==
>>>
>>> How much as I like the sentiment of "lets move to git will it
>>> solve all these problems" lets be honest here, git.blender.org <
>> http://git.blender.org>'s
>>> speed is nothing to write home about either it may or may not be
>>> as glitchy as svn, but it still wouldn't be fast.
>>>
>>> from my 300mbit connection in western Canada
>>>
>>> Receiving objects: 6% (135931/2078159), 33.34 MiB | 458.00 KiB/s
>>>
>>> total time for a clone off git.blender.org <http://git.blender.org>
>> 27m39s
>>>
>>> Cloning off https://github.com/blender/blender.git however
>>>
>>> total time for a clone off github.com <http://github.com> 1m 52s
>>>
>>> Now amount of finger pointing to client settings will convince me
>>> it's a client setting at this point, but I'll be honest, I am all
>>> the way in western Canada and that could definitely be a factor.
>>>
>>> sooooo, lets investigate!
>>>
>>> Let’s spin up an AWS EC2 instance in London, I'd say that be close
>>> enough to Amsterdam, and let’s rule out any other CPU or Bandwidth
>>> related factors as well, I threw a few dollars at it and got a
>>> cn5.18xlarge instance (72 CPU’s 192gb memory, 100gbit ethernet)
>>> overkill to do a git/svn test? yes.. yes indeed
>>>
>>> first to rule out the blender.org <http://blender.org> servers lets
>> grab a ubuntu iso
>>> off an .nl mirror (
>> https://mirror.nl.datapacket.com/ubuntu-releases/22.04/ )
>>>
>>> 2022-06-17 17:13:21 (232 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’
>> saved [3654957056/3654957056]
>>>
>>> all right bandwidth is NOT going to be an issue on this box!
>>>
>>> On to blender stuff:
>>>
>>> cloning from git.blender.org <http://git.blender.org> 2m32.895s
>>> svn checkout of lib/win64_vc15 2m56.388s (iftop said 65Mbit peak
>> rate)
>>>
>>> yowza!
>>>
>>> All right, so in London, safe to say all is well
>>>
>>> lets move closer to my location, us-east-2 Ohio (same instance type
>> and os)
>>>
>>> grabbing ubuntu from that same mirror
>>>
>>> 2022-06-17 17:31:00 (29.5 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’
>> saved [3654957056/3654957056]
>>>
>>> all right, that has lost quite a bit of steam, but it's
>>> still nothing to sneeze at, just to be sure lets grab
>>> an iso off the Princeton university mirror 120MB/s ok..
>>> good enough...
>>>
>>> onto cloning blender
>>>
>>> Receiving objects: 5% (104235/2078173), 25.44 MiB | 692.00 KiB/s
>>>
>>> well... that's not looking promising... lets wait it out
>>>
>>> cloning from git.blender.org <http://git.blender.org> 17m23.449s
>>>
>>> sadface... lets try svn next, i lost patience and did not let
>>> it finish.. iftop reported a top RX of 5.59Mbit though..
>>>
>>> To summarize:
>>>
>>> Ubuntu iso from mirror.nl.datapacket.com <
>> http://mirror.nl.datapacket.com> (Speed taken from wget)
>>> My home - western Canada - 1.3 MB/s
>>> AWS - London - 232 MB/S
>>> AWS - Ohio - 29.5 MB/s
>>>
>>> Clone of git.blender.org <http://git.blender.org> (time taken from
>> linux time command)
>>> My home - western Canada - 27m39
>>> AWS - London - 2m32
>>> AWS - Ohio - 17m23
>>>
>>> SVN Clone: (peak RX bitrate taken from iftop)
>>> My home - western Canada - 1.5 Mbit/s
>>> AWS - London - 64 Mbit/s
>>> AWS - Ohio - 5.59 Mbit/s
>>>
>>> Think the only thing we really can conclude is
>>> that being further from the server is leading
>>> to an "unhappy" time for the developer. Given
>>> the fact the EC2 instances were 100% identical
>>> between London and Ohio, it's unlikely to be a
>>> client configuration issue.
>>>
>>> --Ray
>>>
>>> On 2022-06-17 10:10 a.m., Dan McGrath via Bf-committers wrote:
>>> > Hi,
>>> >
>>> > Really, I would need to see a capture from both sides to dig into
>> it. I
>>> > need to see what is being sent, and what is received on each side.
>> There
>>> > could be some weird firewall stuff happening that is causing
>> fragmentation
>>> > or blocking TCP options, or window scaling issues, etc.
>>> >
>>> > As for the default settings for tortoisesvn, the connection stuff
>> I was
>>> > referring to was about TCP/IP behaviour, rather than an application
>>> > setting. Either way, I need to see what is actually happening. The
>> fact
>>> > that I can pull 5-8MB/sec (~40mbit) from Canada and not have a
>> single
>>> > interruption is telling. Granted, the host only has around 100 or
>> 200mbit
>>> > allocated, iirc. Multiply that by a few dozen people and you have
>> a problem.
>>> >
>>> > For a simpler test, maybe try a large svn check, perhaps some old
>> FreeBSD
>>> > ports svn repo if you can find one, something also in .NL, and see
>> how it
>>> > goes, just to help eliminate your PC/firewall. Worst case
>> scenario, you can
>>> > also poke me in chat and we can try some server side and client
>> side
>>> > captures.
>>> >
>>> > TBH though, 1gig uplink just isn't a whole lot to serve our user
>> base ;)
>>> >
>>> > On Fri, Jun 17, 2022 at 11:45 AM Tom M <letterrip at gmail.com>
>> wrote:
>>> >
>>> >> On Fri, Jun 17, 2022 at 5:20 AM Dan McGrath <
>> danmcgrath.ca at gmail.com>
>>> >> wrote:
>>> >>> Hi Tom,
>>> >>>
>>> >>> Long time no see :)
>>> >>>
>>> >>> I did some tests from home and found that, aside from slow
>> speeds, I was
>>> >> able to checkout at least 5 or 6 gig of the bf-blender repo
>> without error
>>> >> or having to retransmit.
>>> >>
>>> >> Yeah, looks like I was mistaking really slow download of the
>> larger
>>> >> (.7 GB) libraries as freezing. I was averaging about .3 MB/s. I
>> used
>>> >> a different SVN client and saw that it was indeed downloading (the
>>> >> sliksvn doesn't provide any feedback on files in progress, only
>> once a
>>> >> file completes) - It was taking about 40 minutes for each of the
>>> >> larger files. It did complete eventually after about 5 hours.
>>> >>
>>> >>> I took the liberty of packet capturing the stream from my PC's
>> POV, and
>>> >> placed them in the ftp download area[1], if you care to take a
>> look.
>>> >>> The gist is that I was able to get around 5MB/sec for at least
>> 5G of the
>>> >> checkout, before I cancelled it.
>>> >> Interesting, I usually get up to 3 MB/s download here normally
>> from
>>> >> other sources, but was typically getting .3 MB/s, with occasional
>>> >> bursts of higher speed to a max of .8 MB/s when it wasn't
>> downloading
>>> >> the large files.
>>> >>
>>> >>> Now, my IP is a bit of a special case, so I am wondering if you
>> are just
>>> >> hitting the server so hard with http requests in separate
>> sockets, or
>>> >> something extreme, that is causing the system to block you for a
>> period of
>>> >> time. For comparison, my PC only had about 4 sockets open to the
>> server for
>>> >> the transfer. I would be curious to know if you are doing
>> pipelining, or
>>> >> sending each request in a separate connection.
>>> >> Whatever the sliksvn and tortoisesvn defaults are, I'd assume
>> they are
>>> >> setup for typical usage.
>>> >>
>>> >>> If you can, the next time it happens, see if you are unable to
>> visit
>>> >> Phabricator or resume the svn checkout process. It may not be a
>> firewall
>>> >> issue, but without seeing some packet captures, I wouldn't be
>> able to
>>> >> really guess. Ideally, try to capture a few packets, until
>> failure. At the
>>> >> very least, you will need to snap 20b+20b=40b for IP+TCP headers,
>> but I
>>> >> would suggest a tad higher, maybe 64b.
>>> >>> Cheers,
>>> >>>
>>> >>> Dan
>>> >>>
>>> >>> [1] - https://download.blender.org/ftp/dan/svn-checkout/
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Thu, Jun 16, 2022 at 2:43 PM Tom M via Bf-committers <
>>> >> bf-committers at blender.org> wrote:
>>> >>>> When checking out the libraries from SVN following the
>> directions on
>>> >>>>
>>> >>>> https://wiki.blender.org/wiki/Building_Blender/Windows
>>> >>>>
>>> >>>> The svn checkout of the libraries fails with significant
>> frequency,
>>> >>>>
>>> >>>> if you google you might stumble across using make svnfix
>>> >>>>
>>> >>>>
>> https://www.mail-archive.com/bf-blender-cvs@blender.org/msg151996.html
>>> >>>>
>>> >>>> Even after using it, I'm still having to do Ctrl C, and then
>> make
>>> >>>> svnfix repeatedly (it downloads some files, than randomly
>> freezes,
>>> >>>> sometimes after 20-50 files, sometimes after 100's of files).
>>> >>>>
>>> >>>> Git doesn't seem to have any problems, though.
>>> >>>>
>>> >>>> So might we consider migrating the libs to git, rather than
>> using SVN?
>>> >>>>
>>> >>>> LetterRip
>>> >>>> _______________________________________________
>>> >>>> Bf-committers mailing list
>>> >>>> Bf-committers at blender.org
>>> >>>> List details, subscription details or unsubscribe:
>>> >>>> https://lists.blender.org/mailman/listinfo/bf-committers
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Cheers,
>>> >>> Danny
>>> >>>
>>> >>> ----------------------------------------------------------
>>> >>> Danny McGrath - danmcgrath.ca at gmail.com
>>> >>> GPG key: EDF6 AFF5 2086 F93A 1F59 36A5 44B6 26F3 6968 71CA
>>> >
>>> _______________________________________________
>>> Bf-committers mailing list
>>> Bf-committers at blender.org
>>> List details, subscription details or unsubscribe:
>>> https://lists.blender.org/mailman/listinfo/bf-committers
>>>
>> _______________________________________________
>> Bf-committers mailing list
>> Bf-committers at blender.org
>> List details, subscription details or unsubscribe:
>> https://lists.blender.org/mailman/listinfo/bf-committers
>>
>
>
More information about the Bf-committers
mailing list