[Bf-committers] move prebuilt libs from svn to git?

Zack Brown zacharyb at gmail.com
Sat Jun 18 09:04:46 CEST 2022


Hi everyone,

It's fascinating to see this discussion. It never occurred to me that
anyone might continue to prefer svn once git came into existence, and yet
here is a sample case, right under my nose! I guess svn is still relevant
even after all these years!

But actually I'd be curious to know why revision control is used at all in
this particular case. We're talking about precompiled binaries, right?
That's sort of the poster child for situations where revision control is
*not* what one wants, exactly for the reason why git would be a bad choice.

Please, could someone explain why the blender project doesn't simply use a
dev-accessible shared folder somewhere? I love revision control, but it's
only useful if it's useful.

Be well,
Zack


On Sat, Jun 18, 2022 at 1:10 AM Ray Molenkamp via Bf-committers <
bf-committers at blender.org> wrote:

> Just an update with some new numbers, dan's changes
> brought down the time for a full clone from my home
> connection down to 5m48s (down from 27m39s previously)
>
> well done dan!
>
> --Ray
>
> On 2022-06-17 3:45 p.m., Dan McGrath wrote:
> > Just a heads-up that Ray and I took a packet capture on both ends for me.
> >
> > Thanks to his capture, I noticed that the SVN server host was not
> sending TCP window scale or SACK options. Ultimately this turned out to be
> due to that host using the PF firewall "synproxy" state option, which
> apparently was causing the TCP header options to be removed, which limited
> the window size to 64k.
> >
> > I have corrected this change to match the www host, and we did observe
> some improvements that were hidden from me before due to my and the studio
> IPs special treatment that bypassed this feature.
> >
> > While it should help, I also tuned a few sysctl settings for send and
> receive buffers. In addition, I noticed that the SVN server is also not
> sending gzip/deflate accept headers for the files. No doubt this is going
> to be a factor. However, there was a note from 2016 from myself explaining
> that Brecht asked for the SVN compression to be disabled due to some client
> problems. So, this will require some research, perhaps next week.
> >
> > Anyway, hopefully it helps a bit until I can review some more settings.
> >
> > Enjoy your weekend everyone!
> >
> >
> > Dan
> >
> > On Fri, Jun 17, 2022, 2:22 PM Ray Molenkamp via Bf-committers <
> bf-committers at blender.org> wrote:
> >
> >     =8<==[Editors note]=8<==
> >     I'll be honest I'm 100% convinced mailinator will ruin
> >     the little formatting I have done, I put a copy of this
> >     email on https://developer.blender.org/P3014.
> >     =8<====8<====8<====8<==
> >
> >     How much as I like the sentiment of "lets move to git will it
> >     solve all these problems" lets be honest here, git.blender.org <
> http://git.blender.org>'s
> >     speed is nothing to write home about either it may or may not be
> >     as glitchy as svn, but it still wouldn't be fast.
> >
> >     from my 300mbit connection in western Canada
> >
> >     Receiving objects:   6% (135931/2078159), 33.34 MiB | 458.00 KiB/s
> >
> >     total time for a clone off git.blender.org <http://git.blender.org>
> 27m39s
> >
> >     Cloning off https://github.com/blender/blender.git however
> >
> >     total time for a clone off github.com <http://github.com> 1m 52s
> >
> >     Now amount of finger pointing to client settings will convince me
> >     it's a client setting at this point, but I'll be honest, I am all
> >     the way in western Canada and that could definitely be a factor.
> >
> >     sooooo, lets investigate!
> >
> >     Let’s spin up an AWS EC2 instance in London, I'd say that be close
> >     enough to Amsterdam, and let’s rule out any other CPU or Bandwidth
> >     related factors as well, I threw a few dollars at it and got a
> >     cn5.18xlarge instance (72 CPU’s 192gb memory, 100gbit ethernet)
> >     overkill to do a git/svn test? yes.. yes indeed
> >
> >     first to rule out the blender.org <http://blender.org> servers lets
> grab a ubuntu iso
> >     off an .nl mirror (
> https://mirror.nl.datapacket.com/ubuntu-releases/22.04/ )
> >
> >     2022-06-17 17:13:21 (232 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’
> saved [3654957056/3654957056]
> >
> >     all right bandwidth is NOT going to be an issue on this box!
> >
> >     On to blender stuff:
> >
> >     cloning from git.blender.org <http://git.blender.org>  2m32.895s
> >     svn checkout of lib/win64_vc15 2m56.388s (iftop said 65Mbit peak
> rate)
> >
> >     yowza!
> >
> >     All right, so in London, safe to say all is well
> >
> >     lets move closer to my location, us-east-2 Ohio (same instance type
> and os)
> >
> >     grabbing ubuntu from that same mirror
> >
> >     2022-06-17 17:31:00 (29.5 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’
> saved [3654957056/3654957056]
> >
> >     all right, that has lost quite a bit of steam, but it's
> >     still nothing to sneeze at, just to be sure lets grab
> >     an iso off the Princeton university mirror 120MB/s ok..
> >     good enough...
> >
> >     onto cloning blender
> >
> >     Receiving objects:   5% (104235/2078173), 25.44 MiB | 692.00 KiB/s
> >
> >     well... that's not looking promising... lets wait it out
> >
> >     cloning from git.blender.org <http://git.blender.org>  17m23.449s
> >
> >     sadface... lets try svn next, i lost patience and did not let
> >     it finish.. iftop reported a top RX of 5.59Mbit though..
> >
> >     To summarize:
> >
> >     Ubuntu iso from mirror.nl.datapacket.com <
> http://mirror.nl.datapacket.com> (Speed taken from wget)
> >     My home - western Canada - 1.3  MB/s
> >     AWS     - London         - 232  MB/S
> >     AWS     - Ohio           - 29.5 MB/s
> >
> >     Clone of git.blender.org <http://git.blender.org> (time taken from
> linux time command)
> >     My home - western Canada - 27m39
> >     AWS     - London         -  2m32
> >     AWS     - Ohio           - 17m23
> >
> >     SVN Clone: (peak RX bitrate taken from iftop)
> >     My home - western Canada -  1.5 Mbit/s
> >     AWS     - London         -   64 Mbit/s
> >     AWS     - Ohio           - 5.59 Mbit/s
> >
> >     Think the only thing we really can conclude is
> >     that being further from the server is leading
> >     to an "unhappy" time for the developer. Given
> >     the fact the EC2 instances were 100% identical
> >     between London and Ohio, it's unlikely to be a
> >     client configuration issue.
> >
> >     --Ray
> >
> >     On 2022-06-17 10:10 a.m., Dan McGrath via Bf-committers wrote:
> >     > Hi,
> >     >
> >     > Really, I would need to see a capture from both sides to dig into
> it. I
> >     > need to see what is being sent, and what is received on each side.
> There
> >     > could be some weird firewall stuff happening that is causing
> fragmentation
> >     > or blocking TCP options, or window scaling issues, etc.
> >     >
> >     > As for the default settings for tortoisesvn, the connection stuff
> I was
> >     > referring to was about TCP/IP behaviour, rather than an application
> >     > setting. Either way, I need to see what is actually happening. The
> fact
> >     > that I can pull 5-8MB/sec (~40mbit) from Canada and not have a
> single
> >     > interruption is telling. Granted, the host only has around 100 or
> 200mbit
> >     > allocated, iirc. Multiply that by a few dozen people and you have
> a problem.
> >     >
> >     > For a simpler test, maybe try a large svn check, perhaps some old
> FreeBSD
> >     > ports svn repo if you can find one, something also in .NL, and see
> how it
> >     > goes, just to help eliminate your PC/firewall. Worst case
> scenario, you can
> >     > also poke me in chat and we can try some server side and client
> side
> >     > captures.
> >     >
> >     > TBH though, 1gig uplink just isn't a whole lot to serve our user
> base ;)
> >     >
> >     > On Fri, Jun 17, 2022 at 11:45 AM Tom M <letterrip at gmail.com>
> wrote:
> >     >
> >     >> On Fri, Jun 17, 2022 at 5:20 AM Dan McGrath <
> danmcgrath.ca at gmail.com>
> >     >> wrote:
> >     >>> Hi Tom,
> >     >>>
> >     >>> Long time no see :)
> >     >>>
> >     >>> I did some tests from home and found that, aside from slow
> speeds, I was
> >     >> able to checkout at least 5 or 6 gig of the bf-blender repo
> without error
> >     >> or having to retransmit.
> >     >>
> >     >> Yeah, looks like I was mistaking really slow download of the
> larger
> >     >> (.7 GB) libraries as freezing.  I was averaging about .3 MB/s.  I
> used
> >     >> a different SVN client and saw that it was indeed downloading (the
> >     >> sliksvn doesn't provide any feedback on files in progress, only
> once a
> >     >> file completes) - It was taking about 40 minutes for each of the
> >     >> larger files.  It did complete eventually after about 5 hours.
> >     >>
> >     >>> I took the liberty of packet capturing the stream from my PC's
> POV, and
> >     >> placed them in the ftp download area[1], if you care to take a
> look.
> >     >>> The gist is that I was able to get around 5MB/sec for at least
> 5G of the
> >     >> checkout, before I cancelled it.
> >     >> Interesting, I usually get up to 3 MB/s download here normally
> from
> >     >> other sources, but was typically getting .3 MB/s, with occasional
> >     >> bursts of higher speed to a max of .8 MB/s when it wasn't
> downloading
> >     >> the large files.
> >     >>
> >     >>> Now, my IP is a bit of a special case, so I am wondering if you
> are just
> >     >> hitting the server so hard with http requests in separate
> sockets, or
> >     >> something extreme, that is causing the system to block you for a
> period of
> >     >> time. For comparison, my PC only had about 4 sockets open to the
> server for
> >     >> the transfer. I would be curious to know if you are doing
> pipelining, or
> >     >> sending each request in a separate connection.
> >     >> Whatever the sliksvn and tortoisesvn defaults are, I'd assume
> they are
> >     >> setup for typical usage.
> >     >>
> >     >>> If you can, the next time it happens, see if you are unable to
> visit
> >     >> Phabricator or resume the svn checkout process. It may not be a
> firewall
> >     >> issue, but without seeing some packet captures, I wouldn't be
> able to
> >     >> really guess. Ideally, try to capture a few packets, until
> failure. At the
> >     >> very least, you will need to snap 20b+20b=40b for IP+TCP headers,
> but I
> >     >> would suggest a tad higher, maybe 64b.
> >     >>> Cheers,
> >     >>>
> >     >>> Dan
> >     >>>
> >     >>> [1] - https://download.blender.org/ftp/dan/svn-checkout/
> >     >>>
> >     >>>
> >     >>>
> >     >>> On Thu, Jun 16, 2022 at 2:43 PM Tom M via Bf-committers <
> >     >> bf-committers at blender.org> wrote:
> >     >>>> When checking out the libraries from SVN following the
> directions on
> >     >>>>
> >     >>>> https://wiki.blender.org/wiki/Building_Blender/Windows
> >     >>>>
> >     >>>> The svn checkout of the libraries fails with significant
> frequency,
> >     >>>>
> >     >>>> if you google you might stumble across using make svnfix
> >     >>>>
> >     >>>>
> https://www.mail-archive.com/bf-blender-cvs@blender.org/msg151996.html
> >     >>>>
> >     >>>> Even after using it, I'm still having to do Ctrl C, and then
> make
> >     >>>> svnfix repeatedly (it downloads some files, than randomly
> freezes,
> >     >>>> sometimes after 20-50 files, sometimes after 100's of files).
> >     >>>>
> >     >>>> Git doesn't seem to have any problems, though.
> >     >>>>
> >     >>>> So might we consider migrating the libs to git, rather than
> using SVN?
> >     >>>>
> >     >>>> LetterRip
> >     >>>> _______________________________________________
> >     >>>> Bf-committers mailing list
> >     >>>> Bf-committers at blender.org
> >     >>>> List details, subscription details or unsubscribe:
> >     >>>> https://lists.blender.org/mailman/listinfo/bf-committers
> >     >>>
> >     >>>
> >     >>> --
> >     >>> Cheers,
> >     >>> Danny
> >     >>>
> >     >>> ----------------------------------------------------------
> >     >>> Danny McGrath - danmcgrath.ca at gmail.com
> >     >>> GPG key: EDF6 AFF5 2086 F93A 1F59 36A5 44B6 26F3 6968 71CA
> >     >
> >     _______________________________________________
> >     Bf-committers mailing list
> >     Bf-committers at blender.org
> >     List details, subscription details or unsubscribe:
> >     https://lists.blender.org/mailman/listinfo/bf-committers
> >
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> List details, subscription details or unsubscribe:
> https://lists.blender.org/mailman/listinfo/bf-committers
>


-- 
Zack Brown


More information about the Bf-committers mailing list