<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Courier;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Don’t be too excited too early. The more I work with it, the more it looks like it’s just an elaborate workaround for compiler behavior. It appears that NVCC insists on inlining everything
on Maxwell, ignoring any __noinline__ hints. So far, there are no benefits whatsoever on Kepler, there NVCC appears to do the right thing out of the box.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I submitted a bug report to Nvidia about the difference in stack usage between Kepler and Maxwell last year, and it was marked as resolved and to be shipped in the next CUDA update. So
maybe I shouldn’t spend too much time with it until we see CUDA 8.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">-Stefan</span><span style="font-size:9.0pt;font-family:Calibri;color:#919191"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black"><bf-cycles-bounces@blender.org> on behalf of Thomas Dinges <blender@dingto.org><br>
<b>Reply-To: </b>Discussion list to assist Cycles render engine developers <bf-cycles@blender.org><br>
<b>Date: </b>Tuesday, May 17, 2016 at 4:45 PM<br>
<b>To: </b>Discussion list to assist Cycles render engine developers <bf-cycles@blender.org><br>
<b>Subject: </b>Re: [Bf-cycles] split kernel and CUDA<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-right:0cm" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">That sounds promising, feel free to submit a patch for this and we can check. :)
<o:p></o:p></p>
<div>
<p class="MsoNormal">Am 17.05.2016 um 16:40 schrieb Stefan Werner:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">The patch is surprisingly clean. It removes some of the #ifdef __SPLIT_KERNEL__ blocks and unifies CPU, OpenCL and CUDA a bit more. I didn’t run a speed benchmark, and I wouldn’t even make
speed the ultimate top priority: Right now, the problem we see in the field is that people are unable to use high-end gaming GPUs because the VRAM is so full of geometry and textures that the CUDA runtime doesn’t have room for kernel memory any more. On my
1664 core M4000 card, I see a simple kernel launch already taking ~1600MB of VRAM with almost empty scenes.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">It looks to me like the CUDA compiler reserves room for every stack instance of ShaderData (or other structs) in advance, and that sharing that memory instead of instantiating it separately
is an easy way to reduce VRAM requirements without changing the code much.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">-Stefan</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black"><a href="mailto:bf-cycles-bounces@blender.org"><bf-cycles-bounces@blender.org></a> on behalf of Sergey Sharybin
<a href="mailto:sergey.vfx@gmail.com"><sergey.vfx@gmail.com></a><br>
<b>Reply-To: </b>Discussion list to assist Cycles render engine developers <a href="mailto:bf-cycles@blender.org">
<bf-cycles@blender.org></a><br>
<b>Date: </b>Tuesday, May 17, 2016 at 9:20 AM<br>
<b>To: </b>Discussion list to assist Cycles render engine developers <a href="mailto:bf-cycles@blender.org">
<bf-cycles@blender.org></a><br>
<b>Subject: </b>Re: [Bf-cycles] split kernel and CUDA</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<div>
<div>
<p class="MsoNormal">hi,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal">Lukas Stocker was doing experiments with CUDA split kernel. With the current design of the split it was taking more VRAM actually, AFAIR. Hopefully he'll read this mail and reply in more details.
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Would be cool to have this front moving forward, but i fear we'll have to step back and reconsider some things about how split kernel works together with a regular one.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">There are interesting results on the stack memory! I can see number of spill loads go up tho, did you measure if it gives measurable render time slowdown? And how messy is the patch i wonder :)<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal">On Tue, May 17, 2016 at 8:47 AM, Stefan Werner <<a href="mailto:swerner@smithmicro.com" target="_blank">swerner@smithmicro.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<p class="MsoNormal">Hi,<br>
<br>
Has anyone experimented with building a split kernel for CUDA? It seems to me that this could lift some of the limitations on Nvidia hardware, such as the high memory requirements on cards with many CUDA cores or the driver time out. I just tried out what happens
when I take the shared ShaderData (KernelGlobals.sd_input) from the split kernel into the CUDA kernel, as opposed to creating separate ShaderData structs on the stack, and it looks like it has an impact:<br>
<br>
before:<br>
ptxas info : Compiling entry function 'kernel_cuda_branched_path_trace' for 'sm_50'<br>
ptxas info : Function properties for kernel_cuda_branched_path_trace<br>
68416 bytes stack frame, 1188 bytes spill stores, 3532 bytes spill loads<br>
<br>
after:<br>
ptxas info : Compiling entry function 'kernel_cuda_branched_path_trace' for 'sm_50'<br>
ptxas info : Function properties for kernel_cuda_branched_path_trace<br>
58976 bytes stack frame, 1256 bytes spill stores, 3676 bytes spill loads<br>
<br>
-Stefan<br>
<br>
_______________________________________________<br>
Bf-cycles mailing list<br>
<a href="mailto:Bf-cycles@blender.org">Bf-cycles@blender.org</a><br>
<a href="https://lists.blender.org/mailman/listinfo/bf-cycles" target="_blank">https://lists.blender.org/mailman/listinfo/bf-cycles</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="color:#666666">With best regards, Sergey Sharybin</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Bf-cycles mailing list<o:p></o:p></pre>
<pre><a href="mailto:Bf-cycles@blender.org">Bf-cycles@blender.org</a><o:p></o:p></pre>
<pre><a href="https://lists.blender.org/mailman/listinfo/bf-cycles">https://lists.blender.org/mailman/listinfo/bf-cycles</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</body>
</html>