[Bf-cycles] Cycles development newbie

Mon Feb 12 04:21:32 CET 2018

Thank you so much for the detailed explanation once again Lukas. I'll have
a play around and email the mailing list again if I run into any issues.
I'd be happy to write a wiki entry for both the code layout, control flow
and shader addition model once I play around and look at the code myself,
unless you want to do it yourself Lukas, since it's all your effort. Thanks
once again.

Regards,
Bob

On 12 February 2018 at 03:10, Troy Sobotka <troy.sobotka at gmail.com> wrote:

> This effort should be formatted for the Wiki.
>
> With respect,
> TJS
>
> On Sun, Feb 11, 2018, 6:40 PM Lukas Stockner <lukas.stockner at freenet.de>
> wrote:
>
>> Actually, yes, I do have some suggestions for that :)
>>
>> First, you'll want to add a Shader node to Blender itself. The general
>> way to do that is:
>> - Add a file for your node in source/blender/nodes/shader/nodes/ - just
>> copy the file of some other BSDF node that is somewhat similar and adapt
>> it. I'd suggest to ignore the GPU/GLSL stuff for the first test, you can
>> always add that later.
>> - Add your new file in source/blender/nodes/CMakeLists.txt
>> - Call the register_node_* function of your new node from
>> source/blender/blenkernel/intern/node.c
>> - Add its prototype to source/blender/nodes/NOD_shader.h
>> - Allocate a Node ID for it in source/blender/blenkernel/BKE_node.h
>> - If you have some custom element like a checkbox of an enum in addition
>> to the regular node inputs:
>>   * Define the RNA type of your node in source/blender/makesrna/
>> intern/rna_nodetree.c
>>   * Define a drawing function in source/blender/editors/space_
>> node/drawnode.c
>> - Connect everything by adding a line for your node in
>> source/blender/nodes/NOD_static_types.h
>> - Add the node to the Node editor menu in release/scripts/startup/
>> nodeitems_builtins.py
>>
>> Now for the node in Cycles:
>> - Define the node in intern/cycles/nodes.h and nodes.cpp
>> - Sync the node in intern/cycles/blender/blender_shader.cpp
>>
>> I'd recommend to focus on the SVM implementation first and only worry
>> about OSL once that works.
>>
>> And of course, the actual BSDF in the kernel:
>> - Add a closure ID in intern/cycles/kernel/svm/svm_types.h
>> - Add the BSDF implementation in intern/cycles/kernel/closure/ (the
>> shader seems somewhat complex, so a new file probably is best)
>> - Include your new BSDF file in intern/cycles/kernel/closure/bsdf.h
>> - Add the eval and sample functions in the switch statements in
>> intern/cycles/kernel/closure/bsdf.h
>> - Add closure setup in the SVM interpreter in
>> intern/cycles/kernel/svm/svm_closure.h
>>
>> Oh, and three notes on closure implementation:
>> - Closures in Cycles return f(wo, wi)*cos_theta_i, not just f(wo, wi) as
>> it's done in most other renderers I've seen so far (pbrt, Mitsuba etc.)
>> - The I vector in evaluation/sampling is what's usually referred to as wo
>> in other renderers.
>> - For debugging, you can just not return SD_BSDF_HAS_EVAL from the setup,
>> this will cause Cycles to not call your eval function and not do any
>> light-sampling/bsdf-sampling MIS. Of course, that might cause excessive
>> noise with diffuseish materials and sharp lights.
>>
>> - Lukas
>> On 12.02.2018 03:09, Bob Geller wrote:
>>
>> Thank you so much Lukas for the detailed email. It has definitely helped
>> clear up a few things. Using a debugger was going to the next option but I
>> thought I'd ask the experts first. I will do so anyway and pop any
>> questions on this group if I need any more help. I'm looking to add more
>> BSDFs to the Cycles engine as that forms most of the interest like the
>> Irawan & Marschner woven cloth shader that I see is missing. Any
>> suggestions on adding BSDFs other than the normal closure and osl shader
>> additions? Anyway, hopefully I can get a handle of the code-base quickly
>> and get cracking.  Hope you have a great day! Thanks again!
>>
>> Regards,
>> Bob
>>
>> On 11 February 2018 at 19:21, Lukas Stockner <lukas.stockner at freenet.de>
>> wrote:
>>
>>> Hi,
>>>
>>> regarding the overall layout, there is a Wiki article (
>>> https://wiki.blender.org/index.php/Dev:Source/Render/Cycles/SourceLayout)
>>> but it doesn't really say much, so here's my attempt on an overview:
>>>
>>> Overall, Cycles can be divided into four layers:
>>> - Kernel code - the computations that are executed on the chosen device,
>>> doing the actual rendering
>>> - Device code - handles memory management and is able to execute tasks
>>> on the different compute devices by calling the kernel
>>> - "Host" code - handles data synchronization to the device and control
>>> flow during rendering, exposes an API
>>> - Blender code - Glue logic between Blender and Cycles, synchronizes
>>> data from Blender into the Cycles-specific formats
>>> Of those four, the first three are independent of Blender and also used
>>> in e.g. the standalone renderer.
>>>
>>> When you hit F12, Blender internally starts a rendering job (not going
>>> into details here). Effectively, it ends up calling functions in the Cycles
>>> addon (blender/addon/, all paths relative to intern/cycles/). These in turn
>>> call the Cycles Python API (as defined in blender/blender_python.cpp),
>>> which wraps the BlenderSession (in blender/blender_session.cpp).
>>>
>>> This Blender session allocates a Cycles Session (render/session.cpp) and
>>> Scene (render/scene.cpp) and then fills the Cycles-internal scene
>>> representation (consisting of Shaders, Meshes, Lights etc., all defined in
>>> render/ and part of the Scene class) from Blender-internal data through the
>>> code in BlenderSync (blender/blender_sync.cpp).
>>>
>>> Then, it calls the Session to start the rendering. The Session in turn
>>> calls the internal Scene synchronization, which then loops over all the
>>> components of a Scene and calls their device_update functions, which are
>>> responsible for converting the internal host representation into the format
>>> that the kernel expects. Once that is done, it divides the image into tiles
>>> (render/tile.cpp) and then starts a DeviceTask (device/device_task.cpp) for
>>> rendering.
>>>
>>> The device (device/device_{cpu, cuda, opencl, multi}.cpp) then calls the
>>> acquire_tile callback of the task, which makes the Session return tiles to
>>> render. Then, it calls the rendering kernel for that particular device
>>> until the tile is finished, and goes back to acquiring another tile until
>>> they're all rendered.
>>>
>>> In the kernel, most of the code is shared between devices, which means
>>> that we can only use the common subset of C++, CUDA and OpenCL (which is
>>> generally the limiting factor). To archieve this, whenever any language
>>> needs a particular keyword, a ccl_ keyword is defined. Then, for each of
>>> the three languages, these ccl_keywords are #defined to the proper keywords
>>> in kernel/kernel_compat_<device>.h. For example, CUDA requires each
>>> function that's supposed to be executed on the GPU to be prefixed by
>>> __device__. Therefore, we have the ccl_device keyword, which gets #defined
>>> to __device__ for CUDA and to nothing for CPU and OpenCL. This is all there
>>> is to the "cycles compute language" - it's essentially just a bunch of
>>> defines that allows to write code that compiles in all three languages.
>>>
>>> As a result, for many changes you can completely ignore the fact that
>>> your code will also be compiled on GPUs. The main downside is that it
>>> completely screws up code completion in most IDEs.
>>> Also, here's the wiki article on the kernel language (a bit outdated
>>> unfortunately): https://wiki.blender.org/index.php/Dev:Source/Render/
>>> Cycles/Kernel
>>>
>>> Generally, to get an overview of the control flow, I'd recommend looking
>>> at:
>>> - BlenderSession::create_session, BlenderSession::reset_session,
>>> BlenderSession::render
>>> - BlenderSync in general
>>> - Session::run_cpu
>>> - Scene::device_update
>>> - CPUDevice::thread_render
>>> - Session::acquire_tile
>>> - kernel_path_trace
>>>
>>> And ultimately, the best approach for figuring out control flow of
>>> course always is searching and a debugger.
>>>
>>> Hope this helps :)
>>>
>>> - Lukas
>>>
>>>
>>> On 11.02.2018 13:09, Bob Geller wrote:
>>>
>>> Hi,
>>>
>>> My name is Bob and I'm trying to get into cycles development to
>>> hopefully contribute in the future. Is there any documentation to
>>> understand the system better and especially understand the flow of control
>>> in the renderer? Also, I'm struggling to find enough info about cycles
>>> kernel development and the cycles compute language. I would appreciate if
>>> someone can point me in the right direction to help understand this better.
>>> Thank you for your time and sorry for mailing the whole group, I wasn't
>>> sure who to contact.
>>>
>>> Regards,
>>> Bob
>>>
>>>
>>> _______________________________________________
>>> Bf-cycles mailing listBf-cycles at blender.orghttps://lists.blender.org/mailman/listinfo/bf-cycles
>>>
>>>
>>>
>>> _______________________________________________
>>> Bf-cycles mailing list
>>> Bf-cycles at blender.org
>>> https://lists.blender.org/mailman/listinfo/bf-cycles
>>>
>>>
>>
>>
>> _______________________________________________
>> Bf-cycles mailing listBf-cycles at blender.orghttps://lists.blender.org/mailman/listinfo/bf-cycles
>>
>>
>> _______________________________________________
>> Bf-cycles mailing list
>> Bf-cycles at blender.org
>> https://lists.blender.org/mailman/listinfo/bf-cycles
>>
>
> _______________________________________________
> Bf-cycles mailing list
> Bf-cycles at blender.org
> https://lists.blender.org/mailman/listinfo/bf-cycles
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.blender.org/pipermail/bf-cycles/attachments/20180212/8f83e45b/attachment.html>