Twice a month we revisit some of our reader’s favorite posts from throughout the history of Cgtuts+. Today we’re taking a look back at Ahmed Fathi’s tutorial on compositing Vray render layers in Photoshop. This tutorial was originally published back in July of 2010, and generated a lot of interest. So if you missed it the first time, why not check it out now.

In this tutorial Ahmed Fathi takes a look at how to composite together V-ray render layers using blending-modes and masks in Photoshop. Once completed, this process allows you to change or tweak any aspect of your image in seconds without having to re-render a thing! Ahmed also covers a few extra post production techniques such as Chromatic Aberration and Depth Of Field, as well as how to emulate a Cross-processed look.

Step 1

As this is a compositing tutorial, not a lighting/rendering tutorial, I’ll assume that you have at least a basic knowledge of V-ray, and that you are able to render out your own scenes already. We’re going jump straight ahead into setting up the different render elements for the compositing process.

In order to make V-ray render out the different layers, we first have to enable them in the V-ray Render Elements tab within the Render Settings window. Once in the tab, we want to enable the following render elements as shown :

• VrayDiffuseFilter
• VrayMtlID
• VrayObjectID
• VrayRawGlobalIllumination
• VrayRawLighting
• VrayRawShadow
• VrayReflection
• VrayRefraction
• VraySpecular
• and VrayZDepth.

Most of these elements don’t need much work to get them right, but we are going to need to take a few steps to set up the VrayMtlID, VrayObjectID and VrayZDepth layers.

Step 2

We’ll start with the VRayZDepth element. The ZDepth layer is a black and white map that is used to tell Photoshop how far each object in our scene sits away from the render camera – the further the object is from the camera, the darker it will appear in the Zdepth layer. Typically the Min value is used to tell the compositor which objects will be in focus.

In order to correctly setup a Zdepth map, we have to adjust the Min & Max distances that V-ray should calculate, and therefore what appears as white (the Min value) and black (the Max value) within our scene. Select the VRayZDepth item in the elements list. At the bottom you’ll see the zdepth min and zdepth max values we need to adjust.

Step 3

To get a sense of what values you should use for the min and max values, you should use a tape helper object to measure the distance between the camera and your closest and furthest objects.
To simplify things, let’s say sphere #2 in the image below is the closest sphere we want to be in focus, whilst Sphere #6 should be slightly out of focus. We would use two tapes to measure those distances and put them in as your Min & Max Values.

Step 4

Here’’s what the VRayZdepth render for this composite looked like after setting the Min to 20 and Max to 70 meters

Step 5

The VrayMtlID render layer creates an image with a different solid colour for each material in your scene, and to use it, we need to adjust your material IDs. In your material editor you will find an icon with the number 0 on it (as shown below.) If you click the 0, a grid of numbers from 0 to 15 will appear – this is your material ID. Go through your scene and apply a different number to each material you want to have a different color in your MtlID element. As we have 16 numbers, we can have 16 different materials appearing in our MtlID render layer.

Step 6

Here’’s how your VRayMtlID element would look like after setting up the ID’s for each required material. Your result may have different colors, but the important thing is that they are separate from one another.

Step 7

VRayObjectID element is just like material IDs element mentioned above, but it outputs a different colour based on the different objects in your scene – the Materials are irrelevant to it. To set this up, right click your desired object, select Object Properties and give each object a different number in the Object ID field in the G-Buffer section.

Step 8

I only needed to use the VRayObjectID element in order to make selecting the cars and the surrounding buildings easier in post. Here’’s how it looks after rendering.

Step 9

The last element we need to add in is the ambient occlusion layer, but as you’ll remember, this wasn’t added to our render elements list in V-ray. We actually need to render it separately after your initial render has been completed, but don’t worry, there is an easy, fast way to do it!

Apply a VrayLightMtl to your entire scene, and then add a VrayDirt map into the color channel. You’ll then need to go in and tweak the dirt map settings until you get a good, clean result result. For this scene, I got a nice looking AO map with a Radius of 2 in my dirt map settings.

Note: if you use a V-ray Physical Camera in your scene, (which I highly recommend,) you will need to turn off your Exposure and Vignette options in your camera settings in order to render the AO pass properly.

Step 10

After a couple of trials with the dirt map radius, I ended up with the following AO element. I also increased the dirt map subdivs from 8 to about 64 which helped to smooth out the final render, and it still rendered out relatively fast.

Step 11

Just a couple more steps and we’ll be ready to fire up Photoshop and start compositing! But first, how do we get all of these elements out of V-ray? For starters, there is the obvious way of saving them to file one at a time, however there are infact two techniques used for saving the different channels to disk and we’ll cover both of them in the next few steps.

The first method is really helpful, especially for those who use a Linear Workflow, and that is saving all the render elements out into a single .EXR file. To do that you should first select Enable built-in Frame Bufferfrom the V-Ray::Frame buffer menu, and then turn on the “Render to V-Ray Raw image file option ”below. With that done, we can click browse.

Step 12

Browse to the place where you want to save your file, and enter your file name making sure to add the .EXR extension onto the end of it. Then select All Files(*.*) as your file type. This will allow you to save your render as a single .EXR file that contains all your different render layers.

If, however, you try to open your new .EXR file in Photoshop, you will only see the first render layer. To see the others, you’ll need to install a commercial Photoshop plug-in called ProEXR, and that is why I prefer the next method!

Note: EXR files burn in any gamma correction as it assumes you’ll be using a linear workflow. If your image appears washed out in Photoshop, go to Image > Adjustments > Exposure, and set your Gamma to 0.454, which is the inverse of 2.2 (calculated by dividing 1.0 by 2.2).

Step 13

The second method (my preferred one) is to save all your elements out manually as .TGA files. I prefer TGAs to JPGs as they are 32-Bit, and can hold much more color Information than the normal 8-Bit JPG file. Another bonus is that they support having a built-in alpha channe.

Note: if you have never used the V-Ray Frame Buffer to render out elements before, you can find all of your elements in the top left drop down menu that says Diffuse by default.

Step 14

Now that you have finished our 3Ds Max and V-Ray Part, fire up Photoshop and let’s start playing! First a small tip to help you get all of the different files into layers in one Photoshop document. It is a script built into Photoshop that will stack your files as layers and arrange them in alphabetical order. Go to File > Scripts > Load Scripts as Stack, and then browse for your files, select them all and click Ok. Some people prefer bringing in the elements one at a time, however I do prefer this method myself.

Step 15

This is actually your first step in compositing! We’ll start by turning off the visibility on all layers except for the Z-Depth. Select all of the contents of this layer (Ctrl+A), copy them (Ctrl+C) and then switch to your Channels tab.
Some people prefer adding a little bit of Gaussian blur to their Z-Depth before using it, however as most of my clients hate DOF, thinking that it’s a needless loss of detail, I do not do this! In the end it is a matter of personal preference.

Step 16

Once in the Channels tab, first check to see that your original Alpha layer is present. If it isn’t, open up the Diff.tga file seperately, switch to it’s Channels tab, and drag and drop the Alpha channel from there onto our Z-depth image – an Alpha1 channel should appear. You can then close the Diff.tga file.

Now, click the Create New Channel and paste your Zdepth render into it. You can then switch back to the Layers tab and delete your Z-Depth layer as we don’t need it anymore.

Step 17

To follow along with me in the next few steps, first arrange your layers in the order shown. The RGB layer at the top of the stack is the raw render, straight out of V-ray. I’ve kept it there so that we have something to compare to at the end.

Step 18

First, duplicate your Diffuse layer by right clicking it and choosing the Duplicate option, and then move the copy under your RawLight layer as shown.

Step 19

Turn on visibility for your Diffuse layer and your RawGI layer. Set your Raw GI blending mode to Multiply.

Step 20

Next hold down the Alt key and click between you RawGI layer and your Diffuse layer (the cursor will change into this two circles and an arrow icon when you’re in the correct place). This little bent arrow that appears indicates that this layer (RawGI) is only affecting the Diffuse layer. This is called a clipping mask, and you can find out more about it using a simple online search. There are tons of tutorials out there!

Step 21

Using the same technique that you learned in the last two steps. Enable visibility for both “Diffuse Copy” and RawLight. Set the blending mode for the RawLight Layer to Multiply then use it to only affect “Diffuse Copy” layer (Alt+click between the two layers).

Step 22

For the sake of better organization, group both the Diffuse & RawGI layers into a group, and the Diffuse Copy & RawLight layers into a different group. To create a group you can click on the small folder icon at the bottom right of your layers (I should say sorry for people who actually know all this basic stuff. I am just trying not to let anything pass by for beginners too!)

Step 23

Change the blending mode the of “Diffuse + Raw Light” Group to Linear dodge (Add). This adds the information contained in this group to the information contained in the group “below”. The image should start looking more natural now.

Step 24

At this point things look much better, but you may be asking yourself – “why is my glass black?” Well, most of the glass’ information in the final render is contained in the reflection and refraction passes, and we haven’t composed them just yet.

So, enable visibility for your Reflection Layer and set it’s blending mode to Linear Dodge (Add) and see the difference. One of the main benefits of having a composite like this is that, for example, on this reflection layer you can paint/paste in any reflection you want to appear in the windows.

Note: I should explain a little bit what linear Dodge (Add) blending mode does. It adds the color information of the pixels to each other. We know that for example Pure White is 255 and Pure Black is 0. So adding pure black adds 0 whilst adding pure white adds 255. Therefore, pure black (0) + pure white (255) = Pure white (255), Mid grey (125) + Mid grey (125) = White (250) and so on. That is why when you use this mode you no longer see any of your black, because it’s a zero-value color.

Step 25

To illustrate how easy it is to edit any of your elements (color correct it, adjust exposure, etc…) I will assume that I now want to change the reddish color of my stone texture. If we hadn’t used this multi-pass compositing method, we would have two options:

1. Adjust the texture itself in Photoshop and then re-render, perhaps completing a dozen test renders before you were satisfied with the result, and could render a HQ render. This obviously takes a lot of time and patience!
2. Using selection tools in Photoshop, we could select the texture and apply the desired corrections to it. The first problem with this is that selection is really a very tedious job, and when you come to add color corrections, you’’ll be affecting all of the image – your shadows and highlights will look odd and overall things won’’t look that good.

Step 26

Well, remember that element called MtlID that we rendered out? It is time to enable that layer. The only use for this layer is to create MUCH faster selections with only a couple of mouse clicks. Using this layer we will easily select our stone texture in no time at all.

Step 27

With the layer selected, go to Select > Color Range. Then use the eye dropper tool to sample the magenta color on our materials layer that represents the ID of the stone texture. With that done, press OK and you have your selection.

Step 28

Now turn off the visibility of your MtlID layer. Then while keeping that selection active, add an Exposure adjustment layer. This will create the adjustment layer and automatically add a mask so that it only affects your Stone. It doesn’t matter where you have this layer now, we will move it later on.

Step 29

In the Exposure adjustment layer’s settings, set your Exposure to +0.80 to brighten it up a bit.

Step 30

As we want this layer to affect our diffuse only, you will have to place it right above the Diffuse layer (remember that one inside the group?) Move it into position as shown and remember to Alt-click between the layers to make it only affect your Diffuse channels.

Just as a side note; whether you add it to your Diffuse layer or your “Diffuse Copy” Layer shouldn’t make a difference at all. Some corrections might require you to place them on both Diffuse layers though, and it’s just a matter of trial and errors until till you fully understand it all.

Step 31

Take your Refraction layer one step down so that it sits right above your Reflection layer. Enable both and set their blending modes to Linear Dodge (Add). I’ve added another Exposure adjustment, with the Exposure value set to +0.80, to my Refraction layer as I wanted it to appear a little brighter.

Note: Editing the Refraction layer is one of the best ways to adjust the tint of clear glass in interior renders.

Step 32

Enable your Specular layer and once again change its blending mode to Linear Dodge (Add).

Step 33

Duplicate your Specular layer (Ctrl+J) – notice it still has Linear dodge (Add) as its blending mode – and then with the copy selected, goto Filter > Blur > Gaussian blur.

Step 34

Set your Gaussian Blur Radius to around 2.5 pixels and press OK. Set your layer Opacity to around 65%. This should help create a specular bloom effect around your specular highlights.

Note: These values are not constant numbers as they depend on your resolution, and of course your taste. Just don’t over-do it!

Step 35

Enable your RawShadow Layer and invert it (Ctrl + I).

Step 36

You might be surprised that your shadows are bluish but this is in fact normal, they always are during the day because V-ray’s GI has a bright blue sky. Set your RawShadow layer’s blending mode to Multiply and it’s Opacity to around 20%.

Step 37

This step is not a must but I like enhancing my shadows a bit as it gives the image a bit more contrast. Add a “Color Balance” adjustment layer, making it only affect the RawShadow layer, and with the Tones value set to Shadows, give your shadows a reddish tint. Set the opacity for this adjustment layer down to around 20%, although again, this is just a matter of personal preference.

In other scenes when I find my shadows a little too sharp for my taste, I add a small radius Gaussian blur filter to the shadow layer, which helps smooth out those hard edged shadows a lot.

Step 38

Now enable your Ambient Occlusion layer. Set its blending mode to Multiply and its Opacity to around 10%. Ambient Occlusion shouldn’’t be too obvious in your final render, as it just helps you enhance the look of the little details in your final image.

Step 39

At this point, we are pretty much done with the compositing! To sum it all up here’’s a simple equation that I used until I memorised the different blending modes. If you follow it through, you’ll see it exactly matches what we’ve done in this tutorial!

Step 40

The only thing missing now is a sky with a few clouds, and that’’s what we are going to add next. You can also use a bluish gradient or, if you prefer, you can cut out the V-ray sky from your original render and paste it behind this image. To do any of these things however, we first need to access our alpha information.

Go to the Channels pane and Ctrl-click on your main alpha channel’’s thumbnail. This will make an automatic selection of your image ignoring any transparent pieces so that we can easily add in a background image.

Step 41

Invert your selection (Ctrl + Shift + I) so that we only have the sky background selected.

In my original render I used a cyan-ish gradient with some stock clouds. However as I don’t have the rights to redistribute those we will have to find another image. Using Google images search the words “sky field” or clouds field or something similar. Set your search options to Large Images only so you get high resolution pictures only. I found this one on the first page.

Step 42

Save the image and open it in Photoshop. Select all of its contents (Ctrl + A), copy them (Ctrl + C), and as we still have that selection we made using our alpha, use Photoshop’’s command “Paste Into” (Ctrl + Shift + V) command. This should automatically add a mask to our new sky layer. Rename this new layer something like Sky BG
.
Note: In Photoshop CS5 this shortcut changed to (Ctrl + Alt + Shift + V).

Step 43

Using Free transform (Ctrl + T) adjust the size of your sky until it fits the image. Remember to hold Shift while resizing to uniformly scale.

Then add a “Color Balance” adjustment layer over your sky BG only (again using the clipping mask method) and play with the settings until you’re happy with the result. I liked mine with a bit of a cyan tint to it.

Step 44

We are now done with the compositing! Now I’m going to cover some post production tips like adding motion blur to cars, adding DOF using ZDepth, and adding chromatic aberration. You have come a really long way, so let’’s compare our composite to the original V-ray output (the RGB layer).

As you can see, apart from the skies, they both look extremely similar, the only difference being that in our composite, anything can be changed or tweaked without having to re-render anything.

Step 45

Now begins the fun part – post production! There are many different workflows for this part, it all just depends on your own personal preferences. I will be showing you some of the techniques I use, to help you get started.

First make sure you have the look of your render exactly how you want it. If you want to increase reflections or refractions, add more specular highlights, now is the time!

Now merge all your visible layers into one single layer by selecting the top-most visible layer and pressing Ctrl + Alt + Shift + E. This will paste a merged copy of all your layers into a new layer called “Layer 1”.

Step 46

Making sure you’ve got the new Layer 1 selected, use the same “select color range” technique we learned earlier (steps 26 & 27) but this time on the “Object ID” layer, to select the cars.

Step 47

We want to create a new layer containing only the cars, and to do this we press Ctrl + J. Rename the new layer (initially called Layer 2) to “Cars Motion Blur” then delete the merged “Layer 1”.

Step 48

With the Cars Motion Blur layer selected, go to Filter > Blur > Motion Blur.

Step 49

Alter the direction of motion so that it matches the direction of the cars, and then choose a medium Distance (I used around 25 pixels). Click OK to finalise.

Step 50

Lower the Opacity of this layer to around 30%. The motion blur is complete!

Step 51

Now to add some chromatic aberration. Please be advised that CA is an effect that should be used as little as possible, as too much can cause your renders to look blurry and ugly. Before attempting to use this effect on your own renders I suggest you read more about this phenomenon, so that you know where and when CA should appear. When done right however, this effect can really enhance the realism of your picture.
Use the same shortcut as before (step 45) to merge your visible layers together, and rename this new layer “Merge”.

Step 52

Making sure the Merge layer is selected, go to Filter > Distort > Lens Correction. We will use this filter again in a moment to add a vignette, so remember where to find it!

Step 53

Deselect Show Grid and then experiment with the values for “Fix Red/Cyan Fringe” and “Fix Blue/Yellow” Fringe (I used +5 & -5) until you get a suitable, subtle effect. Then set your Edge mode to Edge Extension to prevent your picture from having transparent edges, although if this issue is visible, you’ve likely set your values too high already!

Although this is the native way of adding CA in Photoshop, There are many other ways people use. Some people use three different layers with the red, green and blue data and shift them manually, other people use plug-ins. I decided to show you the native way as almost all the plug-ins are commercial.

Some of the greatest commercial plug-ins for photoshop post-production are “Magic Bullet Photolooks”, 55 mm Digital Film Tools and Knoll Light Factory ”. One of my favorites is “Nik Software: Color Efex Pro”. If you can afford these then by all means they are worth it, but if you can’t you now know how to add the effect manually!

Step 54

Now that you have completed your CA effect, it’s time to add some DOF. Remember that we pasted our ZDepth render element into a new “channel? We will use that now! Start by going to Filter > Blur > Lens Blur.

Step 55

In Source choose your Zdepth channel (in my case Alpha 2) and enable the Invert checkbox. Radius controls the amount of blurriness, so experiment with that although don’t over-do it! I chose a radius of 4 and then pressed OK.

As a rule of thumb if a filter/effect is really noticeable, it is too much! (Unless of course it is some kind of an artistic approach). Maybe take some time to surf around in the CG forums and see how the pros use DOF in their images. Try to learn the best ways to add it in without hurting your own visualization. Also, remember that some clients hate DOF, so be careful with it!

Step 56

Another effect that looks really good on some pictures (and awful on others) is cross processing. Cross processing is basically playing with your color curves to achieve a more dramatic look. So add a Curves adjustment layer on top of your “Merge” layer.

Step 57

Go to the Channel drop down menu in the adjustment layer, and adjust your curves individually by selecting them one at a time. Here’’s the Red channel curve that I used.

Step 58

Here’’s the Green channel curve.

Step 59

Finally, here’’s the Blue channel curve. Remember that this is just an example; hundreds of looks can be achieved with this method, and your curves will very likely vary from one shot to another depending on the look you’’re after.

Step 60

I felt the final effect of this adjustment was too much for my taste, so I decreased the Opacity of the adjustment layer to around 65%.

Step 61

The final step is to add a vignette to your render. Some people prefer adding a black layer with an oval shaped soft mask, however I actually prefer adding it using Magic Bullet Photolooks. For this tutorial, we’ll stick with the built-in tools, so let’s add it using the Lens Correction Filter.

First we have to merge our Merge layer with the Curves adjustment layer. To do that use the shortcut Ctrl + Alt + Shift + E as before, and then, with the new layer selected, go to Filter > Distort > Lens Correction.

Step 62

I used an Amount of “-20” for this render, however it’s important that you don’’t over-do the vignette, as it just won’’t look that good!

Step 63

With that, your image is complete! I really hope you enjoyed this tutorial and learned at least a new trick or two. If you have any comments, questions or criticisms please go ahead and post in the comments below, and I’’ll be more than happy to answer you.

Don’t miss more CG tutorials and guides, published daily – subscribe to Cgtuts+ by RSS.