I was just talking to a colleague about his problems using displacement maps with VRay, and then remembered my confusion when I first tried to work with them. So here’s a post about it!
Normally in Maya/mental ray, when you want to apply a displacement map you just create a displacement material and connect it to the shading group you want displaced. If you want to adjust the amount of displacement, you actually grade the image itself by adjusting the color gain and offset on the file node. It’s simple, it works, whatever.
You can still apply displacement like that in VRay, but there is a better and more flexible way to handle it using VRayDisplacement sets. They’re kind of like VRayObjectProperty sets, but they act as a sort of container for displacement settings instead of generic render settings and object IDs. In order to use these sets, you want to select the objects to displace with a single map, and go to Create > V-Ray > Apply single VRayDisplacement node to selection. A set will be created, visible in the Outliner.
Next up is to assign a displacement texture to the set. This means you don’t have to connect a displacement shader to any shading group; the set will handle that connection. When you select this set, the Attribute Editor will give you just two options: a checkbox saying “override global displacement,” and a plug for a displacement material. Check the box on, and then connect a texture to the displacement material (not a material, but a texture). I usually run a file texture through a Luminance node first to make the connection easier (file.outColor –> luminance.value), unless I’m using a vector displacement map in which case I’m using color information instead of just luminance or alpha.
So where are all the displacement options? You have to add them. If you don’t change anything, the displacement will use the default values set in the VRay render settings under Settings > Default Displacement and Subdivision. This defaults to an edge length of 4 pixels, a maximum subdivision number of 256 (this is a lot of subdivisions!!), and a displacement amount of 1.0 (which is usually way too high). These are terrible values for most scenes, the displacement will look grossly exaggerated and it will take forever to render.
In order to tweak the settings, you need to add the appropriate attributes to the VRayDisplacement set. With the set selected, open the Attribute Editor and select Attributes > V-Ray > Displacement Control and Attributes > V-Ray > Subdivision and Displacement Quality. Now you have a ton of options to play with, the most important of which are Displacement Amount (color gain), Displacement Shift (color offset), Edge Length, and Max Subdivs. I recommend starting with a displacement amount of 0.1-0.5, and a max subdivs of maybe 16-32 before starting to increase those settings.
I have no idea why these attributes have to be added manually, but hell, it’s still better than mental ray.