Mar 212016

I’m trying to build a system right now that can automatically substitute environment variables for paths on a per-scene basis. When a user opens a Maya file, I want to parse that file and set temporary environment variables called $SHOW, $SEQ and $SHOT that point to the show’s root folder, the sequence’s folder in that show, or the shot in that sequence. The variable names and paths I’m trying to get are pretty specific to the current pipeline I’m working on, but this essentially lets me use relative paths in Maya but WITHOUT using Maya’s outdated and inflexible workspace system. I don’t want an individual Maya project for every single shot and asset in the entire show, and I don’t want all of my Maya scenes to be forced to live within a single workspace, either.

I’ve solved this problem before by using symbolic links to basically force all of the Maya projects generated for every single shot and asset to link to the same place. This makes for a pretty busy-looking file system, though, and symlinks only work on Unix-based servers (i.e. not Windows). This system I’m building now looks a lot more like Houdini’s global variables… as long as I define $SHOW or $SHOT or whatever, I can path a reference or a texture to $SHOT/assets/ and it doesn’t necessarily have to be within a Maya workspace, and it’s also not an absolute path.

Read more below… Continue reading »

Jan 212015

I’m working on a production right now that involves sending absolutely enormous animated meshes (6.5 million polygons on average with a changing point count every frame) out of Houdini and into Maya for rendering. Normally it would be best to just render everything directly out of Houdini, but sometimes you don’t have as many Houdini licenses (or artists) as you’d like so you have to make do with what you have.

If the mesh were smaller, or at least not animated, I’d consider using the Alembic format since V-Ray can render it directly as a VRayProxy, but for a mesh this dense animating over a long sequence, the file sizes would be impossibly huge since Alembics can’t be output as sequences. Trying to load an 0.5 TB Alembic over a small file server between 20 machines trying to render the sequence and you can see why Alembic might not be ideal for this kind of situation.

Hit the jump below to see the solution.
Continue reading »

Mar 252014

I’ve been messing around with using Ptex in VRay for Maya and ran into a particularly weird little problem involving a normal map exported from Mudbox. It’s probably easier to show it than to tell about it (Fig. 1):

bad ptex bottle

Fig. 1. Check out the wax wrapper up top… all chunky and gross.

That wax wrapper looks terrible… not at all like the original sculpt. The mesh is set to render as a subdivision surface (using the custom VRay attributes from the Attribute Editor), but the details are all mangled like it’s not subdividing the surface at all.

Even weirder is what happens when I only render a small region of the image (Fig. 2):

bad ptex with region render

Fig. 2. The details are suddenly cleaner! All I changed was enabling region render.

This was really confusing, and although I’m still not 100% sure what’s going on in VRay to cause this (I’m fairly certain this is a bug), there is at least a solution.

What’s probably happening here is that choosing to render this mesh as a subdivision surface is changing the point count of the object BEFORE any Ptex information is applied to the mesh during rendering. Ptex is very sensitive to your geometry… any change in point count could potentially break things. You’re not allowed to polySmooth objects that will have Ptex textures, for example, or Maya won’t know what points to assign Ptex information to.

The way to get around this is to put the object into a VRay displacement set. Assign displacement and subdivision attributes to the displacement set like you normally would, but don’t attach a displacement map, and make sure the Displace Amount is set to 0. Rendering this way gets you the more correct image (Fig. 3):

correct ptex

Fig. 3. This looks a lot more like the original sculpt from Mudbox.

This seems like buggy behavior to me more than some technical thing I’ve overlooked, but thankfully the workaround is pretty minimally difficult. If you have any better insight as to what exactly is happening here or if there’s a less hacked way to prevent it, I’d love to hear it.

Feb 112013

Many operations in Maya will run faster if Maya doesn’t have to refresh the viewport while running them. For example, if you switch the viewport to only show the Graph Editor before baking animation, or caching particles or Alembic geometry, the operation will happen much faster than if Maya had to actually display the geometry for each frame that it’s being baked. There is probably a way via the API to tell Maya’s viewport not to refresh, but since I don’t know shit about the API, here’s a workaround using a few of Maya’s less-documented MEL commands and some Python.

I wrote this method assuming that artists would want to cache out information frequently without it disrupting their workflow. That means that I needed to first store the user’s current panel layout, then switch it to something that doesn’t require a refresh on every frame (like the Graph Editor), and then restore the previously restored layout.

I prefer coding in Python, but some of the procedures I’m running are MEL functions that are found in scripts/startup and so they’re not documented and they don’t have Python equivalents. A hybrid approach is the best way to handle it, since MEL is terrible.

Click for some Python code… Continue reading »

Sep 292012

I’m working on a project right now that involves exporting cached geometry from Houdini to Maya. The Alembic node makes that a fairly painless process now that Houdini and Maya both support Alembic import/export, although it turns out that getting any data other than point positions and normals is kind of a hassle. I tried renaming the Cd point attribute in Houdini to all kinds of things in the hopes that Maya would recognize the data, but Maya wasn’t having any of it. That’s when I checked out the script editor in Maya a little more closely and saw this:

// Error: line 0: Connection not made: 'output_AlembicNode.prop[0]' -> 'subnet1.Cd'. Data types of source and destination are not compatible. // 
// Error: output_AlembicNode.prop[0] --> subnet1.Cd connection not made //

Maya is creating this “Cd” attribute on the mesh, but it has no idea what to do with the data so it throws an error. The Alembic node, though, still contains that data, as the 0th index of the array “prop.” Now all you have to do is get that data from the Alembic node onto the vertex color somehow…

The SOuP plugin for Maya has the answer. There is a node called “arrayToPointColor” that will read an array of data and apply it to the point color of the mesh it’s connected to. Create the arrayToPointColor node, and feed it your geometry (mesh.worldMesh[0] –> arrayToPointColor.inGeometry) and then feed it your data array from the AlembicNode (AlembicNode.prop[0] –> arrayToPointColor.inRgbaPP). If you have more than one point attribute exporting from Houdini, you may want to check the script editor to make sure you know which index of AlembicNode.prop you are supposed to be connecting.

Finally, make a new mesh node and connect arrayToPointColor.outGeometry –> newMesh.inMesh. If you were to select some vertices and look at their properties in the Component Editor, you should see values attached to the “red” “green” and “blue” vertex attributes.

All that’s left to do at this point is to connect this color data to a texture that can read it. In mental ray, you’d create a mentalrayVertexColors node, and connect newMesh.colorSet[0].colorName –> mentalrayVertexColors.cpvSets[0]. If you don’t see a colorSet[0].colorName property on your mesh, try selecting the mesh, then go to Polygons > Colors > Color Set Editor. You should see a colorSet1… just select it, click “Update” and you should have the property you’re looking for. Then connect the mentalrayVertexColors node to any shader. See Fig. 1 for the example network.

Fig. 1: Node Editor network connections to link Alembic attributes to vertex color.

You can also just remove the middleman entirely at this point, and delete the original shape node. Then connect the AlembicNode.outPolyMesh[0] to arrayToPointColor.inGeometry. This is probably a good idea if only because it will stop Maya from throwing annoying errors every time you select the geometry because of that missing “Cd” connection.

Jun 202012

I just ran into a hideous problem when I was trying to bake out cameras and animation from one scene, and get the animation into another. I always keep my lighting scenes separate from the animators’ scenes so they can’t mess up my shots. Unfortunately, the animator had used a Scene Time Warp, and then baked out the animation, which of course never quite lined up correctly and we couldn’t figure out what was going on. Turns out that keys baked under a Time Warp will not actually bake the Time Warp itself.

Here’s a script, then, to bake the effects of the time warp on all selected objects. It then (optional, but recommended) deletes the time warp from the scene entirely:

Continue reading »

Feb 222012

I’ve been working on some tools recently that open groups of Maya scenes in batch mode, running Python or MEL operations on each file (for example, to quickly add a render layer and apply materials to a large group of objects at once, or to get the names of all the references in a scene) and then either returning data or saving the file. This sort of thing could be done in the usual windowed Maya interface, but this is slow and means the user has to sit there and watch Maya open a bunch of files, run operations and all that. Running scripts on lots of files at once is much faster if you’re running Maya in standalone (“batch”) mode.

There are other, more informative posts on how to get Python running in standalone mode (I learned a lot from this one in particular), but the gist of it is that you start by running /bin/mayapy.exe, and then initialize the Maya Python interpreter by calling the following function:

import maya.standalone as standalone

Once that’s done you can start scripting like normal, starting with import maya.cmds as cmds and working from there. Some commands are not available in standalone mode, especially anything to do with UI.

Anyways, while working on a script that is meant to process huge heaps of unsorted Maya files, I realized that crashes were going to be a frequent problem when opening ancient, broken scenes, so I couldn’t just load an instance of Maya in standalone mode from a script and let it run through everything in a loop. I needed to be able to process one file at a time, and then either return some useful information if the process is successful, or let me know if Maya crashes so I can avoid that file later on or at least mark it.

To do this, I used Python’s subprocess module. Again, I’m not going to go into a ton of detail about the module, it’s pretty complicated, but I’ll give a quick example of how I’m using it to return information from a Maya process when I am calling the script from Maya (or any other program, really).

Let’s say I want this script to add a new render layer to several scenes, and add all meshes in each scene to that new layer. First, I’ll write the script that actually creates the layer and assigns objects.

import sys
import maya.standalone as std
import maya.cmds as cmds
filename = sys.argv[1]
layername = sys.argv[2]

The sys.argv[1] and sys.argv[2] are easy ways to pass variables to an external script. sys.argv[n] just means the nth command line argument, so sys.argv[1] is the first command-line argument when running the script (after calling for the script name, for example, by running mayapy.exe When we call this script, we’ll pass it a file to open, and a layer name to add.

def addRenderLayer(filename,layername):
        newLyr = cmds.createRenderLayer(n=layername,empty=1,makeCurrent=1)
        meshes ='mesh')
        xforms = []
        for i in meshes:
            xf = cmds.listRelatives(i,p=1)[0]
        return newLyr
    except Exception, e:


Here we’re creating an empty render layer based on the “layername” argument, then we’re getting all the meshes in the scene. In most scenes, people add transforms to render layers, not their individual shape nodes, so we’ll do this by creating an empty list “xforms,” then going through each mesh in a loop and finding their first related transform node (hence the listTransforms(i,p=1)[0] since listTransforms returns a list) and appending it to “xforms,” then adding “xforms” to the new render layer. Then we just save the file and return the name of the layer (I just like having return values for my functions).

We also write the name of the new layer to stdout. I’ll go into more detail about this later, but it lets us send information back to the program that calls this script later on. There’s also an exception handler that will write any error encountered to the stderr stream, and will exit with code -1, which is sort of a convention for “this program exited abnormally.”

Save this file as somewhere in your default scripts directory.

Next, we’re going to make the process that we run from Maya…

import maya.cmds as cmds
import subprocess
# replace mayaPath with the path on your system to mayapy.exe
mayaPath = 'c:/program files/autodesk/maya2011/bin/mayapy.exe'
# replace scriptPath with the path to the script you just saved
scriptPath = 'c:/users/henry/desktop/'
def massAddRenderLayer(filenames,layername):
    for filename in filenames:
        maya = subprocess.Popen(mayaPath+' '+scriptPath+' '+filename+' '+layername,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
        out,err = maya.communicate()
        exitcode = maya.returncode
        if str(exitcode) != '0':
            print 'error opening file: %s' % (filename)
            print 'added new layer %s to %s' % (out,filename)

Okay, a lot going on here. The subprocess.Popen function is calling mayapy.exe with three parameters: first, the script we want to run ( followed by the two arguments that script wants (the filename and the layer name), then the input stream and the output stream which are both set to subprocess.PIPE. I’m not quite knowledgeable enough about the subprocess module to go into specifics, but basically when you run a process there are three “streams”: the input, output and error streams, usually called stdin,stdout and stderr. When you set the input and output streams to subprocess.PIPE when calling Popen(), you can be sure that the process will receive any input stream you give it (which we are not doing here; instead we’re using the sys.argv method shown above) and also communicate back any output it gives.
The next procedure, communicate(), returns two variables: stdout and stderr. We can use this for debugging or monitoring progress. Normally in the Maya script editor, if your script fails for whatever reason, you will get some kind of (possibly) useful information back from an error. However, you won’t have any idea what goes wrong with a subprocess unless you are listening to the error stream.
communicate() does a couple other awesome things. First, and most importantly, it waits for the subprocess to complete before continuing with your script. This is REALLY IMPORTANT, as you definitely don’t want a script like this to continue running until the process is finished. Second, it sets a few variables on the Popen() object, one of which is Popen.returncode. If you have ever slogged through Maya render logs and noticed lines like “Maya exited with status: 210” or “Maya exited with status: 0,” those numbers are exit codes. Status 0 means the program exited normally; any other exit code means something went wrong. Since I want to be notified if something goes wrong when opening or editing a scene, I check to see if Popen.returncode is anything other than zero, and if it is, I print out the error stream and leave a notice that says something went wrong. If everything worked, I just want to know that it worked, and so I print a string. Since I wrote the output layer to sys.stdout, when I read stdout from the subprocess, I get the name of the newly created layer. I do this since the layer might not come out with exactly the name I wanted if another node in the scene had the same name as the new layer!

Now, to actually run the script. If you were to run massAddRenderLayer() from Maya, as in the example, you’d call it like this:

# define a list of filenames to iterate through
filenames = ['file1','file2','file3']
renderLayerToAdd = 'someNewLayer'

# run procedure, assuming you've already defined it
massAddRenderLayer(filenames, renderLayerToAdd)

I’ve been using the subprocess module along with Maya standalone more and more as I make more complex applications. It’s incredibly useful to be able to process many files at a time… if you get any good ideas, you know where to post them.

Jan 302012

I was just talking to a colleague about his problems using displacement maps with VRay, and then remembered my confusion when I first tried to work with them. So here’s a post about it!

Normally in Maya/mental ray, when you want to apply a displacement map you just create a displacement material and connect it to the shading group you want displaced. If you want to adjust the amount of displacement, you actually grade the image itself by adjusting the color gain and offset on the file node. It’s simple, it works, whatever.

You can still apply displacement like that in VRay, but there is a better and more flexible way to handle it using VRayDisplacement sets. They’re kind of like VRayObjectProperty sets, but they act as a sort of container for displacement settings instead of generic render settings and object IDs. In order to use these sets, you want to select the objects to displace with a single map, and go to Create > V-Ray > Apply single VRayDisplacement node to selection. A set will be created, visible in the Outliner.

Next up is to assign a displacement texture to the set. This means you don’t have to connect a displacement shader to any shading group; the set will handle that connection.  When you select this set, the Attribute Editor will give you just two options: a checkbox saying “override global displacement,” and a plug for a displacement material. Check the box on, and then connect a texture to the displacement material (not a material, but a texture). I usually run a file texture through a Luminance node first to make the connection easier (file.outColor –> luminance.value), unless I’m using a vector displacement map in which case I’m using color information instead of just luminance or alpha.

So where are all the displacement options? You have to add them. If you don’t change anything, the displacement will use the default values set in the VRay render settings under Settings > Default Displacement and Subdivision. This defaults to an edge length of 4 pixels, a maximum subdivision number of 256 (this is a lot of subdivisions!!), and a displacement amount of 1.0 (which is usually way too high). These are terrible values for most scenes, the displacement will look grossly exaggerated and it will take forever to render.

In order to tweak the settings, you need to add the appropriate attributes to the VRayDisplacement set. With the set selected, open the Attribute Editor and select Attributes > V-Ray > Displacement Control and Attributes > V-Ray > Subdivision and Displacement Quality. Now you have a ton of options to play with, the most important of which are Displacement Amount (color gain), Displacement Shift (color offset), Edge Length, and Max Subdivs. I recommend starting with a displacement amount of 0.1-0.5, and a max subdivs of maybe 16-32 before starting to increase those settings.

I have no idea why these attributes have to be added manually, but hell, it’s still better than mental ray.

Jan 272012

I’ve been working on a set of tools for making flexible, twistable arms and legs quickly. I haven’t rigged in a long time, so this has been an arduous process, but I’ll post notes about what I run into along the way.

One problem that I remember having over and over (since I’ve never rigged frequently enough to ever learn the correct method) is that when applying an IK handle to a joint chain, and then applying a pole vector constraint to that IK handle, the joints can sometimes move subtly. It’s not always a big deal, but it can be a problem when you’re trying to have multiple matching skeletons for FK and IK control, and suddenly your rotations don’t line up anymore.

The fix, as it turns out, is really easy. Before you apply the pole vector constraint, you need to place your controller (whatever is going to drive the pole vector) exactly between the start of the IK chain, and the end effector, e.g., the shoulder and the wrist. This is quickly done by selecting the start and end joints, and then the controller, and creating a point constraint (do not maintain offset). Then delete the constraint. Next, use an aim constraint to point the controller directly at the middle joint (say, the elbow). Delete the constraint. Then you just need to move the controller along its pointing axis a little ways, to push it away from the bones. For example, if your aim vector for the constraint was +X, then push your controller a few units forward in its local X axis. Now you can create the pole vector and the joints shouldn’t snap at all.

Aug 222011

I am a huge advocate of using shared render layers in a file referencing pipeline. It’s probably the only reasonable way of allowing lighters, animators, riggers and FX artists to work on shots and assets simultaneously. That being said, it isn’t without its weaknesses (many of which I talked about in an earlier post). The latest one I encountered happens with scenes with TONS of objects… not necessarily a high polygon count, but just lots and lots of individual DAG nodes. Take a look… Continue reading »