First we start by adding a surface Shader to the model.
Attach a mib_amn_occlusion node to the Out Color of the surface Shader.
Adjust the color and values to your linking.

The render should look something like this

Now to add the lines
Go to the SG node of  the surface Shader

Under mental ray > Contours > Enable contour render

In the render settings under the feature tab.
Contours > Enable Contour rendering.
Draw By property Difference > Enable Around all ply faces.

The final render of  Khunm

Go to Windows > Setting and Preferences > Preferences  under Applications
Located Mudbox.exe and the location of your temp folder.

Note: Mac users need to navigate to the application folder using the folder icon rather than manually inputting the path in the field.
Double-click the .app file and then navigate to Contents/MacOS/Mudbox.

Once you use the export to Mudbox option, you will notice a temp .FBX file generated in your temp location and that will be automatically deleted once Mudbox launches it, so make sure to save before closing the application.

This is the default behavior for Maya 2011 and that can be easily fixed by going to the Shader Attribute Editor > Hardware Texturing and change the texture channel from Color to Combined Textures.

Maya should now display the mesh as Mudbox does.

Render time 17:23 min

In order to incorporate the iray function in your favorite 3d app, you will need to have an installed copy of mental ray Standalone 2011.
I have been using Maya in my tests so here is an example of what I’ve done.

Copy the following 3 .dll files
cudart.dll
libiraymr.dll
libiray.dll
From: C:\Program Files\Autodesk\mrstand3.8.1-adsk2011\bin
To: C:\Program Files\Autodesk\Maya2011\bin

To get iray to appear in our render options, you will need to use the following MEL script
(You will require at least one render with mental ray in order to get the midefaultoption node to appear in your scene.)

setAttr -type "string" miDefaultOptions.stringOptions[28].name "iray";
setAttr -type "string" miDefaultOptions.stringOptions[28].value "on";
setAttr -type "string" miDefaultOptions.stringOptions[28].type "boolean";

Note:   Since iray is not currently supported with Maya, you want to make sure that you only use mental ray nodes, this way the engine can render them without giving any errors.
For the example, the IBL node is not currently supported, instead I used the mib_lookup in the camera environment slot.
For DOF I used mental ray physical lens Shader:

In the Shader SG node, turn On Suppress all Maya Shaders and turn Off Export with shading Engine.

If I would to render as is, my render would come something like this:

Render time 2:29 min

In order to control the quality of the render, you would also need to enable the progressive rendering:

setAttr -type “string” miDefaultOptions.stringOptions[34].name “progressive”;
setAttr -type “string” miDefaultOptions.stringOptions[34].type “boolean”;
setAttr -type “string” miDefaultOptions.stringOptions[34].value “on”;

setAttr -type “string” miDefaultOptions.stringOptions[35].name “progressive subsampling size”;
setAttr -type “string” miDefaultOptions.stringOptions[35].type “integer”;
setAttr -type “string” miDefaultOptions.stringOptions[35].value “1″;

setAttr -type “string” miDefaultOptions.stringOptions[36].name “progressive subsampling mode”;
setAttr -type “string” miDefaultOptions.stringOptions[36].type “string”;
setAttr -type “string” miDefaultOptions.stringOptions[36].value “sparse”;

setAttr -type “string” miDefaultOptions.stringOptions[37].name “progressive subsampling pattern”;
setAttr -type “string” miDefaultOptions.stringOptions[37].type “string”;
setAttr -type “string” miDefaultOptions.stringOptions[37].value “linear”;

setAttr -type “string” miDefaultOptions.stringOptions[38].name “progressive min samples”;
setAttr -type “string” miDefaultOptions.stringOptions[38].type “integer”;
setAttr -type “string” miDefaultOptions.stringOptions[38].value “4″;

setAttr -type “string” miDefaultOptions.stringOptions[39].name “progressive max samples”;
setAttr -type “string” miDefaultOptions.stringOptions[39].type “integer”;
setAttr -type “string” miDefaultOptions.stringOptions[39].value “100″;

setAttr -type “string” miDefaultOptions.stringOptions[40].name “progressive max time”;
setAttr -type “string” miDefaultOptions.stringOptions[40].type “integer”;
setAttr -type “string” miDefaultOptions.stringOptions[40].value “0″;

setAttr -type “string” miDefaultOptions.stringOptions[44].name “progressive error threshold”;
setAttr -type “string” miDefaultOptions.stringOptions[41].type “float”;
setAttr -type “string” miDefaultOptions.stringOptions[41].value “0.0130″;

setAttr -type “string” miDefaultOptions.stringOptions[42].name “progressive occlusion cache points”;
setAttr -type “string” miDefaultOptions.stringOptions[42].type “integer”;
setAttr -type “string” miDefaultOptions.stringOptions[42].value “0″;

setAttr -type “string” miDefaultOptions.stringOptions[43].name “progressive occlusion cache rays”;
setAttr -type “string” miDefaultOptions.stringOptions[43].type “integer”;
setAttr -type “string” miDefaultOptions.stringOptions[43].value “128″;

setAttr -type “string” miDefaultOptions.stringOptions[44].name “progressive occlusion cache max frame”;
setAttr -type “string” miDefaultOptions.stringOptions[44].type “integer”;
setAttr -type “string” miDefaultOptions.stringOptions[44].value “32″;

setAttr -type “string” miDefaultOptions.stringOptions[45].name “progressive occlusion cache exclude”;
setAttr -type “string” miDefaultOptions.stringOptions[45].type “integer”;
setAttr -type “string” miDefaultOptions.stringOptions[45].value “0″;

For more info regarding progressive rendering, please refer to Matteo Magnazzi blog.

Now we can control the quality by changing the render time:

Note:   To confirm that the iray is active you should see the following message in your output window

RC 0.10 info : progressive: using iray renderer
IRAY 0.10 info : loading iray rendering plugin from C:\Program Files\Autodesk\Maya2011\bin\\libiray.dll
IRAY 0.10 info : loading iray integration plugin from C:\Program Files\Autodesk\Maya2011\bin\\libiraymr.dll
IRAY 0.10 info : found 1 CUDA devices
IRAY 0.10 info : defaulting to use all CUDA devices

Render time 5:31 min

Going with higher settings:

Render time 10 min

Final thoughts:
iray is still beta and not intended to be used with Maya yet, so expect lots of crashes and the occasional freeze.

If you get black render, rename or nvcuda.dll in C:\Windows\System32.
Restart and try again.

In this simple scene we have a texture file with alpha assigned to a separate geometry.

If we would to render, we get this result.

A regular AO pass would look something like so

This in theory is accurate, since ambient occlusion doesn’t recognize alpha in texture files.
To get around this limitation, we will be using the mib_fg_occlusion node found under Mental ray lights in the Hypershade (don’t ask me why it’s setting there)

Note: The mib_fg_occlusion node requires having Final Gather to be turned on.

Create a surface Shader and assign the  mib_fg_occlusion to it’s out color.
This Shader will act as the AO for the geometry without transparency.

Create another surface Shader that will be assigned to the transparency object.

Open the SG node of that new shader, under mental ray > custom shader and add mib_transparency node (found under mental ray Sample Compositing)
To get the AO to appear, we just need to add the same mib_fg_occlusion node to the Input of the mib_transparency.

Finally just add the mask of your texture file to the Trnsp slot of the mib_transparency.

To get a better render of our AO, we need to increase the FG settings.

Here is a screenshot for my shading network.