Tutorial: Ambient Occlusion in Maya - by Alchemist101

[benclark]

AMBIENT OCCLUSION IN MAYA8 and PHOTOSHOP

This tutorial covers the basics of how to setup and bake ambient occlusion maps in Maya8. The process is similar for previous versions of Maya only the menu names will be different.

As with most things in Maya there are probably many different ways to do this but this method works well for me and is reasonably quick to setup.

1. SETTING UP THE SHADER NETWORK

- First open the hypershade window and from the Maya nodes list MMB drag a Surface Shader node into the work area

- Now click on the "Create Maya Nodes" button at the top of the nodes list and change it to MentalRay nodes.

- Then under the textures tab MMB drag an 'mib_amb_occlusion' node onto the work area

- Now connect the 'OutValue' of the Occlusion node to the 'OutColor' of the surface shader. This is the default connection between these 2 nodes so you can do this by simply holding down Ctrl and MMB dragging the occlusion node over the shader node.

2. SETTING UP YOUR SCENE

- In order to get good results from Ambient Occlusion the scene environment must be white. To do this open the Outliner window and select the Persp camera. In the attribute editor under the Environment tab slide the Background Color to 100% white

- Now open the Render Globals window and select MentalRay as your renderer and Production as the quality preset. Now change the multi-pixel filter to "Lanczos".
(this will automatically change your Quality preset to 'Custom' like in the picture)

- Now close the render Globals window and assign the 'Surface Shader' you have setup to your model. It should turn the model completely black.

[/size]

3. TWEAKING

- Frame your model and hit render. It should render quickly and look quite grainy and crap at this stage, like this....

- Now we need to tweak the settings a little. Open the attribute editor and then in the Hypershade window click on the Ambient Occlusion node. The Occlusion node's attributes will now appear in the attribute editor. These are the settings I use for most objects...



- The first setting to change is 'Samples'. Increasing this will remove the grainy look but will drastically increase render times. 256 samples is good for most objects.

- Do not change the 'Bright' and 'Dark' settings. These affect how light/dark the occlusion value is, but it is a lot better to render with the default extremes and tweak them in Photoshop at a later stage if you need to.

- The spread setting affects how far the occlusion shadow spreads over the surface. Again the default setting of 0.800 is good for most objects, any lower tends to be too extreme, and a high spread can cause darkening problems when it is multiplied over your colour layers in Photoshop.

4. BAKING TO TEXTURE

Once you have the occlusion looking good in a render, the next stage is to bake it to a texture.

- First make sure you have specified a project folder. To do this select File> Project > Set, and select a file. This file is where your baked texture will automatically be saved. If you forget to set a project folder the bake will be saved in your default projects file in your My Documents folder.

- Make sure that you have UV mapped your model.
- Make sure you check the face normals of your model are facing the right way or the bake will produce a black picture
- Make sure you have softened the edges of your model and created hard edges where needed
- Save a copy of your scene at this stage as Maya can crash while baking to texture
- Now choose the rendering menu set and select Lighting/Shading > Batch Bake (Mental Ray) > Options box
These are settings that I use and have never had to change them, so they should work for most objects.....

- Select your object in the viewport and hit the 'Convert and Close' button to start baking the occlusion. When you do this Maya will not respond to any clicking, sometimes the screen will go blank and it generally looks like it has crashed but don't worry it will come back eventually. This can take anywhere from a few minutes to a few hours depending on how complex the scene is.

5. USING YOUR BAKED OCCLUSION MAP

- Once the bake is completed open the project file that you specified earlier.

- In that folder select RenderData > MentalRay > Lightmap > select your baked texture file and open this with Photoshop.

- Drag the Occlusion bake on top of your model's texture layer(s) and select Multiply as the layer blending mode. You may also want to apply a slight Gaussian blur to your occlusion layer, especially if it being used on an organic surface. For hard sharp edged surfaces like this one no blur was applied.


This picture shows the object and a basic 3-light setup with and without occlusion in the texture.
The effect of the AO map is to fake GI lighting in real time.

6. BAKING OCCLUSION FROM HIGH RES TO LOW RES MODELS

Ambient occlusion maps can also be used to compliment normal maps. The occlusion values from a high poly model can be baked to the UV layout of a low poly model. This helps to sell the normal mapped details if used in a subtle way.

This function is found in the transfer maps window under lighting/shading in the Render menu set.


As an example I will use this alien head model with normal map already extracted......

Now we will bake an ambient occlusion map to compliment the normal map...

In the Transfer Maps window....

1. Select your low poly mesh in the viewport and click add selected to load your target mesh

2. Do the same for your high poly mesh

3. Click the "Ambient" box

4. Use the settings shown (remember to specify a folder to save the map into)

The resulting Occlusion map looks like this....

Baking occlusion larger than 512x512 using this method can take forever so be prepared to wait a while


This map can then be taken into Photoshop and multiplied over the model's texture in the same way as before....

[Thonz]

Wow, verry nice tutorial.
I'm a max user, but it helped me understand AO a lot better.
Thanks!

[CopticMike]

AAAHHHH THank you SOOO MUCH!!! Im a maya user, and you really helped me out, now the occulsion mystery, has been demystified! Thank you again

[st0lve]

To me, this looks kinda like a lighting map from 3ds max's render to texture feature, if you use a skylight that is.
Is there any difference? :O

[|Buddy|]

More like MR render AO in RTT but yeah You're right.

[SpaceRabbit]

Dumb question, but does your hi-res model require UV mapping, or just the low res. I was reading the documentation for this in Maya 7 and they kept stressing that both models had to have cleam UV mapping which seems like such a waste of time to have to UV both your hi-res and low res.

Thanks for the info, and great tutorial!

[benclark]

Thanks for the feedback guys, SpaceRabbit - nope you dont have to lay out the UVs on the high poly model

[SpaceRabbit]

Snap, thats what I wanted to hear! Thanks Alchemist!

[ghib]

Anyone have any ideas whether it's possible to bake an AO map from a normal or displacement map rather than using a hi-res mesh. Current version of Illuminate Labs Turtle is able to do this as it bakes maps at rendertime. Would be handy to do this instead of importing massive polycount objects into maya.

[Nysuatro]

So people use a ambiant occlusion for having shadows in there texture?