Correct me but i think there is no way to tell the ambient occlusion/environment light to be aware of transparent materials, regardless of it's type (mask, z, ray). Parts of the object behind transparent surfaces get rendered as dark as if said surface would be not transparent at all. this applys for aproximated AO/EL as well as for raytraced.