Over the last few weeks I’ve been writing my own version of a layered texture node for Maya and VRay. I’ve always missed the overlay blendmodes and the interface drove me insane.
So here it is, a layered texture node that works with Maya and Vray:
Download Azog’s Bane v1.0

Apparently this only works with Maya 2016, feel free to get in touch with me to get the source files if you want to compile this yourself or work off of it.

A little bit about the process:
First up I looked into creating plugins for Maya. There are two different types of Maya plugins, command plugins and dependency graph plugins. I first wrote the node in Python and everything seemed to work fine until I plugged it into a VRay shader. That’s when I realized that you also have to write the VRay counter part of the plugin so VRay can interpret the plugin on render time. The documentation for those plugins can be found here: C:\Program Files\Chaos Group\V-Ray\Maya 2016 for x64\docs. That meant I had to learn the basics of C++ which was ok since there are tons of example plugins. Unfortunately VRay did not bring any support for compound attributes which forced me to spell out the individual layer attributes (color0, color1, etc.) instead of having a dynamic compound array (layer[0].color, layer[1].color, etc.). I hope they bring that support soon. Then I realized that my Maya plugin only worked with the software renderer and I figured out that you have to write a hardware shader override to have it work in the Viewport 2.0 and Harware 2.0 render engine which is used to display the Material Viewer.
This is done in the xml files. About that time I also rewrote the plugin in C++ to hopefully improve the performance. The azogsBane_parameters text file is then used to tell vray how to interpret the maya plugin.

Unfortunately there are some bugs that I didn’t get around to fix yet:
1. The VRay preview crashes as soon as you resize the window.
2. The Color and Luminosity blend mode do not work.
3. Moving layers up and down with some of them connected and some not, can cause the colors to be reset.

If you have and comments or questions, please let me know.