Quad Patch Tessellation in Unity

Howdy there, Defectfans!

Edit: This shader does not work in Unity 5. Naturally, a lot has changed in the Unity shader codebase, and this code simply doesn’t compile anymore. It probably could be fixed easily, but it was a hack at the time, and I haven’t revisited it since its original posting. One more update of note: at some point around Unity 4.6/5.0, they re-introduced the “Keep Quads” import setting on mesh importers, which makes a large portion of what you read here obsolete!! I believe that you can get the best of both worlds in Unity 4.6, where the shader should compile, and you can also import meshes with quads natively, and you don’t need all of that OBJ importer nonsense.

By popular demand, I’m posting the source and explanation of the tessellation shader on display in my HIT YOUTUBE VIDEO!!! OK 1,754 views hardly constitutes a hit in YouTube terms, but some people have commented and asked for source, so here is! Wait, first a little explanation of WTF is going on and why I went through all the work to hack together quad-based tessellation when Unity already has some perfectly good tessellation shaders with source available. A very helpful discovery (pointed out by a wonderful chap on the IRC channel) is that by adding #pragma debug to any Unity shader will produce shader soucre when you click to Open Compiled Shader. This was truly helpful in figuring this all out, and really on every bit of shader work I’ve done on shaders since then. As a disclaimer/warning, everything that I’m talking about in this post requires Unity 4 and a DirectX 11 capable graphics card on Windows. I honestly haven’t tried any of this on OS X but I’m pretty sure that all of this stuff uses DirectX-specific shaders. The effect might be possible in OpenGL, but I don’t think it is possible in Unity using OpenGL. I’m sure Aras will have that taken-care-of soon enough though :)

First of all, the why was quad patch tessellation so much better than tri-patch? Perhaps a diagram can help (courtesy of a vital source which helped me work this out):

What quad tessellation might look like

What triangle tessellation might look like

What triangle tessellation ends up looking like

So what’s going on there? First I should mention that the first image isn’t exactly what you might see with quad patch tessellation–or more precisely isn’t something you would see with my shader. That said, it wouldn’t be a difficult result to produce if it was worth producing. Specifically, what you would see is that square quad being divided evenly in X and Y, as opposed to unevenly (you may notice there are 4 divisions in X and 3 in Y). The difference between the second two image has to do with how tessellation levels “repeat.” If you had to tessellate the middle image further, what would you do? Split each triangle into 4 smaller triangles? That’s not very convenient if you maybe just want to tessellate “a bit,” which the third representation allows. If you’re confused how, just play around with the “Edge Length” parameter of Unity’s built-in tessellation shader to see what’s going on.

It turns out that this method of tessellation works great on meshes that are “naturally” triangulated but it puts verts in odd places on meshes that are supposed to be quads. What do I mean by “supposed” to be quads? Think about what the third diagram would look like on a triangulated square. I’d get out a drawing program and draw this but I trust your imagination. If I get the time, I’ll come back later and draw it. Anyway, the case that I was working on was a mesh exported out of Mudbox with a displacement map (later a vector displacement map). Mudbox works with quads, meaning that when you move up/down in levels of subdivision, it simply takes a rectangle and turns it into 4 similar (in a Geometry sense) rectangles. When you work with your model in Mudbox, you’re working with such a topology. Well, you’re technically working with a displacement map, but Mudbox is showing you a mesh that is so-subdivided.

What happens when you apply the mudbox displacement map to triangle patch tessellation? Well, check out the difference:

Yes, I know I mis-spelled “tessellation” on the right there. I don’t have the original anymore so it will forever be the case

It’s impossible to say which is “better” except as a matter of preference. Please ignore the difference in lighting and the fact that one uses a normalmap and the other doesn’t. This was pretty early on in the process, and I hadn’t added the bump channel yet. What you’re supposed to see is how the example on the left has much more even subdivision, which ends up matching the original (not shown… d’oh!) much more closely. Pay particular attention to the “ridges” on his neck. In the triangular tessellation, they are kind of “jagged” due to where the “extra” vertices end up being placed. Granted, you might notice that there are a whole lot more triangles in the legs and feet on the left vs. the left. I ended up adding a distance parameter to later versions of the shader, which helped a lot, but it’s still not perfect.

So, if you’re not yet convinced that quad patch tessellation is superior, well, that’s all I’ve got. I’m perfectly happy to admit that I might be doing something wrong, but my first instinct that was quads would be the answer, and I ran with that. So, now you can see why I was convinced to go down the insane road of circumventing Unity’s mesh import process (which some would argue is half of why you should use Unity in the first place). Here’s how:

After deciding that I wasn’t missing something, and that you can’t just import a mesh with quads, I started to despair. After some back-and-forth on the forums, and learning some other stuff I can’t talk about, I’m pretty sure that I’m not just doing something wrong, and quad import really isn’t supported (though quad meshes themselves are). After some quick googling, to my rescue came a GitHub project for importing OBJ meshes at runtime into Unity. Thanks a bunch hammmm @ github! With a couple of quick modifications, which can be found in my own fork of that repo, I got real-live quad meshes in Unity. I’ll also point out that this was a very quick and easy solution that I turned to after trying to convert tri-based meshes back to quads after normal import. I tried a few tactics like just joining triangles with the “first available” adjacent triangles, and doing this while filtering for coplanarity, neither of which worked for a sufficiently complex mesh. Wasted a lot of time there 😛

If you’re looking for a better explanation of my procedure for getting quad-based meshes into Unity, let me know, and I’ll draft up another post about just that. Suffice it to say it was an essential step, but is immaterial to this post. Now on to the shader. I won’t claim any ownership or authorship of this code. It was shamelessly lifted from this pastebin. It’s not line-for-line the same, but I pretty much let that example hold my hand. To my everlasting shame, I can’t for the life of me find the post that linked me to that PB! I think it might have been on the Unity forums, or a blog somewhere. I’ll include the text in this post, just in case the original dies at some point. I wish I could credit a person and not an anonymous PB, but in lieu of a person, I still want to give credit where it’s due. If this is your code, come forward silent rogue, and receive thy prize!

Moving on, let me break down my understanding of this shader, and my own. Tessellation shaders take advantage of a DX11-specific shader pass called the Domain pass. I won’t pretend to understand more than I actually do. Essentially the “missing link” as it were was the ability to create extra vertices in the shader, rather than modifying the actual mesh data. How exactly this differs from a geometry shader, which is possible with DX9, I’m honestly not sure. Either way, I’ll break the shader down by its different passes:

Vertex: Really nothing of note here. I used a pass-through geometry shader which only applies the ModelView matrix transformation equally to all vertics.
Hull: This pass provides information to the Domain shader. The only “computation” that is done here uses a distance parameter to vary the “amount” of tessellation (number of subdivisions). You don’t see that in most of the earlier images/videos, but it’s there now! :) For some reason this is broken up into the actual hull function (one line, just like vertex) and a constant function. Yeah, I guess I should learn more about stuff before writing it up =\
Domain: Here’s where the “magic” happens. Essentially, this function is responsible for “splitting up” all of the mesh data for delivery to the fragment function. It takes in a parameter that I apparently don’t use (go figure), a float2 that represents “how far” along either edge of the quad the we are here (I assume “at this pixel”), and an array of vertices for “source data.” In this case, since the shader works on quad meshes (the whole point here!) it takes in 4 verts per pass of the domain shader, representing one quad of the mesh. For each of the various data points, we essentially call lerp 3 times to interpolate the value. For example:

1
2
3
4
float3 topMidpoint = lerp(patch[0].vertex, patch[1].vertex, UV.x);
float3 bottomMidpoint = lerp(patch[3].vertex, patch[2].vertex, UV.x);
 
float3 pos = lerp(topMidpoint, bottomMidpoint, UV.y);

TopMidPoint and BottomMidPoint are the midpoint positions along the top and bottom edge (using UV.x as the proportion for how far along), and pos is the midpoint between those two points. A diagram:
lerp1
0–|–1
| _ | lerp3
| |
3–|–2
lerp2

There’s a better one out there, or I could draw one, but yeah. I need this post to not take up much more of my time.

The above Lerp process is repeated for Normals, UVs, and tangents, after which some other work happens which I took from various decompiled bump map shaders.
Fragment: Again, the fragment shader, while it looks complicated, is actually rather standard. What I didn’t get from the existing Tessellation shader in StandardAssets I got from decompiled shaders.

And now, without further adieu, Appendix A: The quad patch tessellation shader. You might notice that the code is there twice in a row! This is apparently what you have to do if you want normal maps to respond to more than one light. I’m sure that I’m also missing a whole lot of fallback code, extra lighting stuff, and whatever. This was a very quick effort put together over a few days. What you see is what you get!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
Shader "Tessellation/Vector Displacement Quad" {
	Properties {
		_MainTex ("Main Texture", 2D) = "white" {}
		_Color ("Main Color", Color) = (1,1,1,1)
		_SpecColor ("Specular Color", Color) = (0.5, 0.5, 0.5, 1)
		_SpecMap ("Specmap", 2D) = "white" {}
		_BumpMap ("Normalmap", 2D) = "bump" {}
		_DispTex ("Disp Texture", 2D) = "gray" {}
		_TessEdge ("Edge Tess", Range(1,25)) = 2
		_Displacement ("Displacement", Range(0, 1)) = 0.1
		_fadeDist ("Start Distance", Range(0, 15)) = 0.1
	}
	SubShader {
		Pass {
			Tags {"LightMode" = "ForwardBase"}
 
			CGPROGRAM
			#pragma vertex vert
			#pragma fragment frag
			#pragma hull hull
			#pragma domain domain
 
			#pragma fragmentoption ARB_precision_hint_fastest
			#pragma multi_compile_fwdbase
			#include "HLSLSupport.cginc"
			#include "UnityShaderVariables.cginc"
			#define UNITY_PASS_FORWARDBASE
			#include "UnityCG.cginc"
			#include "Lighting.cginc"
			#include "AutoLight.cginc"
 
			float _TessEdge;
			float _Displacement;
			float _distFalloff;
			float _fadeDist;
			fixed4 _Color;
			half _Shininess;
 
			sampler2D _MainTex;
			SamplerState	sampler_MainTex;
			uniform float4 _MainTex_ST;
			uniform float4 _BumpMap_ST;
 
			Texture2D _DispTex;
			SamplerState	sampler_DispTex;
			//Texture2D _BumpMap;
			sampler2D _BumpMap;
			SamplerState	sampler_BumpMap;
			Texture2D _SpecMap;
			SamplerState	sampler_SpecMap;
 
			struct Input {
				float2 uv_MainTex;
				float2 uv_BumpMap;
			};
 
			void surf (Input IN, inout SurfaceOutput o) {
				fixed4 tex = tex2D(_MainTex, IN.uv_MainTex);
				o.Albedo = tex.rgb * _Color.rgb;
				o.Gloss = tex.a;
				o.Alpha = tex.a * _Color.a;
				o.Specular = _Shininess;
				o.Normal = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap));
			}
			struct appdata {
				float4 vertex : POSITION;
				float4 tangent : TANGENT;
				float3 normal : NORMAL;
				float2 texcoord : TEXCOORD0;
				float2 texcoord1 : TEXCOORD1;
			};
			struct tess_appdata {
				float4 vertex : POS;
				float4 tangent : TANGENT;
				float3 normal : NORMAL;
				float2 texcoord : TEXCOORD0;
				float2 texcoord1 : TEXCOORD1;
			};
			struct PS_INPUT
			{
				float4 pos   : POSITION;
				float3 normal     : NORMAL;
				float4 tangent     : TANGENT;
				float4 uv   : TEXCOORD;
				float3 lightDir : TEXCOORD1;
				float3 viewDir : COLOR;
				fixed3 vlight : TEXCOORD2;
				LIGHTING_COORDS(3,4)
			};
			struct PS_RenderOutput{
				float4 f4Color      : SV_Target0;
			};
			struct HS_CONSTANT_OUTPUT
			{
				float edges[4]  : SV_TessFactor;
				float inside[2] : SV_InsideTessFactor;
			};
 
			struct HS_OUTPUT
			{
				float3 pos  : POS;
				float3 normal : NORMAL;
				float4 uv : TEXCOORD;
				float4 tangent: TANGENT;
				float3 lightDir : TEXTCOORD1;
				float3 viewDir : COLOR;
				fixed3 vlight : TEXCOORD2;
			};
 
			void vert(inout appdata v){
				v.vertex = mul(UNITY_MATRIX_MV, v.vertex);
			}
			HS_CONSTANT_OUTPUT HSConstant( InputPatch<appdata, 4> ip )
			{
				HS_CONSTANT_OUTPUT output;
 
				output.edges[0] = _TessEdge / ((-ip[0].vertex.z - _ProjectionParams.y)/_fadeDist);
				output.edges[1] = _TessEdge / ((-ip[1].vertex.z - _ProjectionParams.y)/_fadeDist);
				output.edges[2] = _TessEdge / ((-ip[2].vertex.z - _ProjectionParams.y)/_fadeDist);
				output.edges[3] = _TessEdge / ((-ip[3].vertex.z - _ProjectionParams.y)/_fadeDist);
 
				output.inside[0] = _TessEdge / ((-ip[0].vertex.z - _ProjectionParams.y)/_fadeDist);
				output.inside[1] = _TessEdge / ((-ip[0].vertex.z - _ProjectionParams.y)/_fadeDist);
 
				return output;
			}
 
			[domain("quad")]
			[partitioning("integer")]
			[outputtopology("triangle_cw")]
			[outputcontrolpoints(4)]
			[patchconstantfunc("HSConstant")]
			appdata hull( InputPatch<appdata, 4> ip, uint cpid : SV_OutputControlPointID, uint pid : SV_PrimitiveID )
			{
				return ip[cpid];
			}
 
			[domain("quad")]
			PS_INPUT domain( HS_CONSTANT_OUTPUT input, float2 UV : SV_DomainLocation, const OutputPatch<tess_appdata, 4> patch )
			{
				PS_INPUT o;
 
				float3 topMidpoint = lerp(patch[0].vertex, patch[1].vertex, UV.x);
				float3 bottomMidpoint = lerp(patch[3].vertex, patch[2].vertex, UV.x);
 
				float3 pos = lerp(topMidpoint, bottomMidpoint, UV.y);
 
				float2 uvtopMidpoint = lerp(patch[0].texcoord, patch[1].texcoord, UV.x);
				float2 uvbottomMidpoint = lerp(patch[3].texcoord, patch[2].texcoord, UV.x);
 
				o.uv.xy = lerp(uvtopMidpoint, uvbottomMidpoint, UV.y);
 
				topMidpoint = lerp(patch[0].normal, patch[1].normal, UV.x);
				bottomMidpoint = lerp(patch[3].normal, patch[2].normal, UV.x);
 
				//float4 pNormal = _BumpMap.Sample( sampler_BumpMap, input.uv );
 
				float3 normal = lerp(topMidpoint, bottomMidpoint, UV.y);
				o.normal= normal;// unity_LightColor[0].rgb * max( 0, dot( normal, unity_LightPosition[0].xyz ) ) + UNITY_LIGHTMODEL_AMBIENT.rgb;
 
				//pos = mul(UNITY_MATRIX_MV, pos);
				float3 disp = _DispTex.SampleLevel (sampler_DispTex, o.uv, 0).rgb * _Displacement;
				pos += normal * disp;
 
				o.pos = mul (UNITY_MATRIX_P, float4(pos, 1));
 
				float4 tangenttopMidpoint = lerp(patch[0].tangent, patch[1].tangent, UV.x);
				float4 tangentbottomMidpoint = lerp(patch[3].tangent, patch[2].tangent, UV.x);
 
				o.tangent = lerp(tangentbottomMidpoint, tangenttopMidpoint, UV.y);
				appdata v;
				v.normal = o.normal;
				v.tangent = o.tangent;
				TANGENT_SPACE_ROTATION;
				// To view space
				o.uv.xy = TRANSFORM_TEX(o.uv.xy, _MainTex);
				o.uv.zw = TRANSFORM_TEX(o.uv.xy, _BumpMap);
				o.lightDir = mul(rotation, ObjSpaceLightDir(o.pos));
				o.viewDir = mul(rotation, ObjSpaceViewDir(o.pos));
				float3 worldN = mul((float3x3)_Object2World, SCALED_NORMAL);
				o.vlight = ShadeSH9(float4(worldN,1.0));
				float3 worldPos = mul(_Object2World, o.pos).xyz;
				  o.vlight += Shade4PointLights (
					unity_4LightPosX0, unity_4LightPosY0, unity_4LightPosZ0,
					unity_LightColor[0].rgb, unity_LightColor[1].rgb, unity_LightColor[2].rgb, unity_LightColor[3].rgb,
					unity_4LightAtten0, worldPos, worldN );
				// //input.uv = I.uv;
				o.tangent = o.tangent;
				// o.uv = v.uv;
				TRANSFER_VERTEX_TO_FRAGMENT(o);
 
				return o;    
			}
 
			fixed4 frag( PS_INPUT input) : COLOR{
					Input surfIN;
				surfIN.uv_MainTex = input.uv.xy;
				surfIN.uv_BumpMap = input.uv.zw;
				#ifdef UNITY_COMPILER_HLSL
				SurfaceOutput o = (SurfaceOutput)0;
				#else
				SurfaceOutput o;
				#endif
				o.Albedo = 0.0;
				o.Emission = 0.0;
				o.Specular = 0.0;
				o.Alpha = 0.0;
				o.Gloss = 0.0;
				surf (surfIN, o);
				//o.Albedo = tex2D(_MainTex, input.uv);
				  fixed atten = LIGHT_ATTENUATION(input);
				  fixed4 c = 0;
				  //c = LightingBlinnPhong (o, input.lightDir, normalize(half3(input.viewDir)), atten);
				  c = LightingLambert (o, input.lightDir, atten);
				  c.rgb += o.Albedo * input.vlight;
				  c.a = o.Alpha;
				  return c;
			}
 
			ENDCG
		}
		Pass {
			Tags {"LightMode" = "ForwardAdd"}
			Blend One One
 
			CGPROGRAM
			#pragma vertex vert
			#pragma fragment frag
			#pragma hull hull
			#pragma domain domain
 
			#pragma fragmentoption ARB_precision_hint_fastest
			#pragma multi_compile_fwdadd_fullshadows
			#include "HLSLSupport.cginc"
			#include "UnityShaderVariables.cginc"
			#define UNITY_PASS_FORWARDADD
			#include "UnityCG.cginc"
			#include "Lighting.cginc"
			#include "AutoLight.cginc"
 
			float _TessEdge;
			float _Displacement;
			float _distFalloff;
			float _fadeDist;
			fixed4 _Color;
			half _Shininess;
 
			sampler2D _MainTex;
			SamplerState	sampler_MainTex;
			uniform float4 _MainTex_ST;
			uniform float4 _BumpMap_ST;
 
			Texture2D _DispTex;
			SamplerState	sampler_DispTex;
			//Texture2D _BumpMap;
			sampler2D _BumpMap;
			SamplerState	sampler_BumpMap;
			Texture2D _SpecMap;
			SamplerState	sampler_SpecMap;
 
			struct Input {
				float2 uv_MainTex;
				float2 uv_BumpMap;
			};
 
			void surf (Input IN, inout SurfaceOutput o) {
				fixed4 tex = tex2D(_MainTex, IN.uv_MainTex);
				o.Albedo = tex.rgb * _Color.rgb;
				o.Gloss = tex.a;
				o.Alpha = tex.a * _Color.a;
				o.Specular = _Shininess;
				o.Normal = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap));
			}
			struct appdata {
				float4 vertex : POSITION;
				float4 tangent : TANGENT;
				float3 normal : NORMAL;
				float2 texcoord : TEXCOORD0;
				float2 texcoord1 : TEXCOORD1;
			};
			struct tess_appdata {
				float4 vertex : POS;
				float4 tangent : TANGENT;
				float3 normal : NORMAL;
				float2 texcoord : TEXCOORD0;
				float2 texcoord1 : TEXCOORD1;
			};
			struct PS_INPUT
			{
				float4 pos   : POSITION;
				float3 normal     : NORMAL;
				float4 tangent     : TANGENT;
				float4 uv   : TEXCOORD;
				float3 lightDir : TEXCOORD1;
				float3 viewDir : COLOR;
				fixed3 vlight : TEXCOORD2;
				LIGHTING_COORDS(3,4)
			};
			struct PS_RenderOutput{
				float4 f4Color      : SV_Target0;
			};
			struct HS_CONSTANT_OUTPUT
			{
				float edges[4]  : SV_TessFactor;
				float inside[2] : SV_InsideTessFactor;
			};
 
			struct HS_OUTPUT
			{
				float3 pos  : POS;
				float3 normal : NORMAL;
				float4 uv : TEXCOORD;
				float4 tangent: TANGENT;
				float3 lightDir : TEXTCOORD1;
				float3 viewDir : COLOR;
				fixed3 vlight : TEXCOORD2;
			};
 
			void vert(inout appdata v){
				v.vertex = mul(UNITY_MATRIX_MV, v.vertex);
			}
			HS_CONSTANT_OUTPUT HSConstant( InputPatch<appdata, 4> ip )
			{
				HS_CONSTANT_OUTPUT output;
 
				output.edges[0] = _TessEdge / ((-ip[0].vertex.z - _ProjectionParams.y)/_fadeDist);
				output.edges[1] = _TessEdge / ((-ip[1].vertex.z - _ProjectionParams.y)/_fadeDist);
				output.edges[2] = _TessEdge / ((-ip[2].vertex.z - _ProjectionParams.y)/_fadeDist);
				output.edges[3] = _TessEdge / ((-ip[3].vertex.z - _ProjectionParams.y)/_fadeDist);
 
				output.inside[0] = _TessEdge / ((-ip[0].vertex.z - _ProjectionParams.y)/_fadeDist);
				output.inside[1] = _TessEdge / ((-ip[0].vertex.z - _ProjectionParams.y)/_fadeDist);
 
				return output;
			}
 
			[domain("quad")]
			[partitioning("integer")]
			[outputtopology("triangle_cw")]
			[outputcontrolpoints(4)]
			[patchconstantfunc("HSConstant")]
			appdata hull( InputPatch<appdata, 4> ip, uint cpid : SV_OutputControlPointID, uint pid : SV_PrimitiveID )
			{
				return ip[cpid];
			}
 
			[domain("quad")]
			PS_INPUT domain( HS_CONSTANT_OUTPUT input, float2 UV : SV_DomainLocation, const OutputPatch<tess_appdata, 4> patch )
			{
				PS_INPUT o;
 
				float3 topMidpoint = lerp(patch[0].vertex, patch[1].vertex, UV.x);
				float3 bottomMidpoint = lerp(patch[3].vertex, patch[2].vertex, UV.x);
 
				float3 pos = lerp(topMidpoint, bottomMidpoint, UV.y);
 
				float2 uvtopMidpoint = lerp(patch[0].texcoord, patch[1].texcoord, UV.x);
				float2 uvbottomMidpoint = lerp(patch[3].texcoord, patch[2].texcoord, UV.x);
 
				o.uv.xy = lerp(uvtopMidpoint, uvbottomMidpoint, UV.y);
 
				topMidpoint = lerp(patch[0].normal, patch[1].normal, UV.x);
				bottomMidpoint = lerp(patch[3].normal, patch[2].normal, UV.x);
 
				//float4 pNormal = _BumpMap.Sample( sampler_BumpMap, input.uv );
 
				float3 normal = lerp(topMidpoint, bottomMidpoint, UV.y);
				o.normal= normal;// unity_LightColor[0].rgb * max( 0, dot( normal, unity_LightPosition[0].xyz ) ) + UNITY_LIGHTMODEL_AMBIENT.rgb;
 
				//pos = mul(UNITY_MATRIX_MV, pos);
				float3 disp = _DispTex.SampleLevel (sampler_DispTex, o.uv, 0).rgb * _Displacement;
				pos += normal * disp;
 
				o.pos = mul (UNITY_MATRIX_P, float4(pos, 1));
 
				float4 tangenttopMidpoint = lerp(patch[0].tangent, patch[1].tangent, UV.x);
				float4 tangentbottomMidpoint = lerp(patch[3].tangent, patch[2].tangent, UV.x);
 
				o.tangent = lerp(tangentbottomMidpoint, tangenttopMidpoint, UV.y);
				appdata v = (appdata)0;
				v.normal = o.normal;
				v.tangent = o.tangent;
				TANGENT_SPACE_ROTATION;
				// To view space
				o.uv.xy = TRANSFORM_TEX(o.uv.xy, _MainTex);
				o.uv.zw = TRANSFORM_TEX(o.uv.xy, _BumpMap);
				o.lightDir = mul(rotation, ObjSpaceLightDir(o.pos));
				o.viewDir = mul(rotation, ObjSpaceViewDir(o.pos));
				// //input.uv = I.uv;
				o.tangent = o.tangent;
				// o.uv = v.uv;
				TRANSFER_VERTEX_TO_FRAGMENT(o);
 
				return o;    
			}
 
			fixed4 frag( PS_INPUT input) : COLOR{
					Input surfIN;
				surfIN.uv_MainTex = input.uv.xy;
				surfIN.uv_BumpMap = input.uv.zw;
				#ifdef UNITY_COMPILER_HLSL
				SurfaceOutput o = (SurfaceOutput)0;
				#else
				SurfaceOutput o;
				#endif
				o.Albedo = 0.0;
				o.Emission = 0.0;
				o.Specular = 0.0;
				o.Alpha = 0.0;
				o.Gloss = 0.0;
				surf (surfIN, o);
				//fixed4 c = LightingBlinnPhong (o, normalize(input.lightDir), normalize(half3(input.viewDir)), LIGHT_ATTENUATION(input));
				fixed4 c = LightingLambert (o, normalize(input.lightDir), LIGHT_ATTENUATION(input));
				c.a = 0.0;
				return c;
			}
 
			ENDCG
		}
 
	}
}

And Appendix B: the original post that started it all. Again, I’m re-posting this from PasteBin to account for the eventuality that it’ll be taken down.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
#ifndef TESS_CG_INCLUDED
#define TESS_CG_INCLUDED
 
#include "UnityCG.cginc"
#include "AutoLight.cginc" 
#include "HLSLSupport.cginc"
 
float _TessEdge;
float _Displacement;
float _distFalloff;
float _fadeDist;
 
uniform float4 _LightColor0; 
 
Texture2D _MainTex;
SamplerState	sampler_MainTex;
uniform float4 _MainTex_ST;
uniform float4 _BumpMap_ST;
 
Texture2D _DispTex;
SamplerState	sampler_DispTex;
Texture2D _BumpMap;
SamplerState	sampler_BumpMap;
//Texture2D _SpecMap;
//SamplerState	sampler_SpecMap;
struct appdata
{
	float4 position   : POSITION;
	float3 normal     : NORMAL;
	float4 tangent     : TANGENT;
	float2 uv   : TEXCOORD;
	float3 lightDir : TEXCOORD1;
	float3 viewDir : COLOR;
};
struct PS_INPUT
{
	float4 position   : POSITION;
	float3 normal     : NORMAL;
	float4 tangent     : TANGENT;
	float2 uv   : TEXCOORD;
	float3 lightDir : TEXCOORD1;
	float3 viewDir : COLOR;
};
struct PS_RenderOutput{
	float4 f4Color      : SV_Target0;
};
struct HS_CONSTANT_OUTPUT
{
	float edges[4]  : SV_TessFactor;
	float inside[2] : SV_InsideTessFactor;
};
 
struct HS_OUTPUT
{
	float3 position  : POS;
	float3 normal : NORMAL;
	float2 uv : TEXCOORD;
	float4 tangent: TANGENT;
	float3 lightDir : TEXTCOORD1;
	float3 viewDir : COLOR;
};
 
void vert(inout appdata v){
	TANGENT_SPACE_ROTATION;
	// To view space
	v.position = mul(UNITY_MATRIX_MV, v.position);
	v.normal = mul ((float3x3)UNITY_MATRIX_IT_MV, v.normal);
	//v.uv = TRANSFORM_TEX(v.uv, _MainTex);
	v.lightDir = mul(rotation, ObjSpaceLightDir(v.position));
	v.viewDir = mul(rotation, ObjSpaceViewDir(v.position));
	//input.uv = I.uv;
}
HS_CONSTANT_OUTPUT HSConstant( InputPatch<appdata, 4> ip, uint pid : SV_PrimitiveID )
{
	HS_CONSTANT_OUTPUT output;
 
	//float4 viewPos = mul(glstate.matrix.modelview[0], v.vertex);
	float distCoeff = (-ip[0].position.z * _distFalloff - _ProjectionParams.y)/_fadeDist; //ip[0].position.z * _distFalloff;//(ip[0].position.z  - _ProjectionParams.y) * _distFalloff;
	output.edges[0] = _TessEdge / ((-ip[0].position.z - _ProjectionParams.y)/_fadeDist);
	output.edges[1] = _TessEdge / ((-ip[1].position.z - _ProjectionParams.y)/_fadeDist);
	output.edges[2] = _TessEdge / ((-ip[2].position.z - _ProjectionParams.y)/_fadeDist);
	output.edges[3] = _TessEdge / ((-ip[3].position.z - _ProjectionParams.y)/_fadeDist);
 
	output.inside[0] = _TessEdge / ((-ip[0].position.z - _ProjectionParams.y)/_fadeDist);
	output.inside[1] = _TessEdge / ((-ip[0].position.z - _ProjectionParams.y)/_fadeDist);
 
	return output;
}
 
[domain("quad")]
[partitioning("integer")]
[outputtopology("triangle_cw")]
[outputcontrolpoints(4)]
[patchconstantfunc("HSConstant")]
HS_OUTPUT hull( InputPatch<appdata, 4> ip, uint cpid : SV_OutputControlPointID, uint pid : SV_PrimitiveID )
{
	HS_OUTPUT Output;
	Output.position = ip[cpid].position;
	Output.normal = ip[cpid].normal;
	Output.uv = ip[cpid].uv;
	Output.lightDir = ip[cpid].lightDir;
	Output.tangent = ip[cpid].tangent;
	return Output;
}
 
[domain("quad")]
appdata domain( HS_CONSTANT_OUTPUT input, float2 UV : SV_DomainLocation, const OutputPatch<HS_OUTPUT, 4> patch )
{
	appdata Output;
	float3 topMidpoint = lerp(patch[0].position, patch[1].position, UV.x);
	float3 bottomMidpoint = lerp(patch[3].position, patch[2].position, UV.x);
 
	float3 position = lerp(topMidpoint, bottomMidpoint, UV.y);
 
	//Output.position = float4(lerp(topMidpoint, bottomMidpoint, UV.y), 1);
	//
	//O.f4Diffuse.a = 1.0f;
	float2 uvtopMidpoint = lerp(patch[0].uv, patch[1].uv, UV.x);
	float2 uvbottomMidpoint = lerp(patch[3].uv, patch[2].uv, UV.x);
 
	Output.uv = lerp(uvtopMidpoint, uvbottomMidpoint, UV.y);
 
	float3 normaltopMidpoint = lerp(patch[0].normal, patch[1].normal, UV.x);
	float3 normalbottomMidpoint = lerp(patch[3].normal, patch[2].normal, UV.x);
 
	//float4 pNormal = _BumpMap.Sample( sampler_BumpMap, input.uv );
 
	float3 normal = lerp(normaltopMidpoint, normalbottomMidpoint, UV.y);
	Output.normal= unity_LightColor[0].rgb * max( 0, dot( normal, unity_LightPosition[0].xyz ) ) + UNITY_LIGHTMODEL_AMBIENT.rgb;
 
	float disp = _DispTex.SampleLevel (sampler_DispTex, Output.uv, 0).r * _Displacement;
	position += normal * disp;
 
	Output.position = mul (UNITY_MATRIX_P, float4(position, 1));
 
	float3 lighttopMidpoint = lerp(patch[0].lightDir, patch[1].lightDir, UV.x);
	float3 lightbottomMidpoint = lerp(patch[3].lightDir, patch[2].lightDir, UV.x);
 
	//float4 pNormal = _BumpMap.Sample( sampler_BumpMap, input.uv );
 
	Output.lightDir = lerp(lighttopMidpoint, lightbottomMidpoint, UV.y);
 
	float4 tangenttopMidpoint = lerp(patch[0].tangent, patch[1].tangent, UV.x);
	float4 tangentbottomMidpoint = lerp(patch[3].tangent, patch[2].tangent, UV.x);
 
	//float4 pNormal = _BumpMap.Sample( sampler_BumpMap, input.uv );
 
	Output.tangent = lerp(tangentbottomMidpoint, tangenttopMidpoint, UV.y);
 
	return Output;    
}
 
[domain("quad")]
appdata vec_domain( HS_CONSTANT_OUTPUT input, float2 UV : SV_DomainLocation, const OutputPatch<HS_OUTPUT, 4> patch )
{
	appdata Output;
	float3 topMidpoint = lerp(patch[0].position, patch[1].position, UV.x);
	float3 bottomMidpoint = lerp(patch[3].position, patch[2].position, UV.x);
 
	float3 position = lerp(topMidpoint, bottomMidpoint, UV.y);
 
	//Output.position = float4(lerp(topMidpoint, bottomMidpoint, UV.y), 1);
	//
	//O.f4Diffuse.a = 1.0f;
	float2 uvtopMidpoint = lerp(patch[0].uv, patch[1].uv, UV.x);
	float2 uvbottomMidpoint = lerp(patch[3].uv, patch[2].uv, UV.x);
 
	Output.uv = lerp(uvtopMidpoint, uvbottomMidpoint, UV.y);
 
	float3 normaltopMidpoint = lerp(patch[0].normal, patch[1].normal, UV.x);
	float3 normalbottomMidpoint = lerp(patch[3].normal, patch[2].normal, UV.x);
 
	//float4 pNormal = _BumpMap.Sample( sampler_BumpMap, input.uv );
 
	float3 normal = lerp(normaltopMidpoint, normalbottomMidpoint, UV.y);
	Output.normal= unity_LightColor[0].rgb * max( 0, dot( normal, unity_LightPosition[0].xyz ) ) + UNITY_LIGHTMODEL_AMBIENT.rgb;
 
	float disp = _DispTex.SampleLevel (sampler_DispTex, Output.uv, 0).r * _Displacement;
	position += normal * disp;
 
	Output.position = mul (UNITY_MATRIX_P, float4(position, 1));
 
	float3 lighttopMidpoint = lerp(patch[0].lightDir, patch[1].lightDir, UV.x);
	float3 lightbottomMidpoint = lerp(patch[3].lightDir, patch[2].lightDir, UV.x);
 
	//float4 pNormal = _BumpMap.Sample( sampler_BumpMap, input.uv );
 
	Output.lightDir = lerp(lighttopMidpoint, lightbottomMidpoint, UV.y);
 
	float4 tangenttopMidpoint = lerp(patch[0].tangent, patch[1].tangent, UV.x);
	float4 tangentbottomMidpoint = lerp(patch[3].tangent, patch[2].tangent, UV.x);
 
	//float4 pNormal = _BumpMap.Sample( sampler_BumpMap, input.uv );
 
	Output.tangent = lerp(tangentbottomMidpoint, tangenttopMidpoint, UV.y);
 
	return Output;    
}
float4 frag( PS_INPUT input) : SV_TARGET{
 
	//float3 lightDir = normalize( _WorldSpaceLightPos0);
	half3 normal = UnpackNormal(_BumpMap.Sample( sampler_BumpMap, TRANSFORM_TEX(input.uv, _BumpMap) ));
	float4 Albedo = _MainTex.Sample( sampler_MainTex, TRANSFORM_TEX(input.uv, _MainTex) );
	float pxlAtten = dot( normal, normalize(input.lightDir ));
	float3 diff = Albedo * pxlAtten;
	//half4 c = SpecularColorLight( input.lightDir, input.viewDir, normal, Albedo, Albedo.rgb, Albedo.r, pxlAtten );
	return float4( diff, 1 ) * 0.5;
	//return Albedo;
 
	// float3 lightColor = float3(0,0,0);
 
	// float4 c = _MainTex.Sample( sampler_MainTex, TRANSFORM_TEX(input.uv, _MainTex) );
	// float3 n =  UnpackNormal(_BumpMap.Sample( sampler_BumpMap, input.uv ));
	// float lengthSq = dot(input.lightDir, input.lightDir);
	// float atten = 1.0 / (1.0 + lengthSq * unity_LightAtten[0].z);
	// Angle to the light
	// float diff = saturate (dot (n, normalize(input.lightDir)));   
	// lightColor += _LightColor0.rgb * (diff * atten);
	// c.rgb = lightColor * c.rgb; 
	// return c; 
 
}
#endif

~ by Schoen on February 6, 2013.

9 Responses to “Quad Patch Tessellation in Unity”

  1. Thank you so much for posting this! I can’t thank you enough. I’ve wanted to try this for some time, as I too was dissatisfied with the unity tesselation shaders.

    Two questions though –
    1 since you are using a custom obj importer, how would you apply animations to a model that has been quad tesselated? Like an animated prop or skinned character? This is one nice advantage of using the Unity ones, since they seem to effortlessly mix with normal shaders.
    2 from following your blog, it seems like you guys do mostly mobile dev, and this shader (like you said) is basically PC (and high end ones) only. What was your motivation for this? A quick aside or some plans for a tech demo or with some pretty intense graphics?

    Sorry if this is a dumb question, I haven’t really delved too far into your code just yet.

    thank you again! big fan now

    • Hey John, thanks!

      As to your questions:
      1) HA! Yes, this is a pain in the butt! The short answer is that I haven’t found a satisfactory way to do this. I ended up writing another script to essentially copy the mesh over to the rigged model (imported as tris). This took a long time to do and never stuck around (in other words, building or quitting Unity would destroy the data). The final video does show an animated character but I was only able to capture this once.
      2) You’re absolutely right, this totally won’t work on mobile devices since OpenGL ES is nowhere near supporting shader model 5 (or 4, or 3, or 2). The reasoning for this was mostly “because we can.” Our tech artist was the one to come up with the idea (and we created this thing for the Unity DirectX 11 demo) but I was actually most interested in the other half of my contribution, which was the particle shader (as yet unfinished) which simulates the particles on the GPU with DirectCompute. Cool stuff!

      Thanks again for your comment, I hope this helps!

  2. Is there any way you could share the animation copying script too? I’d like to hack around on this, as I’d like to get quad patch working in a running game :) only feature unity is lacking that I really want right now

    • So this is a really belated reply, but here goes: I’ll throw together a zip at some point when I revisit this project. The animation copying basically didn’t work. It took literally 10 minutes no matter how I went about it, and the result couldn’t be saved, and thus won’t translate to builds. I was able to get as far as running it in the editor to get the video, but that’s as far as it goes.

  3. Hi,

    Thanks for the great Info, I am trying to implement this now however when I import a Quad mesh ( .obj quad plane mesh exported from Max) and apply the above shader it is disappearing? any hint will be helpful! Thanks again.

  4. Hi,

    Yes I am using your importer and it is importing fine, however the Mesh is disappearing when I choose the tesselation shader for it’s material, thanks again.

  5. Hi There,

    This tessellation shader seems really promising, however when I plug it into Unity 5 i get the following shader errors…

    “undeclared identifier ‘ip’ at line 117 (on d3d11)” and
    “syntax error: unexpected token ‘&’ at line 113 (on d3d11)”

    This seems to point to what I would normally expect is a text conversion type occuring on line 113 (the “InputPatch<appdata, 4> ip,” part) and a few other places. I’ve never worked with domain shaders before though so I don’t know if this is standard code? If you have any help or advice it would be greatly appreciated :)

    • Hoho, I was wondering if this would happen eventually. I know this isn’t what you want to hear, but I haven’t thought about this project since I put it out there. It was hacked together based on what I could find out there about quad patch tessellation at the time, over 2 years ago. The missing (until I just fixed it now) image might have been a clue. 😉 I also haven’t ever worked with a shader this complicated before or since, so I’d have to do some research to answer your question.

      Have you looked through the Unity built-in shader sources? At the time, there were tri-patch tessellation shaders which served as a good reference for this shader. Perhaps there’s a simple change in nomenclature or built-in values that you might see if you diff the 4.x tessellation shaders (or CGIncludes shared by all the shaders, more likely) with the 5.x versions.

      The person who can answer definitively is @arasp. After a little more research, I might send him a tweet 😉

Leave a Reply

Your email address will not be published. Required fields are marked *