There are two common methods for displaying Line of Sight and and Fog of War in an RTS. The geometry hidden from the player can either be modified before being rendered to the screen, or a second pass can be used to paint over the top to cut out areas that should not be seen.
Painting over the top
Painting blackness over the top of the world was a very common approach for older 2D RTS’. It can be seen in Age of Empires 1 by the dithering pattern present in revealed foggy areas. This approach has the benefit of working very well with DirectDraw (and GDI) compatible hardware, but also nicely decouples the fog of war code from the rest of the game rendering. Civilization has another neat version of this for its cloudy fog of war.
Painting on the surface
This seems to be the approach used in most modern RTS games. A trivial implementation would require a texture to be generated matching the terrain size, such that each pixel corresponds to one cell of the terrain. Each of these pixels contains visibility data for that cell, with a shader reading that data to darken or remove sections of the terrain. This method forces tight coupling between the geometry rendering and fog of war, but allows more advanced rendering; properly handling transparency, filters, and working well with 3D geometry.
LOS in Unity
Unity has sufficient support to easily handle the shaders necessary for painting line of sight darkening right onto the geometry. I like to implement this with a global shader texture and scale/translation to map from world XZ coordinates to the LOS texture. To work with the indie version of Unity, this texture needs to be generated on CPU and copied over each time it changes; this slower than doing the work directly on the GPU, but for smaller maps, it is sufficiently fast.
Each frame, the LOSManager clears the red channel of each pixel to 0. It then iterates through each entity in the world owned by the player, and paints a white circle onto the texture around the unit (importantly, it sets the red and green channels to 255). This means that every pixel that is currently visible will have a red and green value of 255. Every pixel that was previously visible and is now fogged will have a red value of 0 and green value of 255. Pixels that have not yet been revealed will be black. In the UnityRTS implementation, the blue channel is used to store the amount of time since the area was visible, and the alpha channel stores ambient occlusion, though these are not important features.
The shader needed to render line of sight on objects is quite simple. It can be implemented in a surface shader by adding the following lines:
half4 fow = tex2D(_FOWTex, TRANSFORM_TEX(IN.worldPos.xz, _FOWTex));
o.Albedo = t.rgb * (fow.r + fow.g) / 2;
Where t is the texel from _MainTex, and _FOWTex is the fog of war texture (along with _FOWTex_ST). This code will cause anything in full visibility to be draw as normal, anything revealed but not visible to be half-brightness, and anything hidden to be black.