drm/vmwgfx: Prune modes based on available VRAM size

This needs to be reviewed once we support screen objects and don't rely
on VRAM for the frame-buffer.

Also fix some integer overflow issues pointed out by Michel Daenzer.

Signed-off-by: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drivers/gpu/drm/vmwgfx/vmwgfx_fb.c

@@ -144,6 +144,13 @@ static int vmw_fb_check_var(struct fb_var_screeninfo *var,
 		return -EINVAL;
 	}
 
+	if (!vmw_kms_validate_mode_vram(vmw_priv,
+					info->fix.line_length,
+					var->yoffset + var->yres)) {
+		DRM_ERROR("Requested geom can not fit in framebuffer\n");
+		return -EINVAL;
+	}
+
 	return 0;
 }