MSAA Fix: Resolved Framebuffer Reused as Render Target

Summary #

MSAA was configured correctly and worked during the first rendered frames, but the stable MulleCG window eventually looked identical to NoAA. The problem was not in OpenGL multisampling, nanovg geometry, the nanovg shader, stencil strokes, or the MSAA resolve operation.

The framebuffer reuse cache recycled a single-sample resolved framebuffer and later reused it as the render target for a request that required an MSAA framebuffer. Reuse compatibility checked only the framebuffer dimensions and ignored its texture flags.

The fix makes framebuffer reuse require both:

Matching bitmap dimensions
Matching texture flags, including CGImageFlagMSAA

Rendering Architecture #

The retained/threaded MulleUI rendering path uses two graphics contexts:

An offscreen context renders the view hierarchy.
A display context presents the completed image in the visible window.

For an antialiased window, one frame is intended to follow this sequence:

beginRender requests an offscreen framebuffer whose texture flags contain CGImageFlagMSAA.
MulleCG creates or reuses a four-sample framebuffer backed by a GL_TEXTURE_2D_MULTISAMPLE color texture and multisampled stencil buffer.
nanovg flushes its drawing commands into that framebuffer while GL_MULTISAMPLE is enabled.
endRender resolves the multisampled image with glBlitFramebuffer(..., GL_NEAREST) into a new single-sample framebuffer.
The single-sample resolved framebuffer is handed to the display context.
The display context draws the resolved texture into the visible window.
After presentation, the framebuffer is recycled to the offscreen context so an allocation may be avoided on a later frame.

The resolve target must be single-sample because a multisample texture cannot be sampled by the ordinary nanovg image shader used by the display context. Consequently, the framebuffer returned by endRender is intentionally a single-sample framebuffer even though the original render target was MSAA.

What Was Wrong #

The cache lookup is implemented by -[CGContext framebufferWithSize:textureFlags:nameUTF8String:] in src/CGContext+CGFramebuffer.m.

Before the fix, it repossessed a framebuffer from _scratchRenderedFramebuffer and checked only its size:

 1framebuffer = [self repossessRenderedFramebuffer];
 2oldSize     = [framebuffer bitmapSize];
 3
 4if( ! CGPixelSizeEqualToPixelSize( oldSize, size))
 5{
 6   framebuffer = [CGFramebuffer framebufferWithBitmapSize:size
 7                                              textureFlags:textureFlags
 8                                            nameUTF8String:nameUTF8String
 9                                                    context:self];
10}

This treats all framebuffers with equal dimensions as interchangeable. They are not interchangeable. In particular:

An MSAA framebuffer has CGImageFlagMSAA and a multisample color attachment.
Its resolved framebuffer does not have CGImageFlagMSAA and has a single-sample color attachment.

Both may be 128x128, but they have different rendering semantics.

Failure Sequence #

The failure depended on the retained/threaded reuse cycle:

The framebuffer cache was initially empty.
The offscreen context created a proper MSAA framebuffer and rendered into it.
endRender resolved it into a single-sample framebuffer.
The resolved framebuffer was provided to the display thread.
After displaying it, the display thread recycled that single-sample framebuffer into _scratchRenderedFramebuffer.
On a later frame, beginRender requested another MSAA framebuffer.
framebufferWithSize:textureFlags:nameUTF8String: repossessed the old resolved framebuffer.
Its dimensions matched the requested dimensions.
Because texture flags were not compared, the single-sample framebuffer was accepted as the render target.
nanovg then rendered the frame directly into that single-sample FBO.
The displayed result was jagged, even though the context still logged NVG_MSAA and called glEnable(GL_MULTISAMPLE).

Enabling GL_MULTISAMPLE cannot add multisampling to a single-sample framebuffer. The framebuffer attachments themselves must have multiple samples.

The first one or two frames could therefore be correctly antialiased. The stable result became NoAA only after a resolved framebuffer completed the cross-thread recycle cycle. This is why inspecting only context creation or an early frame made the pipeline appear correct.

Why the Initial Suspects Were Innocent #

A pure OpenGL research program in research/msaa/msaa_test.c isolated the individual parts of the pipeline.

At 128x128 it produced:

Mode	Gray pixels	Result
NoAA	0	Jagged
MSAA renderbuffer	562	Antialiased
MSAA texture	562	Antialiased
MSAA plus stencil stroke passes	562	Antialiased
MSAA plus reduced nanovg shader	526	Antialiased

These tests established that:

GL_TEXTURE_2D_MULTISAMPLE works.
glBlitFramebuffer(..., GL_NEAREST) correctly resolves four samples.
nanovg-style blending and stencil stroke passes preserve sample coverage.
The reduced nanovg shader preserves sample coverage.
Slanted line and circle geometry produces visible MSAA coverage.

The problem therefore required the full retained framebuffer lifecycle and was not reproducible in a one-shot render/resolve test.

apitrace Evidence #

The useful traces were captured with EGL on this system:

1apitrace trace --api egl -o research.trace ./build/msaa_test
2apitrace trace --api egl -o msaa.trace mulle-sde run msaa-test

Tracing with --api gl captured GL calls but did not intercept EGL context and swap creation correctly, producing incomplete or non-replayable traces.

Before the fix #

An early MulleCG frame was correct:

Resolve call 2452 copied the four-sample render texture into a single-sample framebuffer.
Presentation draw 2506 copied the resolved texture into the visible backbuffer.
Both snapshots had:

1black=15254 white=604 gray=526

The only intermediate values were the expected four-sample resolve levels:

164, 127, 191

This proved that the real nanovg shader, real nanovg stencil stroke passes, multisample texture, resolve, and screen presentation all worked.

Later in the trace:

Call 2797 bound recycled framebuffer 2 directly as the next render target.
It was the old single-sample resolve framebuffer.
nanovg rendered the next frame into that framebuffer without a new multisample color attachment.
Presentation draw 2949 then contained:

1black=15434 white=950 gray=0

That is the same black/white character as the NoAA control and explains the stable jagged window.

After the fix #

The next rendering cycle no longer accepted the recycled resolve framebuffer. The trace instead showed:

A new texture created at call 2744
glTexStorage2DMultisample(..., samples = 4, ...) at call 2754
A new FBO created and bound at calls 2764-2765
The multisample texture attached at call 2766
A four-sample stencil attachment created at call 2769
The complete MSAA FBO bound for rendering at call 2774

Early and later presentation snapshots were identical:

1black=15254 white=604 gray=526

The NoAA control remained:

1black=15434 white=950 gray=0

The Fix #

framebufferWithSize:textureFlags:nameUTF8String: now compares the existing framebuffer's texture flags with the requested flags:

 1CGFramebuffer   *framebuffer;
 2CGPixelSize     oldSize;
 3NSUInteger      oldTextureFlags;
 4
 5framebuffer     = [self repossessRenderedFramebuffer];
 6oldSize         = [framebuffer bitmapSize];
 7oldTextureFlags = [framebuffer textureFlags];
 8
 9if( ! CGPixelSizeEqualToPixelSize( oldSize, size) ||
10    oldTextureFlags != textureFlags)
11{
12   framebuffer = [CGFramebuffer framebufferWithBitmapSize:size
13                                              textureFlags:textureFlags
14                                            nameUTF8String:nameUTF8String
15                                                    context:self];
16}

A single-sample resolve framebuffer therefore cannot satisfy a render request containing CGImageFlagMSAA, even when the dimensions are identical.

Validation #

Validation performed after the fix:

The MulleCG demo project builds successfully with mulle-sde craft.
LSP reports no diagnostics in CGContext+CGFramebuffer.m.
Post-fix apitrace snapshots retain 526 gray edge pixels in both early and later presented frames.
The corresponding NoAA trace retains zero gray edge pixels.
The research program reports 562 gray pixels for its MSAA renderbuffer, multisample texture, and stencil variants, and 526 for the reduced nanovg shader variant.

For visual comparison:

1cd research/msaa
2cmake --build build
3./build/msaa_test

and:

1cd demo
2mulle-sde craft
3mulle-sde run msaa-test       # MSAA
4mulle-sde run msaa-test -a    # NoAA

The -a option clears UIWindowStyleAntiAlias; the default mode requests MSAA.

Note About the Old Pixel Callback #

The didRenderFrameCallback diagnostic used glReadPixels against whichever default-buffer stage was current when the callback ran. Its counts varied and did not reliably describe the frame that was subsequently presented.

Those callback counts led to several false hypotheses. The decisive evidence came from apitrace snapshots taken directly:

After the MSAA resolve
After the display-context presentation draw
Again after framebuffer recycling had occurred

The bug was a resource compatibility error in framebuffer reuse, not a timing or synchronization failure.

last updated: 2026-08-23