Camera2 Pass Images from ImageReader to MediaRecorder

[mrousavy]

Hi!

I'm trying to create a Camera2 CameraCaptureSession that is capable of four outputs:

1. On-screen preview (SurfaceView, up to 1080p)
2. Photo capture (ImageReader, up to 8k photos)
3. Video Capture (MediaRecorder/MediaCodec, up to 4k videos)
4. Frame Processing (ImageReader, up to 4k video frames)

Unfortunately Camera2 does not support attaching all of those four outputs (Surfaces) at the same time, so I'm going to have to make a compromise.

The compromise that seemed most logical to me was to combine the two video capture pipelines into one, so that the Frame Processing output (#4, ImageReader) redirects the frames into the Video Capture output (#3, MediaRecorder).

How do I write the Images from the ImageReader:

val imageReader = ImageReader.newInstance(4000, 2256, ImageFormat.YUV_420_888, 3)
imageReader.setOnImageAvailableListener({ reader ->
  val image = reader.acquireNextImage() ?: return@setOnImageAvailableListener
  callback.onVideoFrameCaptured(image)
}, queue.handler)

val captureSession = device.createCaptureSession(.., imageReader.surface)

..into the Surface from the MediaRecorder?

val surface = MediaCodec.createPersistentInputSurface()
val recorder = MediaRecorder(context)
..
recorder.setInputSurface(surface)

---

I'm not too familiar with OpenGL, so any help here would be appreciated.

[tonykwok]

OpenGL, as its name implies, is a "graphic" library that is good at "drawing".
So, in short, it can not help you write an image into surface, but it can help "draw" the image into the surface.

1. extract YUV data from the image
2. create a texture and upload the YUV as its data
3. wrap the surface into EglWindowSurface
4. draw the texture into the EglWindowSurface
   (You can find samples in webrtc)

OR, do this manually, just like "memcpy" (very inefficient, not recommended):

1. MediaCodec will notify you when it's ready to accept a new input frame by callback: #onInputBufferAvailable(index, ...)
2. Get the image at position "index" as the callback tells
3. "memcpy" your image to this image
4. call MediaCodec.queueInputBuffer(index, ...) to return the image back to MediaCodec
   (You can find samples in androidx/heifwriter)

Hope these help.

[mrousavy]

Thanks Tony!

I was able to use ImageWriter which works for now, but I'll soon need to write my custom OpenGL pipeline for that because some images aren't correctly oriented. Thanks!