sim/camera: add AVFoundation backend and multi-cam support by PeterBee97 · Pull Request #18745 · apache/nuttx

PeterBee97 · 2026-04-15T14:00:04Z

Summary

Add AVFoundation backend and multi-cam support, enabling macOS users as well as Linux users to use multiple cameras on Simulator.

Impact

Impact on users: Backward compatible
Affected platforms:
- SIM on macOS
- SIM on Linux
Scope:
- Enhances the SIM camera framework for multi-instance support
- Adds a new macOS AVFoundation camera backend
- Fixes mkallsyms host-tool behavior for Objective-C symbols on macOS
Non-affected platforms:
- No impact on other boards or non-SIM targets
- Existing Linux SIM camera behavior remains supported

Testing

Tested on:

Host OS: macOS Tahoe on Apple Silicon, Ubuntu x86_64
Target configurations: sim:nxcamera

acassis

@PeterBee97 WOW! Amazing! Please update the nxcamera Documentation/ to include this new parameter: https://nuttx.apache.org/docs/latest/applications/system/nxcamera/index.html

acassis · 2026-04-15T14:29:43Z

@PeterBee97 please include this picture in the Documentation page, it is funny :-)

PeterBee97 · 2026-04-16T04:23:17Z

@PeterBee97 WOW! Amazing! Please update the nxcamera Documentation/ to include this new parameter: https://nuttx.apache.org/docs/latest/applications/system/nxcamera/index.html

As u wish :)

xiaoxiang781216 · 2026-04-16T12:06:46Z

+  char resolved_path[PATH_MAX];
+  int ret;
+
+  ret = host_video_resolve_device_path(host_video_dev_path, resolved_path,


why not call host_video_is_capture_device directly

Because I wanted to filter-out non-capture video device. For now I have two cameras connected, but 4 video devices under /dev in host. I intentionally mapped /dev/video0 and /dev/video2 on host to /dev/video0 and /dev/video1 in NuttX. Is this behavior desired or do you prefer mapping directly without changing numbers?

xiaoxiang781216 · 2026-04-16T12:07:07Z


-  fd = open(host_video_dev_path, O_RDWR | O_NONBLOCK);
-  if (fd < 0)
+  ret = host_video_resolve_device_path(host_video_dev_path, resolved_path,


why not call open with host_video_dev_path directly

Same reason as above. I wanted to map dev numbers without "holes".

acassis

@PeterBee97 Amazing! Thank you very much!

simbit18 · 2026-04-17T12:25:12Z

Hi @PeterBee97 please fix

====================================================================================
Configuration/Tool: sim/nxcamera
2026-04-17 09:51:22
------------------------------------------------------------------------------------
  Cleaning...
  Configuring...
  Building NuttX...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  485k  100  485k    0     0  1893k      0 --:--:-- --:--:-- --:--:-- 1893k
sim/posix/sim_host_v4l2.c: In function 'host_video_get_device_path_by_index':
Error: sim/posix/sim_host_v4l2.c:147:15: error: implicit declaration of function 'strlcpy'; did you mean 'strncpy'? [-Werror=implicit-function-declaration]
  147 |           if (strlcpy(devpath, path, devpathlen) >= devpathlen)
      |               ^~~~~~~
      |               strncpy
cc1: all warnings being treated as errors
make[1]: *** [Makefile:408: sim_host_v4l2.o] Error 1
make[1]: Target 'nuttx' not remade because of errors.
make: *** [tools/Unix.mk:568: nuttx] Error 2
make: Target 'all' not remade because of errors.
/github/workspace/sources/nuttx/tools/testbuild.sh: line 386: /github/workspace/sources/nuttx/../nuttx/nuttx.manifest: No such file or directory
  [1/1] Normalize sim/nxcamera
Removing libs/libbuiltin/bin/
Removing libs/libbuiltin/kbin/
Removing libs/libc/kbin/
Removing libs/libnx/bin/
Removing libs/libnx/kbin/
Removing mm/kbin/
/github/workspace/sources/nuttx /github/workspace/sources/nuttx
HEAD detached at pull/18745/merge
nothing to commit, working tree clean
/github/workspace/sources/nuttx
/github/workspace/sources/apps /github/workspace/sources/nuttx
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
/github/workspace/sources/nuttx
Build Attempt 1 of 1
===================================================================================

xiaoxiang781216 · 2026-04-20T09:10:58Z

 static FAR struct imgsensor_s **g_capture_registered_sensor = NULL;
 static size_t g_capture_registered_sensor_num;
-static FAR struct imgdata_s *g_capture_data = NULL;
+static FAR struct imgdata_s **g_capture_registered_data = NULL;


move the common change to new patch

xiaoxiang781216 · 2026-04-20T09:12:56Z


+# Add dummy.c to ensure that we have at least one object.
+# On some platforms like macOS, we can't create an empty archive.
+ifeq ($(strip $(ASRCS) $(CSRCS)),)


move to new patch

xiaoxiang781216 · 2026-04-20T09:14:01Z

+
+          snprintf(devpath, sizeof(devpath), "%s%d",
+                   CONFIG_SIM_CAMERA_DEV_PATH, i);
+          ret = capture_initialize(devpath);


it's betterr to call capture_register if you want to support multiple camera

xiaoxiang781216 · 2026-04-20T09:16:53Z

+      return -EINVAL;
+    }
+
+  return avf_try_format(vdev->backend, width, height, fmt, denom, numer);


why not let arch/sim/src/sim/macos/sim_host_avfoundation_backend.m implement host_video_xxx and remove avf_xxx directly

Add dummy.c so video can build an archive even when there are no other objects (e.g. on macOS). Signed-off-by: bijunda <bijunda@bytedance.com>

Move common v4l2 capture framework changes to a standalone commit. Signed-off-by: bijunda <bijunda@bytedance.com>

Allow the SIM camera/V4L2 capture framework to manage multiple imgdata instances with dynamic mounting. This avoids cross-talk between camera streams when multiple devices are used. Signed-off-by: Peter Bee <bijunda@bytedance.com>

Adapt the macOS AVFoundation backend for multi-device discovery, camera index mapping and capture startup compatibility. This enables reliable use of multiple cameras through the SIM camera framework. Signed-off-by: Peter Bee <bijunda@bytedance.com>

/Library/Developer/CommandLineTools/usr/bin/c++filt: error: unknown argument '-[' Signed-off-by: Peter Bee <bijunda@bytedance.com>

Document the nxcamera command usage, including device selection, resolution, and pixel format options. Also mention recent support for multi-instance camera mounting and the macOS AVFoundation backend on the SIM platform. Signed-off-by: Peter Bee <bijunda@bytedance.com>

PeterBee97 requested review from Donny9, jerpelea, masayuki2009, pkarashchenko, xiaoxiang781216 and yamt as code owners April 15, 2026 14:00

github-actions bot added Area: Build system Arch: simulator Issues related to the SIMulator Size: XL The size of the change in this PR is very large. Consider breaking down the PR into smaller pieces. Board: simulator labels Apr 15, 2026

anchao previously approved these changes Apr 15, 2026

View reviewed changes

acassis requested changes Apr 15, 2026

View reviewed changes

simbit18 previously approved these changes Apr 15, 2026

View reviewed changes

PeterBee97 dismissed stale reviews from simbit18 and anchao via 1131750 April 16, 2026 04:21

PeterBee97 force-pushed the avfoundation branch from d0fdb9c to 1131750 Compare April 16, 2026 04:21

PeterBee97 force-pushed the avfoundation branch from 1131750 to f1c856b Compare April 16, 2026 06:23

PeterBee97 requested a review from acassis April 16, 2026 06:26

simbit18 previously approved these changes Apr 16, 2026

View reviewed changes

xiaoxiang781216 reviewed Apr 16, 2026

View reviewed changes

acassis previously approved these changes Apr 16, 2026

View reviewed changes

PeterBee97 dismissed stale reviews from acassis and simbit18 via c2ea6da April 17, 2026 09:16

PeterBee97 force-pushed the avfoundation branch from f1c856b to c2ea6da Compare April 17, 2026 09:16

xiaoxiang781216 reviewed Apr 19, 2026

View reviewed changes

Comment thread tools/mkallsyms.sh

Comment thread arch/sim/src/sim/posix/sim_host_v4l2.c Outdated

Comment thread arch/sim/src/sim/posix/sim_host_v4l2.c Outdated

PeterBee97 force-pushed the avfoundation branch 2 times, most recently from 6b24b6b to 6ae23c9 Compare April 20, 2026 02:50

xiaoxiang781216 reviewed Apr 20, 2026

View reviewed changes

Peter Bee added 6 commits April 20, 2026 21:13

video: ensure video library is non-empty

b98f3b0

Add dummy.c so video can build an archive even when there are no other objects (e.g. on macOS). Signed-off-by: bijunda <bijunda@bytedance.com>

video/v4l2_cap: split out multi-instance registration

5669728

Move common v4l2 capture framework changes to a standalone commit. Signed-off-by: bijunda <bijunda@bytedance.com>

sim: fix mkallsyms for Objective-C symbols on macOS

7dda635

/Library/Developer/CommandLineTools/usr/bin/c++filt: error: unknown argument '-[' Signed-off-by: Peter Bee <bijunda@bytedance.com>

PeterBee97 force-pushed the avfoundation branch from 6ae23c9 to c07e073 Compare April 20, 2026 13:17

PeterBee97 requested a review from GUIDINGLI as a code owner April 20, 2026 13:17

Conversation

PeterBee97 commented Apr 15, 2026

Summary

Impact

Testing

Uh oh!

acassis left a comment

Choose a reason for hiding this comment

Uh oh!

acassis commented Apr 15, 2026

Uh oh!

PeterBee97 commented Apr 16, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PeterBee97 Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PeterBee97 Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

acassis left a comment

Choose a reason for hiding this comment

Uh oh!

simbit18 commented Apr 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PeterBee97 Apr 17, 2026 •

edited

Loading

PeterBee97 Apr 17, 2026 •

edited

Loading