Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

## Unreleased

### Added

- [crunzphp#83] Add `output_realtime` config option to stream task output to its destination (log file or console) as it is produced instead of only after the task finishes

## [v3.7.0] - 2024-08-12

### Changed
Expand Down Expand Up @@ -556,6 +560,7 @@ In `v2` this will result in exception.
[crunzphp#77]: https://github.com/crunzphp/crunz/pull/77
[crunzphp#78]: https://github.com/crunzphp/crunz/pull/78
[crunzphp#79]: https://github.com/crunzphp/crunz/pull/79
[crunzphp#83]: https://github.com/crunzphp/crunz/issues/83
[v1.5.1]: https://github.com/crunzphp/crunz/compare/v1.5.0...v1.5.1
[v1.6.0]: https://github.com/crunzphp/crunz/compare/v1.5.1...v1.6.0
[v1.6.1]: https://github.com/crunzphp/crunz/compare/v1.6.0...v1.6.1
Expand Down
58 changes: 58 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -555,6 +555,58 @@ Method `appendOutputTo()` **appends** the output to the specified file. To overr

It is also possible to send the errors as emails to a group of recipients by setting `email_output` and `mailer` settings in the configuration file.

## Realtime Output

By default a task's output is buffered and only emitted once the task has
finished. For long-running tasks (for example a worker that polls a queue in a
loop) this means no output is visible until the task ends, which makes
monitoring and debugging difficult, especially when logs are collected from
the process' standard output (e.g. Docker/CloudWatch).

Setting `output_realtime` to `true` streams each task's output to its
configured destination as soon as it is produced, instead of buffering it until
the task finishes:

```yaml
# Configuration settings

## ...
output_realtime: true
## ...
```

The output is streamed to the same destination it would normally be written to
at the end of the job, resolved in the usual order:

1. a per-event log file, if the task uses `sendOutputTo()`/`appendOutputTo()`;
2. otherwise the global `output_log_file`, if `log_output` is enabled;
3. otherwise the console (`stdout` for standard output, `stderr` for errors).

So a task logging to a file gets that file written **live** (line by line as it
runs), and a task whose output goes to `stdout` (e.g. for Docker/CloudWatch)
gets it streamed to `stdout` live. The output is forwarded **verbatim** — the
live stream is the raw task output, not the framed `[date] crunz.INFO: ...`
record that buffered mode writes. Because the destination receives the live
stream, the end-of-job framed record to that same destination is not written
again (it would duplicate the stream).

`email_output` is unaffected: it still sends the complete output once the task
finishes. The error reporting sinks (`log_errors` → `errors_log_file`,
`email_errors`) are also unaffected.

> Notes:
>
> - Realtime mode changes **when and in what form** output is written, not how
> much is held in memory: the output is still buffered internally so
> `email_output` can send the complete output at the end.
> - Only the task process' own output is streamed. Output echoed by
> `before()`/`then()` callbacks is still sent via `email_output` but is not
> part of the live stream.
> - If two sinks resolve to the same underlying stream — for example
> `output_log_file: php://stdout` together with a separate consumer of the
> runner's stdout — the output can appear twice on that stream. Point the log
> at a real file, or disable it, when you rely on the live stream.

## Error Handling

Crunz makes error handling easy by logging and also allowing you add a set of callbacks in case of an error.
Expand Down Expand Up @@ -861,6 +913,12 @@ errors_log_file:
# Set this to true if you want to keep the outputs
log_output: false

# By default a task's output is buffered and only emitted once the task
# finishes. Set this to true to stream the output to the console (stdout for
# standard output, stderr for error output) as soon as it is produced. Useful
# for long-running tasks whose logs would otherwise be invisible until they end.
output_realtime: false

# This is the absolute path to the global output log file
# The events which have dedicated log files (defined with them), won't be
# logged to this file though.
Expand Down
12 changes: 12 additions & 0 deletions resources/config/crunz.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,18 @@ errors_log_file: ~
# Set this to true if you want to keep the outputs
log_output: false

# By default a task's output is buffered and only emitted once the task
# finishes. Set this to true to stream the output to its destination as soon as
# it is produced, instead of buffering it. The output is streamed to wherever it
# would normally be written: a per-event log file (sendOutputTo), else the
# global output_log_file (when log_output is true), else the console (stdout for
# standard output, stderr for errors). The live stream is the raw task output
# (not the framed "[date] crunz.INFO:" record), and it replaces the end-of-job
# record for that destination. email_output and error reporting are unaffected.
# Useful for long-running tasks whose logs would otherwise be invisible until
# completion (e.g. when monitored via Docker/CloudWatch).
output_realtime: false

# This is the absolute path to the global output log file
# The events which have dedicated log files (defined with them), won't be
# logged to this file though.
Expand Down
8 changes: 8 additions & 0 deletions src/Configuration/Definition.php
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,14 @@ public function getConfigTreeBuilder(): TreeBuilder
->info('Flag for logging output' . PHP_EOL)
->end()

->booleanNode('output_realtime')
->defaultFalse()
->info(
'Stream task output to the console as it is produced'
. ' instead of only after the task finishes.' . PHP_EOL
)
->end()

->scalarNode('output_log_file')
->defaultValue('/dev/null')
->info('Path to output logs' . PHP_EOL)
Expand Down
25 changes: 25 additions & 0 deletions src/Event.php
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,13 @@ class Event implements PingableInterface
private ?LockFactory $lockFactory = null;
/** @var string[] */
private array $wholeOutput = [];
/**
* Optional sink invoked with each output chunk as it is produced, used to
* stream output in realtime instead of only after the process exits.
*
* @var \Closure(string, string):void|null
*/
private ?\Closure $realtimeCallback = null;
/** @var Lock */
private $lock;
/** @var \Closure[] */
Expand Down Expand Up @@ -295,6 +302,20 @@ public function wholeOutput()
return \implode('', $this->wholeOutput);
}

/**
* Register a sink that receives each output chunk as it is produced.
*
* The sink is invoked from the process output callback with the Symfony
* Process output type ('out'/'err') and the chunk content, enabling
* realtime streaming of output while it is still being generated.
*
* @param \Closure(string, string):void|null $callback
*/
public function setRealtimeCallback(?\Closure $callback): void
{
$this->realtimeCallback = $callback;
}

/**
* Start the event execution.
*
Expand All @@ -309,6 +330,10 @@ public function start()
$this->getProcess()->start(
function ($type, $content): void {
$this->wholeOutput[] = $content;

if (null !== $this->realtimeCallback) {
($this->realtimeCallback)($type, $content);
}
}
);

Expand Down
157 changes: 147 additions & 10 deletions src/EventRunner.php
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@
use Crunz\Logger\Logger;
use Crunz\Logger\LoggerFactory;
use Crunz\Pinger\PingableInterface;
use Symfony\Component\Console\Output\ConsoleOutputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Process\Process as SymfonyProcess;

class EventRunner
{
Expand All @@ -19,6 +21,14 @@ class EventRunner
/** @var Logger|null */
protected $logger;
private ?OutputInterface $output = null;
/**
* Open stream handles used to stream realtime output to a file/stream
* destination, keyed by the event's object id. Closed once the event is
* handled.
*
* @var array<int,resource>
*/
private array $realtimeStreams = [];

public function __construct(
protected Invoker $invoker,
Expand Down Expand Up @@ -75,6 +85,13 @@ protected function start(Event $event): void
$event->logger = $this->loggerFactory->createEvent($event->output);
}

// When realtime output is enabled, stream the task's output to its
// resolved destination (per-event file, global log file, or the console)
// as it is produced instead of only after the process exits.
if ($this->realtimeOutputEnabled()) {
$event->setRealtimeCallback($this->createRealtimeCallback($event));
}

$this->consoleLogger
->debug("Invoke Event's ping before.");

Expand Down Expand Up @@ -132,6 +149,9 @@ protected function manageStartedEvents(): void
$this->consoleLogger
->debug("Task <info>{$id}</info> status: {$runStatus}.");

// Close any realtime file stream opened for this event.
$this->closeRealtimeStream($event);

// Dismiss the event if it's finished
$schedule->dismissEvent($eventKey);
}
Expand Down Expand Up @@ -174,6 +194,17 @@ protected function invoke(array $callbacks = [], array $parameters = [])

protected function handleOutput(Event $event): void
{
// In realtime mode the task's output has already been streamed, as it
// was produced, to its resolved destination (per-event file, global log
// file, or the console). The end-of-job framed emission to that same
// destination is replaced by the live stream, so it is skipped here.
// email_output is additive and still sends from the retained buffer.
if ($this->realtimeOutputEnabled()) {
$this->emailOutput($event);

return;
}

$logged = false;
$logOutput = $this->configuration
->get('log_output')
Expand All @@ -195,15 +226,7 @@ protected function handleOutput(Event $event): void
$this->display($event->getOutputStream());
}

$emailOutput = $this->configuration
->get('email_output')
;
if ($emailOutput && !empty($event->getOutputStream())) {
$this->mailer->send(
'Crunz: output for event: ' . ($event->description ?? $event->getId()),
$this->formatEventOutput($event)
);
}
$this->emailOutput($event);
}

protected function handleError(Event $event): void
Expand All @@ -219,7 +242,10 @@ protected function handleError(Event $event): void
$this->logger()
->error($this->formatEventError($event))
;
} else {
} elseif (!$this->realtimeOutputEnabled()) {
// Without log_errors, a failed task's output is written straight to
// the runner's console here. In realtime mode that output already
// streamed live, so suppress this write to avoid duplicating it.
$output = $event->wholeOutput();

$this->output
Expand Down Expand Up @@ -269,6 +295,26 @@ protected function display($output): void
;
}

/**
* Email the event's output when email_output is enabled.
*
* Kept separate from the logging/display logic so it can run unchanged in
* realtime mode, where the live stream replaces the end-of-job logging but
* the retained buffer is still emailed.
*/
private function emailOutput(Event $event): void
{
$emailOutput = $this->configuration
->get('email_output')
;
if ($emailOutput && !empty($event->getOutputStream())) {
$this->mailer->send(
'Crunz: output for event: ' . ($event->description ?? $event->getId()),
$this->formatEventOutput($event)
);
}
}

private function pingBefore(PingableInterface $schedule): void
{
if (!$schedule->hasPingBefore()) {
Expand Down Expand Up @@ -311,4 +357,95 @@ private function logger(): Logger

return $this->logger;
}

/** Whether realtime output streaming is enabled via configuration. */
private function realtimeOutputEnabled(): bool
{
return (bool) $this->configuration
->get('output_realtime')
;
}

/**
* Build a sink that writes each output chunk to the event's resolved output
* destination as it is produced. When the destination is a file/stream path
* (a per-event `sendOutputTo()` target or the global `output_log_file`), the
* raw chunks are appended to that path. Otherwise output is written to the
* runner console, with standard output going to the main stream and error
* output to the error stream (when available). Chunks are written raw and
* without an added newline so task output is forwarded verbatim.
*
* @return \Closure(string, string):void
*/
private function createRealtimeCallback(Event $event): \Closure
{
$path = $this->realtimeStreamPath($event);
if (null !== $path) {
$stream = @\fopen($path, 'ab');
if (false !== $stream) {
$this->realtimeStreams[\spl_object_id($event)] = $stream;

return static function (string $type, string $content) use ($stream): void {
\fwrite($stream, $content);
};
}
}

$output = $this->output;
if (null === $output) {
return static function (): void {};
}

$errorOutput = $output instanceof ConsoleOutputInterface
? $output->getErrorOutput()
: $output
;

return static function (string $type, string $content) use ($output, $errorOutput): void {
$stream = SymfonyProcess::ERR === $type
? $errorOutput
: $output
;
$stream->write($content, false, OutputInterface::OUTPUT_RAW);
};
}

/**
* Resolve the file/stream path a task's realtime output should be streamed
* to, mirroring the success-path routing precedence: a dedicated per-event
* output file first, then the global output log file. Returns null when the
* output is not directed at a file/stream and should go to the console.
*/
private function realtimeStreamPath(Event $event): ?string
{
if (!$event->nullOutput()) {
return $event->output;
}

$logOutput = (bool) $this->configuration
->get('log_output')
;
if ($logOutput) {
$logFile = $this->configuration
->get('output_log_file')
;
if (\is_string($logFile) && '' !== $logFile) {
return $logFile;
}
}

return null;
}

/** Close and forget any realtime stream opened for the given event. */
private function closeRealtimeStream(Event $event): void
{
$id = \spl_object_id($event);
if (!isset($this->realtimeStreams[$id])) {
return;
}

\fclose($this->realtimeStreams[$id]);
unset($this->realtimeStreams[$id]);
}
}
Loading
Loading