Use Vector Remap Language (VRL) to parse out fields from logs

[BryceStevenWilley]

We've successfully got logs flowing into our Grafana instance, but there's still some additional parsing of fields that we could add to the logs themselves to make it easier to read through and understand the logs in Grafana.

The Fly log-shipper uses Vector to handle the log events, which uses Vector Remap Language (VRL) to parse out the text from the actual log content into JSON fields in the log event.

Tasks:

• change the log format in logback. xml (

  EfileProxyServer/proxyserver/src/main/resources/logback.xml

  Line 5 in d5a8dc9

  <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [server=%X{serverId}] [user=%X{userId}] [session=%X{sessionId}] [corr=%X{correlationId}] [req=%X{requestId}] [op=%X{operation}]%n %-5level %-35logger{35} - %msg %rEx%n</pattern>

  ) to output everything in a JSON format instead of a log line.
  • Slightly worse local development, but worth it IMO.
  • Specifically make sure that exceptions print all to a single line instead of breaking it up into multiple lines
• write the VRL script to add the fields we care about:

   . = parse_json!(string!(.message))
   del(.userId) # better for privacy, less useful than session ID I think
   # is there a way to make the exceptions show new lines again, so they don't show up all on one (wrapped) line in grafana?
  

• figure out where to put that VRL script in the log shipper

[rajeswari1301]

PR #429 is up for the logging change (task 1).

For task 2/3: the VRL script was already there on the efsp-log-shipper machine, in /etc/vector/sinks/loki.toml not tracked in any repo. It already parsed the JSON message and merged fields in, just missing del(.userId). Added that and deployed it and confirmed that loki sink restarted clean and is pushing data.

Worth doing at some point: check that file into a repo somewhere so it's not lost if the machine gets rebuilt.

Grafana's Logs panel has a built-in prompt for that once a field has real newlines in it, which it will once #429 gets deployed.
Next: check real Grafana output to confirm levels/fields/no userId show up as expected, then update the alert to use a real level label instead of text-matching "ERROR"