Brainstorming issue for better comparison functionality

[mre]

A few users mentioned that we should focus on better support for comparing tools.
This is new ground for us and we'd like to get some early feedback from the community to see what is really needed here.
Some ideas we thought about so far:

• Filter by license (proprietary vs open source)
• Filter by type (formatter, linter, metalinter)
• Filter by runtime (commandline, web service, IDE plugin,...)
• Filter by pricing model (one-time costs, monthly fees, free for open source,...)
• Filter by mode of execution: AST-based, rule-based

Thoughts on the above? Anything else that comes to mind?
We're thankful for any input.

[archi]

Hey, thanks for linking here from HN ;) Just a small disclaimer: I work in the field, but this is my personal opinion and not necessarily that of my employer :) Though I'm probably biased.

1. Some companies offer academic licenses as an alternative to expensive "normal" licenses, so for people at universities and associated academia this might also be interesting (as per the blurbs, we're not the only ones in that regard).
2. For commercial offerings, customer support is an important point. Maybe it's interesting what's offered here, e.g. analysis service, qualification support kits or porting a binary analyser to a new architecture. (From the current list, this seems to be neglected - but I believe that, for industrial users, this is very relevant).
3. When using static analysis to certify safety critical systems, the standards covered is also important in certain industries (e.g. DO-178B).
4. How to compare sound (what we do) vs. unsound? E.g. proving properties (like absence-of-runtime-errors) vs. "the property is likely to hold"?
5. Our binary analysis tools (currently not listed) offer several types of analyses (control flow reconstruction, value analysis, worst case execution time, stack usage and some more). Since I'm working on that product I was interested what else you had listed, but right now it's not really possible to filter by e.g. "all WCET tools". Some tools have that kind of info in the blurb, but for e.g. IDA free the "Binary code analysis tool." blurb is rather vague and if I didn't already know IDA, I'd need to check the website to figure out that it's something else.
6. Regarding binary analysis, it's probably interesting which architectures are covered (for us, the 500 character limit for the blurb would probably be not enough :P)

//edit: Ah, regarding this portion of your reply, "or wants to help build it": The part of my free time that's "allocatable to FOSS work" is currently locked to some other project(s) ^^"

[mre]

Thanks for the input @archi.

1. That would be really valuable information, yes. The question is how we can model that in the underlying YAML file here. One idea would be a special_offers field, that could contain ["oss", "academia"]. The question is if that information alone would be helpful. Most like people would want to know what exactly the offer is, which would require quite a bit more research and be harder to standardize.
2. Yes. Problem again is how to come up with a standardized set of "tags" here. Might be a bit much for now unless it can be automated and unified somehow.
3. Good point. I have zero knowledge about standards at the moment, which I'm planning to change soon. We've heard this a few times now, so it might be really important to more experienced users and industry specialists. Can you recommend an up-to-date list of such standards and certifications?
4. No idea. Open for suggetions.
5. Could you add the binary analysis tools you mention? We have a binary section, which might be a good fit. It would greatly help us complete that section. The information right now is quite rudimentary. I suggest to add WCET to the description as it will show up in the Algolia search results for now. Once we find that more tools have that property in the description, we can go and formalize it as a field in the YAML.
6. The architectures might be a bit of a special case for the binaries. I'd personally start by adding the most common architectures in the in the blurb for now. Wouldn't be complete but help with discovery.

No worries if you don't personally find the time to contribute. Your input is quite valuable already. Maybe someone else will be inspired and come forward to add more suggestions or help with the implementation.

[archi]

1. I'd expect an array/list of license tags, allowing for proprietary, academia (but this would ultimately be the same as proprietary: true plus some special_offers array/list of tags).
2. Similarly, maybe add an array of commercial_support tags, and define the possible tags somewhere? I think it's difficult avoiding to manually collect this from websites, or outright ask companies.
3. I work on the analysis backend, so I am not involved in that part ;-) We have an incomplete list on our website, scroll down to "Qualification support". Maybe also check other companies.
4. Maybe add a sound tag? Plus, generally, when a user clicks on a tag, not only list all tagged applications, but also an explanation of the tag.
5. I forwarded this to a colleague.
6. Maybe add an array architectures, and a per-architecture tag (riscv, aarch64, powerpc,...). Though that's a deep rabbit hole (e.g. we differentiate between ppc architectures again).

[AristoChen]

I was looking for a tool for unit test few days ago, and then I found this cool website, which compare between different library, maybe that will help?

[mre]

That looks like a great product to me. We've heard from some people that they are interested in comparing two tools like that. It might be a lot of work, though. It's probably worth it in the long run.