How to handle patches + scrapper

[ForwardFeed]

As we've discussed previously we for now aim to use patches to update the database.
I planned to start to make the web scrapper repository, then I wondered how does the output should've looked like.

How do we handle patches since all data is scattered across multiple files.

• Reviewing the patch like a PR, so in order to submit a patch, one should have to modify all files of the repository and send it as a PR
• Having a patch type and submit the patching in one file, an automation script will convert that to the structure we have.

The first one is pretty simple and less work.

The second one could introduce some interesting opening, for example the ability to extend the formatting and to add extra text solely for the purpose of explaining some values,
for example a patch JSON with a value commented

{
    "value": 15.0,
    "comment": "This actually could be given as set as a range as the source cites multiple values, like 15.5, 16.5? but maybe we should just keep the average or the room temp value first"
}

And then the automation will strip the comment when inserting in the database

also should we keep all patches in memory? to have a sort of historic of it?

[komed3]

Hmm, both solutions are quite sensible. Considering references that must/should be set to properties, the second solution might be a more viable approach. Also allowing for flexibility, even when structural changes are required.

Such an approach is likely to handle parallel PRs and duplicate values more effectively. However, I don't know if it's worth the extra effort. Because there will probably be only few overlapping PRs.

If we decide on the second solution, storing the patches as individual files in a separate folder (such as patches/history or something similar) is easy to do. The good thing is that Git has pretty powerful version control, meaning even without permanent storage, changes remain traceable.

[ForwardFeed]

Quotes in API is something that annoys me a fair lot too.
Even if nowadays nobody is writing request with bare tools.
So escaping quotes shouldn't be a modern problem.
But what do I know.

maybe rather use $_ instead? or $$_ maybe just $$ ?

$$where: {
    
},
$$_where: {

},
$_where: {

},
$where: {
    
}

[komed3]

Those are nice suggestions. I always find that quotes make it a bit more clumsy and slow down coding/writing.

But before I can revisit and improve the API, I need to finally implement the DB layout here so we can use it and you have access to it too. Not much will change, just some renaming here and there. Everything else is just copypasta. 😏

[ForwardFeed]

I saw you scraped away the work you did, was there something deeply off with that design?

[komed3]

Not really, but I'd want to fix a few things, like adding references for hazard information. Besides, I'm working on a list of all the physical and chemical properties so I can make a consistent and accurate mapping of those.

After two weeks of intense work, I've decided to take a break. Sometimes it's helpful to step back to get a fresh approach. But, towards the weekend, I'll continue my work. The abstract types have already been moved to a package for clarity.

[ForwardFeed]

I agree that you work a lot, so making a pause here definitively makes sense.

I checked https://github.com/komed3/devtypes/tree/master and I definitively want to use that module on my next TS projects.
Some QoL types I never thought of making.
It actually may render JS schema obsolete.