Confirming gap in parish-level data for weeks 40-43 in 1665

[zross]

@hepplerj this a phenomenal resource, thank you for providing the data and keeping it so well organized!

I'm comparing the parish-level data (all_bills.csv) against the citywide data (causes_of_death.csv) to see how close the counts are and I'm seeing two gaps that I just wanted to confirm, I hope that's OK.

• I'm seeing that weeks 40-43 in 1665, right in the middle of the plague, have no parish-level data. Does that make sense?
• I'm seeing that weeks 1-13 (shown starting at 53 in the plot below) in 1666 are missing city cause of death data?

Is this correct? These data are do not exist?

[jmotis]

@zross we should have that data, I'll check what's in the original transcription csvs before we uploaded it into the data pipeline. We had a lot of overlapping partial datasets for this decade so it may be something got dropped out of the pipeline by accident.

[zross]

Wow, that would be amazing!

[jmotis]

@zross I've found the 1665 parish data in our original transcription data but it's in a terribly formatted Excel file. Happy to share the file with you if you need it promptly or I'm going to reformat and get the data back into the database today so we've fixed that hole if you can wait a bit.

For the 1666 weeks 1-13, it looks like they're in the upload csvs that start the data pipeline so there's probably a date conversion error somewhere. Long story short: New Year's Day is March 25th so the first 13 weeks of 1666, by our reckoning, are actually still 1665 according to the original dataset. So looking at the original data, weeks 1-13 of "1664" (week 1 is Dec. 20 to 27, week 13 is March 14 to 21) are actually the final week of 1664 and first 12 weeks of 1665, by our modern reckoning. Then weeks 1-13 of "1665" (week 1 is Dec. 19 to 26, week 13 is Mar 13 to 20) are the final week of 1665 and first 13 weeks of 1666, again by our modern reckoning. Looks like both these years have new years in week 14, so weeks 14-53 should read as normal.

I'm going to keep working on this after lunch, but just wanted to confirm for you that the data does exist!

[zross]

OK, great, yes, if you're working on this now I will wait for the Parish data.

And on the citywide data, I think what you're saying is that the data is there already but it may not be recognizable as 1666 1-13 to me so I can just review the data given your notes and track down those files. Correct?

[jmotis]

Yes, I suspect they ended up in the database as weeks 1-13 of 1665 but haven't had a chance to dig into that to confirm. I'm going to clean up the parish data first then come back to the city-wide causes of death.

[jmotis]

1665-missingweeks-cleaned.csv Here's what I'm sending into the data pipeline. I did a lot of manual conversions here we don't normally do, so it's not 100% a match for the headers in all data csv, but should match enough for you to be able to ingest it easily. I included weeks 39 and 44 as a sanity check to make sure I haven't messed up the data transformations.

[jmotis]

In case it helps, here's 1664-66-causes.csv with an extra column explicitly marking out "split year" notation vs. what's printed on the bill. This is the pre-pipeline data format, of just the cause data from 1664-65 (1665 by modern reckoning) and 1665-66 (1666 by modern reckoning). There's some data gaps due to illegible numbers, especially in the last two weeks of 1666 (bills 51, 52, which are bound into a volume in such a way that you can't read one of the columns of data).

[zross]

@jmotis ok, great, thank you! It looks like I can use 1665-missingweeks.cleaned.csv to patch the parish-level missing data from 1665. Exciting! That was the most important gap in the data.

I'm not entirely clear about how to use (or whether I need) 1664-66-causes.csv -- it does not seem to fill the city-wide gaps right? It does not have 1666 weeks 1-13 (modern reckoning). I'll ignore for now.

I'm seeing that @hepplerj has opened a few new issues -- #242, #243, #244. Do you know if these will affect my computation of plague deaths 1665-1666 given that I'm using all_bills.csv and causes_of_death.csv files directly (along with parishes.csv, and weeks.csv for reference).

[jmotis]

The 1664-66-causes.csv has 2 weeks of modern-1664 data (row 2, 3) which are the first two weeks of the 1664-65 bill cycle; then 52 weeks of modern 1665 data (through row 57, including rows 56 and 57 which are bills 1 and 2 of the 1665-66 cycle); then the next weeks are modern 1666 reckoning even though they're still labelled as 1665. does that make sense?

[hepplerj]

Hi @zross -- yes, I have a PR that I should be able to merge today that'll change all_bills.csv, causes_of_death.csv, weeks.csv and parishes.csv as a result of correcting some of the data transformation logic.

[zross]

Great, thanks. I will wait to do more until you merge in the PR. I assume I should still apply the patch that @jmotis provided for the parish-level data in 1665 right?

[zross]

The 1664-66-causes.csv has 2 weeks of modern-1664 data (row 2, 3) which are the first two weeks of the 1664-65 bill cycle; then 52 weeks of modern 1665 data (through row 57, including rows 56 and 57 which are bills 1 and 2 of the 1665-66 cycle); then the next weeks are modern 1666 reckoning even though they're still labelled as 1665. does that make sense?

@jmotis -- yes, I think this makes sense. So while the rows in green in my image below are printed year 1665, they are actually records for 1666.

It sounds like I now have patches for both of the main gaps, thank you! I'll wait on @hepplerj before finishing the data assembly.

Note that there are a few more gaps that are less important from my perspective but might matter to you. If you look at the plot at the top in 1666 there are a few more single week gaps in both parish and city-level data. I'm guessing at least one is due to the fire.

[hepplerj]

Okay, @zross, the fixes should be in place!

[zross]

@hepplerj great! I've pulled and re-run things on my side. I still see the gaps I reported which, I believe, is expected. You fixed other things? But I just want to confirm that this is correct and that I should use the files from @jmotis to apply the temporary gap fix right?

[Editing this note, it's possible I didn't sync your changes so the question is still valid, but it's possible the reason I don't see the gaps filled is because of a failed sync, I will look tomorrow]

[zross]

@hepplerj @jmotis Ok, I updated with your changes and the parish-level data is indeed fixed! Image below. Nice. For the citywide data I assume I should use the file that @jmotis provided above and will go ahead with that unless you tell me otherwise.