Database copy error crash

[jchu-sandag]

Proposed changes

Edits the copy tool such that the script uses a Windows robocopy in the event that the EMME database cannot be read by the EMME script.

Impact

Recent runs of ABM have all resulted in an error in which the EMME databases could not be read by the EMME script on the remote drive. Thus, the databases could not be copied over from the local drive to the remote drive, forcing users to manually go into the system and close the programs before copying over the rest of the EMME databases. This caused downtime in metrics that required these databases to be within the remote drive. This update will instead initiate a Windows robocopy in the event that the EMME script fails to copy over all the databases to the remove drive.

Types of changes

What types of changes does your code introduce to ABM?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

How has this been tested?

• Full 2022 scenario run
• Checks for byte size between both remote and local drive
  • This check has not been checked within a scenario run and has only been checked by manual copy locally

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings

Further comments

Although not the explicit usage of the following method, the _copy_dir method is a good general copy method for instances where other copying methods fail or run into errors. In the future, an additional update to separate the file masks into file_masks and dir_masks could be beneficial in case any specific directories and files would want to be excluded in a single step. However, this update is not within the scope of this PR and would require further testing.

[cundo92]

@jchu-sandag thanks for working on and applying this change. I was able to confirm successful copy of a full model run, including the emme_project directory.

[aber-sandag]

I am somewhat concerned about the effects of overwriting Emme files that are currently in use by Emme. Could you test the following:

• The file copy output shows that all files were copied successfully when copying emme_project with robocopy? Are there any warnings?
• After running, when the project is opened from T drive on another server, can Emme matrices and other Emme data be accessed in Emme?

[cundo92]

I am somewhat concerned about the effects of overwriting Emme files that are currently in use by Emme. Could you test the following:

• The file copy output shows that all files were copied successfully when copying emme_project with robocopy? Are there any warnings?
• After running, when the project is opened from T drive on another server, can Emme matrices and other Emme data be accessed in Emme?

@jchu-sandag feel free to use my test run at "T:\STORAGE-63T\car_tests\log_test\2022_copyTest" for these follow up comments

[JiaXu1024]

I am somewhat concerned about the effects of overwriting Emme files that are currently in use by Emme. Could you test the following:

• The file copy output shows that all files were copied successfully when copying emme_project with robocopy? Are there any warnings?
• After running, when the project is opened from T drive on another server, can Emme matrices and other Emme data be accessed in Emme?

@jchu-sandag feel free to use my test run at "T:\STORAGE-63T\car_tests\log_test\2022_copyTest" for these follow up comments

Hi @jchu-sandag , have you done the test mentioned in Alexander's comments? Any updates?

[jchu-sandag]

While performing some test scenarios, some runs crashed with errors including a Fatal Error 5 (regular scenario run with no update to the script) and connection error to EMME (scenario run with updates from the current PR). These runs were done on two separate servers but both occurred over the weekend run. All TOD databases and transit databases are within the local drive, so it is possible that the scenario run update completed a run but was corrupted during the copy step. @aber-sandag proposed a possible reason for why this might have happened may be the logbook\project.mlbk was copied over the file was still being written into.

More tests will be run to see if this is just a onetime error copying over from local to remote drive or if it may be a common error that will occur. (It's likely that this may be due to a corrupted file during a copy step, since neither the logbook nor the databases can be accessed using the EMME software. However, previous runs done by @cundo92 and I have successfully copied and deleted the files from the local drive and can also be accessed from different servers). Alexander also proposed to retry attempts to close the EMME modeler using methods _m.Modeller().desktop.project.save() and _m.Modeller().desktop.close() to mimic what is done during a manual copy. This should be done to ensure that nothing is being written into or corrupted while the emme_project files are being copied over from local to remote drive.

[JiaXu1024]

@jchu-sandag Have you had a chance to run additional tests to determine whether this was just a one‑time error when copying from the local to the remote drive, or if it’s something that might occur more consistently?
@aber-sandag Could you take another look and add your comments?

[bhargavasana]

@jchu-sandag please provide an update.

cc @cundo92 @zouyang401

[jchu-sandag]

Hi y'all. A total of seven scenario runs has been reviewed using this update. Overall, there were not any major issues that came up and all of the runs ran to completion. Here is a summary of the results:

• All the runs (with the exception of two runs) copied over the exact same bytes from the database files over to the remote drive.
  • The two exceptions included scenario 1258 which contained one extra file within all 6 databases (most likely a macro used by Bently to check the licensing agreement for each server) and scenario 1260 which did not run into a copy error at all (it did not need to go through a Windows robocopy and instead finished using EMME's methods).
• Most of that were specified to delete intermediate files were able to delete the scenario from the C drive. The exception included scenario 1235 which left over the analysis and results directory. However, none of the other runs that have occured afterwards have seem to run into this issue.
• The error referred to in Database copy error crash #432 (comment) has not been seen again. It may be possible that something happened to the server as the logbook was being written into or the hypothesis that @aber-sandag proposed. There hasn't been a case of this happening again however, so it may be a rare occurrence.
• The scenarios were able to be read and opened in a separate server with separate users after being copied over. There were no issues accessing a scenario from other users/servers.

More detailed information about each run can be viewed here: I believe we can close out this request unless @cundo92, @JiaXu1024, or @aber-sandag have anything else to add.

[cundo92]

Hi y'all. A total of seven scenario runs has been reviewed using this update. Overall, there were not any major issues that came up and all of the runs ran to completion. Here is a summary of the results:

• All the runs (with the exception of two runs) copied over the exact same bytes from the database files over to the remote drive.

  • The two exceptions included scenario 1258 which contained one extra file within all 6 databases (most likely a macro used by Bently to check the licensing agreement for each server) and scenario 1260 which did not run into a copy error at all (it did not need to go through a Windows robocopy and instead finished using EMME's methods).

• Most of that were specified to delete intermediate files were able to delete the scenario from the C drive. The exception included scenario 1235 which left over the analysis and results directory. However, none of the other runs that have occured afterwards have seem to run into this issue.

• The error referred to in Database copy error crash #432 (comment) has not been seen again. It may be possible that something happened to the server as the logbook was being written into or the hypothesis that @aber-sandag proposed. There hasn't been a case of this happening again however, so it may be a rare occurrence.

• The scenarios were able to be read and opened in a separate server with separate users after being copied over. There were no issues accessing a scenario from other users/servers.

More detailed information about each run can be viewed here: I believe we can close out this request unless @cundo92, @JiaXu1024, or @aber-sandag have anything else to add.

Nothing else on my end. Thanks for the thorough testing, documentation, and logs @jchu-sandag!