Repo-Mirror/J2B-Kas-tle
9
0
Fork 0
mirror of https://github.com/Kas-tle/java2bedrock.sh.git synced 2025-12-20 07:09:17 +00:00
Code Issues Releases Activity

converter.sh: Don't parallelise write_hash (#17742)

Browse Source 
write_hash appends to scratch_files/hashes.csv without any form of
locking, but is performed in parallel. Additionally there is no
synchronisation to wait for all write_hash invocations to have completed
before reading the CSV. This results in non-deterministic breakage and
missing hashes in the resulting files.

Remove parallelism here to fix the breakage.

Note, just calling wait_for_jobs after the loop reduces the occurrence
of problems, but is insufficient since multiple appends do occasionally
race and hashes may still be missing.
This commit is contained in:
James Hogan
2023-12-24 22:40:27 +00:00
committed by GitHub
parent a6c8903f84
commit 57cc7ddf1b
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
converter.sh
View File

@@ -296,7 +296,7 @@ function write_hash () {
} }
while IFS=, read -r gid predicate path while IFS=, read -r gid predicate path
do write_hash "${predicate}" "${path}" "${gid}" "scratch_files/hashes.csv" & do write_hash "${predicate}" "${path}" "${gid}" "scratch_files/hashes.csv"
done < scratch_files/paths.csv > /dev/null done < scratch_files/paths.csv > /dev/null
jq -cR 'split(",")' scratch_files/hashes.csv | jq -s 'map({(.[0]): [.[1], .[2]]}) | add' > scratch_files/hashmap.json jq -cR 'split(",")' scratch_files/hashes.csv | jq -s 'map({(.[0]): [.[1], .[2]]}) | add' > scratch_files/hashmap.json