This is a note for software engineers out there.
I have moved on to rework the existing code. The processing is done on a dedicated server (32GB of memory, SSD drives for SQL Server. A little over 15,700,000 combinations of taxon needs to be made. Each combination needs to get all data on these taxons that are concurrent in any samples.
- I.e. All samples that have Taxon 123 and Taxon 432 being reported.
The computations are done using Parallel and Concurrent libraries. All indices have been tuned for this analysis. The SQL Server and the utility to calculate the associations are on the same server – so no latency or network impact.
I found that the CPU temperatures exceed the maximum recommended for the CPU chip.
2022-03-27 20:12:08 Bacteria2Bacteria - 15,717,260 Items to Inspect 2022-03-29 22:48:34 Bacteria2Bacteria - 15,717,260 Items Inspected 2022-03-29 22:48:47 805,770 Items found
This analysis likely qualifies for the “big data” label. So why is it worth it? The answer is simple, for some bacteria we have no information on what increases or decreases it. If we know which bacteria is associated with the bacteria growth or reduction, then we can synthesize modifiers of these bacteria that we lack information on.
This has not been implemented in suggestions (and will likely be available at the nerd level when it is).