I’m very much a Scala rookie, only a little experimentation so far. I have an idea for something for work, which will require an arbitrary and variable number of computers to communicate with each other. The communication I do not expect to be very complex, just some messages about the status of certain tasks completed, not completed, or no information (because computer is shut down, frozen, etc.). I have been fascinated with Scala for quite a while, and it occurs to me that this may be an opportunity to finally do something serious and useful with it.
The gist is sort of a peer-to-peer network of modules that have a common internal list of tasks all over the network, along with some general information about when each task is supposed to run and what it is supposed to do. The app instance on each machine would regularly check the list and itself. If there is something on the list that is supposed to happen on the machine it is running on (possibly also on some other machines - not sure about that yet), the app would check to see if things had gone according to plan, and both note those results in itself, and broadcast that information to any other instances running on other machines elsewhere on the network. All other instances would capture and store that information. If the app determines that something had not happened which was supposed to happen, or had results other than those expected, it would also note and broadcast that information, and would send an email to some pre-determined address(es). The app would take note of information that should have been received from other machines, but was not, and send emails out that this was the case, theoretically covering machines that have crashed, or virtual machines that have stopped functioning and nobody noticed, or cases of the app itself not functioning on some machine where it should be running.
The impetus behind this is several minor disasters - a virtual server that was running well, but some automatic OS upgrade lunched it, and nobody noticed until all the functional backups had rolled off the back end, and only backups of the destroyed system were left, and most recently, an automatic backup script that stopped working, unknown to anyone, because the resident anti-virus program suddenly decided that the backup script was dangerous and erased it. Nobody knew about either of these until long after the problem had happened, because humans constantly manually checking on the results of automation that functions normally for weeks, months, years sometimes, just isn’t practical. People get lax, since it’s always fine, and eventually completely forget that the system even exists, until it’s suddenly needed, and…
Has anyone used Scala for something like this? Is it a good choice for such a task? Does it have the proper tools for such broadcasting and receiving information? Can anyone point me to any resources that would be helpful?