If the relative reliability of the SD card is an underlying cause (e.g. tons of writes wearing them out early causing corruption) you might also consider eMMC chips on SD card adapters. A discussion here: https://jamesachambers.com/emmc-to-sd-card-adapters-explaine...
Seems like it's worth considering UPSes for the controllers as well? Depends on the details of the power outages, and the draw of the controllers, and if the controllers handle the equipment being unpowered even though the controller is.
Some of the Wi-Fi enabled SD cards you're mentioning (older models by now, but might still be available on eBay & stuff) can be rooted and you can get a shell. Example: https://gist.github.com/deckar01/6d9b76bdef21eaab0568 (I'm sure there are more models that are vulnerable to similar attacks)
One thing I would suggest keeping in mind with any kind of interference (regardless of how stable it is and how well it adheres to the SD spec & maintains data integrity) is that if something happens to/with the machinery this controller is handling, your modification can easily be blamed by the vendor, even if your modification isn't actually at fault and it's the vendor's shitty software that's at fault. The risk is double considering it doesn't even seem to be able to manage not corrupting a standard SD card.
Thank you! I had no idea those SDs are hackable. We'll look into it as well. Would be nice to get ahold of a hundred SD on ebay. I'll search tomorrow. I like their form-factor and that it fits in SD slot tightly, very clean and no need to fit another box in the control cabinet.
Your point regarding liability: as-is systems are unreliable already so much that the manufacturer selling support contracts to firms like mine, so we to improve reliability through humans driving around and fixing&replacing them. The HW manufacturer has insane lucrative contract themselves and so now they need to fix things up so they re-contracting support of their shitty HW down the chain. Can't say more due to NDA. So "broken" state of controllers is already has, so we are not making any new hardware failure points. I did talk to my lawyer, he said my plans are ok. Thank you for your comment!
Just to clarify, my point about liability was more about if someone gets hurt by the controlled and it goes to (potentially criminal) court for negligence. In this case there is going to be a witch hunt and everyone will try to deflect blame, thus your non-standard modification will be under much scrutiny regardless if it actually played a role. In fact judging by the original reliability (or lack thereof) of the system, you might get blamed for failures that don’t actually have anything to do with your modification.
If there’s no risk of injury/death then the stakes are much lower and indeed since the vendor software itself is already shitty you can’t really make it any worse.
To add to my original comment, if you want to pursue the Wi-Fi SD card route, I suggest using any of the known vulnerabilities to get root on the card and then reverse engineer the card (as in how the SD side is driven) from the inside.
This would effectively let you skip the whole “build a device that emulates an SD card” part.
From there I’d suggest building a Linux image from scratch using Buildroot or Yocto, so you start with a fresh and modern base and don’t have to fight with the SD vendor’s firmware or deal with their vulnerabilities (which might be a liability in your case).
Feel free to get in touch (email in my profile) if you want more guidance.
My experience as a user may be something to consider.
There can be a big difference in performance between different kinds of SD cards, and especially the way they are formatted.
For one thing there is the SD Association, and their ever-evolving specs as well as their own proprietary (!, from Tuxera apparently since 2017) ever-evolving formatting tool available for download free on their website.
Still uses "regular" FAT32 for sizes from 2GB to 32GB, and exFAT from 32GB upward.
This can be good for things from manufacturers that might be expected to conform to the specification as it was at the time, whether intentional or not.
Sometimes it is worthwhile to compare the factory card layout/format (and back it up for possible reinstatement) against that obtained after using various alternative formatting tools. Many times a subsequent tool will mimic the layout of the previous format unless the medium is zeroed or cleaned before re-partitioning & re-formatting.
Also, for some hardware having limited software capabilities like cameras, it can be best to partition, format, and error-check using a full PC in a repeatable way before or after storing material on the SD card. OTOH, sometimes the only way a camera will access a card is if it is formatted on-board using it's own simplified algorithm, after which it may or may not be able to be read or written by all PCs. You should be able to select your partitioning tools and formatting tools independently from each other for each type hardware, filesystem and application. Not always true in practice.
In some of these cases one strategy that sometimes seems best (and could be considered along the Windows mainstream) is to use recent Windows to Clean & partition and format the drive. Especially for FAT32 or NTFS use. A built-in card reader on something like a laptop can be ideal. Or it can be garbage. For even more reliability with FAT32, you might go as far as booting to the DOS from W98SE to reformat /quick which in DOS also allows you to fully label the volume. This may not actually be "quick" if you don't have a rapidly-bootable DOS USB stick handy (or if your floppy has too many cobwebs), but generally you get the full 20th century FAT32 shebang like nothing ever since, with backup boot sector to boot, and with /quick, it doesn't start from scratch. The DOS Quick format (if it can) will be overlaid on the existing 21st century FAT32 layout that Windows 11 (or some other 21st century formatter) thinks is optimized for the media these days. So FAT32 pre-formatted on NT6 (or something), rather than having DOS try to figure it out on virgin solid-state media. Then finish it off by reformatting Quickly in DOS. That's why they call it FAT32 ;)
For Windows users other than employing exFAT for larger drives, it can also be good to try putting 2 or more partitions on an SD card, the first a 32GB FAT32, followed by an NTFS volume(s) taking up the remainder of the drive. Or even a number of 32GB volumes, FAT32 or otherwise. Now that Windows 10/11 has been able to access more than just the first partition on a removable drive.
Also, one investigative tool you may find interesting is called flashbench. It's a little bit arcane and did what it was supposed to do years ago, hasn't been updated since.
If the relative reliability of the SD card is an underlying cause (e.g. tons of writes wearing them out early causing corruption) you might also consider eMMC chips on SD card adapters. A discussion here: https://jamesachambers.com/emmc-to-sd-card-adapters-explaine...
https://hackaday.com/2022/08/08/automated-microsd-card-swapp...
https://hackaday.com/2014/06/08/the-in-circuit-sd-card-switc...
more links: https://hackaday.io/project/19783-sd-card-emulation
Thank you!! I believe I found just the product we need in the comments under the first article: https://fiber-punk.com/products/node-max-by-fiberpunk It's for 3d printers but it should work for us too
glad that helped
Seems like it's worth considering UPSes for the controllers as well? Depends on the details of the power outages, and the draw of the controllers, and if the controllers handle the equipment being unpowered even though the controller is.
you need to use high quality sd cards such as sandisk extreme pro
also think about putting the machinery on some kind of UPS or backup power supply so it can be shut down properly
WD / Sandisk make Industrial MicroSD cards as well meant for this kind of application, you can get them from Digikey and Mouser.
Some of the Wi-Fi enabled SD cards you're mentioning (older models by now, but might still be available on eBay & stuff) can be rooted and you can get a shell. Example: https://gist.github.com/deckar01/6d9b76bdef21eaab0568 (I'm sure there are more models that are vulnerable to similar attacks)
One thing I would suggest keeping in mind with any kind of interference (regardless of how stable it is and how well it adheres to the SD spec & maintains data integrity) is that if something happens to/with the machinery this controller is handling, your modification can easily be blamed by the vendor, even if your modification isn't actually at fault and it's the vendor's shitty software that's at fault. The risk is double considering it doesn't even seem to be able to manage not corrupting a standard SD card.
Thank you! I had no idea those SDs are hackable. We'll look into it as well. Would be nice to get ahold of a hundred SD on ebay. I'll search tomorrow. I like their form-factor and that it fits in SD slot tightly, very clean and no need to fit another box in the control cabinet.
Your point regarding liability: as-is systems are unreliable already so much that the manufacturer selling support contracts to firms like mine, so we to improve reliability through humans driving around and fixing&replacing them. The HW manufacturer has insane lucrative contract themselves and so now they need to fix things up so they re-contracting support of their shitty HW down the chain. Can't say more due to NDA. So "broken" state of controllers is already has, so we are not making any new hardware failure points. I did talk to my lawyer, he said my plans are ok. Thank you for your comment!
Just to clarify, my point about liability was more about if someone gets hurt by the controlled and it goes to (potentially criminal) court for negligence. In this case there is going to be a witch hunt and everyone will try to deflect blame, thus your non-standard modification will be under much scrutiny regardless if it actually played a role. In fact judging by the original reliability (or lack thereof) of the system, you might get blamed for failures that don’t actually have anything to do with your modification.
If there’s no risk of injury/death then the stakes are much lower and indeed since the vendor software itself is already shitty you can’t really make it any worse.
To add to my original comment, if you want to pursue the Wi-Fi SD card route, I suggest using any of the known vulnerabilities to get root on the card and then reverse engineer the card (as in how the SD side is driven) from the inside.
This would effectively let you skip the whole “build a device that emulates an SD card” part.
From there I’d suggest building a Linux image from scratch using Buildroot or Yocto, so you start with a fresh and modern base and don’t have to fight with the SD vendor’s firmware or deal with their vulnerabilities (which might be a liability in your case).
Feel free to get in touch (email in my profile) if you want more guidance.
No MITM from me.
You've got some good advice from professionals.
My experience as a user may be something to consider.
There can be a big difference in performance between different kinds of SD cards, and especially the way they are formatted.
For one thing there is the SD Association, and their ever-evolving specs as well as their own proprietary (!, from Tuxera apparently since 2017) ever-evolving formatting tool available for download free on their website.
https://www.sdcard.org/downloads/formatter/faq/
Lots of users are not aware of this.
Still uses "regular" FAT32 for sizes from 2GB to 32GB, and exFAT from 32GB upward.
This can be good for things from manufacturers that might be expected to conform to the specification as it was at the time, whether intentional or not.
Sometimes it is worthwhile to compare the factory card layout/format (and back it up for possible reinstatement) against that obtained after using various alternative formatting tools. Many times a subsequent tool will mimic the layout of the previous format unless the medium is zeroed or cleaned before re-partitioning & re-formatting.
Also, for some hardware having limited software capabilities like cameras, it can be best to partition, format, and error-check using a full PC in a repeatable way before or after storing material on the SD card. OTOH, sometimes the only way a camera will access a card is if it is formatted on-board using it's own simplified algorithm, after which it may or may not be able to be read or written by all PCs. You should be able to select your partitioning tools and formatting tools independently from each other for each type hardware, filesystem and application. Not always true in practice.
In some of these cases one strategy that sometimes seems best (and could be considered along the Windows mainstream) is to use recent Windows to Clean & partition and format the drive. Especially for FAT32 or NTFS use. A built-in card reader on something like a laptop can be ideal. Or it can be garbage. For even more reliability with FAT32, you might go as far as booting to the DOS from W98SE to reformat /quick which in DOS also allows you to fully label the volume. This may not actually be "quick" if you don't have a rapidly-bootable DOS USB stick handy (or if your floppy has too many cobwebs), but generally you get the full 20th century FAT32 shebang like nothing ever since, with backup boot sector to boot, and with /quick, it doesn't start from scratch. The DOS Quick format (if it can) will be overlaid on the existing 21st century FAT32 layout that Windows 11 (or some other 21st century formatter) thinks is optimized for the media these days. So FAT32 pre-formatted on NT6 (or something), rather than having DOS try to figure it out on virgin solid-state media. Then finish it off by reformatting Quickly in DOS. That's why they call it FAT32 ;)
For Windows users other than employing exFAT for larger drives, it can also be good to try putting 2 or more partitions on an SD card, the first a 32GB FAT32, followed by an NTFS volume(s) taking up the remainder of the drive. Or even a number of 32GB volumes, FAT32 or otherwise. Now that Windows 10/11 has been able to access more than just the first partition on a removable drive.
Also, one investigative tool you may find interesting is called flashbench. It's a little bit arcane and did what it was supposed to do years ago, hasn't been updated since.