From making compilers smarter, going bottom up is not the optimal direction. You want to start with a language that is already by definition memory safe as memory is handled for you (which is basically any of the higher level languages), and then optimize that code down to assembly through recognition of patterns of memory usage, a lot of which can be done through static analysis (i.e looking for when new memory would be handled under the hood, and then tracing who accesses that memory)
Alternatively, if you want to work at a low level, all you would really need to do is make malloc and free smarter, because "memory unsafe" code is basically a case of double free in 99% of the time, as other things are either covered for by existing things like stack canaries, non executable memory sections, retpolines that protect against ROP chains, and so on. So just making sure that when free is called, it actually has permission to deallocate the chunk is all you need. You can do some of that statically, but to fully cover it, you would have some more arguments to the malloc and free call that establish permissions.
As an aside, the latter is why Rust doesn't really have a future. Its just popular enough these days to where its seen as taboo to hate on it because "memory safety", but the reality is, most of the existing well written C/C++ code is memory safe through very basic structuring of programs, and without the cumbersome syntax of Rust borrow checking that allows dev to happen much faster, considering any large codebase in Rust has a shitload of unsafes scattered about due to need to interact with posix libraries.
From making compilers smarter, going bottom up is not the optimal direction. You want to start with a language that is already by definition memory safe as memory is handled for you (which is basically any of the higher level languages), and then optimize that code down to assembly through recognition of patterns of memory usage, a lot of which can be done through static analysis (i.e looking for when new memory would be handled under the hood, and then tracing who accesses that memory)
Alternatively, if you want to work at a low level, all you would really need to do is make malloc and free smarter, because "memory unsafe" code is basically a case of double free in 99% of the time, as other things are either covered for by existing things like stack canaries, non executable memory sections, retpolines that protect against ROP chains, and so on. So just making sure that when free is called, it actually has permission to deallocate the chunk is all you need. You can do some of that statically, but to fully cover it, you would have some more arguments to the malloc and free call that establish permissions.
As an aside, the latter is why Rust doesn't really have a future. Its just popular enough these days to where its seen as taboo to hate on it because "memory safety", but the reality is, most of the existing well written C/C++ code is memory safe through very basic structuring of programs, and without the cumbersome syntax of Rust borrow checking that allows dev to happen much faster, considering any large codebase in Rust has a shitload of unsafes scattered about due to need to interact with posix libraries.
See https://safecpp.org/
The committee basically turned it down.