Old Mozilla / Firefox with XUL / XPCOM[1] was probably about as close as you get. If you were willing to use the low-level interfaces, you could change just about anything as far as I can tell. But that led to security issues and issues with maintaining comparability when there was a need to change those low-level interfaces. So Firefox largely moved away from XPCOM and blocked extensions from calling XPCOM interfaces.
That said, AFAIK Pale Moon still allows use of XPCOM. And I think Seamonkey may be the closest remaining browser to the Mozilla of old. I believe you can still use all of the XUL/XPCOM stuff there, but don't quote me on that.
Interesting stuff - none of these seem to have that Smalltalk-like visual IDE bolted right into them.
Given how big and important a platform the modern web is - I really am starting to think - something like this should exist - where I have a neat little IDE bundled right within the browser using which I can evolve the browser...
Why not go the other way around? For example, with Pharo, there is a WebBrowser project which could be further refined to do more of what you are proposing.
Old Mozilla / Firefox with XUL / XPCOM[1] was probably about as close as you get. If you were willing to use the low-level interfaces, you could change just about anything as far as I can tell. But that led to security issues and issues with maintaining comparability when there was a need to change those low-level interfaces. So Firefox largely moved away from XPCOM and blocked extensions from calling XPCOM interfaces.
That said, AFAIK Pale Moon still allows use of XPCOM. And I think Seamonkey may be the closest remaining browser to the Mozilla of old. I believe you can still use all of the XUL/XPCOM stuff there, but don't quote me on that.
[1]: https://en.wikipedia.org/wiki/XPCOM
[2]: https://en.wikipedia.org/wiki/SeaMonkey
Interesting stuff - none of these seem to have that Smalltalk-like visual IDE bolted right into them.
Given how big and important a platform the modern web is - I really am starting to think - something like this should exist - where I have a neat little IDE bundled right within the browser using which I can evolve the browser...
Not really 'visual', but https://nyxt-browser.com maybe?
Otherwise the less extreme https://luakit.github.io / https://github.com/luakit/luakit comes to mind. It's not dead yet.
Why not go the other way around? For example, with Pharo, there is a WebBrowser project which could be further refined to do more of what you are proposing.
Smalltalk is too old. I'd rather program in Python/Golang - or worst case JS these days.
Nyxt. Also, google is your friend.
I've seen this already - but Lisp seems a bit limiting. I'd have loved to use Py/Go or JS in the least - something more upto date.