Python:Isolation

lastmod 2022-11-25created 2022-03-1718 edits

My travelogue of fumbling around with ways to isolate Python...

The order of my fumbling

Isolated from c with cpython.
Switched to using the exec built-in function in python.
Installed venv and ran it with subprocess.

How to communicate with the isolated Python

Communicate with grpc.
Communicate with grpc+json. -> Large data transfers turned out to be slow.
Communicate with grpc+json+sharedmemory. -> Bytes data is encoded for numpy serialization, but json doesn't support bytes data, which turned out to be a problem.
Communicate with grpc+msgpack+sharedmemory.

Fumbling toward complete isolation and communication

At first, ran it with asyncio.subprocess of the original python/recc and communicated directly inside python: the problem is that remote environments are not supported.
Original python/recc <-- how to communicate? --> isolated venv subprocess: the problem is that the isolated venv has no communication module.
Original python/recc <-- grpc --> isolated venv subprocess: the isolated venv needs grpc.
Original python/recc <-- grpc --> in the original python subprocess, add only venv's site-packages directory to sys.path: the problem is that the original python's site-packages is shared.
Original python/recc <-- grpc --> in the isolated venv subprocess, add only the original python's site-packages directory to sys.path: the problem is that it doesn't work properly.
Original python/recc <-- grpc --> in the original python subprocess, context-changed the variables of the sys package following how venv works: the problem is that async/await statements run asynchronously, so a context change affects the global sys package and it does not work properly.
Original python/recc <-- grpc --> installed all recc-related dependency packages in the isolated venv subprocess.
- Downloaded the dependency files in advance with pip download --dest /download/path -r requirements.txt
- and installed them offline with pip install --no-index -f /download/path -r requirements.txt.
- Assuming offline use, recc copies all files already installed in the original site-packages into {venv-site-packages}/__recc_local__,
- and adds the installed path to the {venv-site-packages}/recc.pth file so that the recc module is recognized automatically without extra work. (However, since it wasn't installed with pip, it doesn't appear in the pip list command output.)
  - See Python:site for details on the *.pth rules.
- Solved with this..