I know there are some seasoned Python developers on this board, so I figured I would bounce this off you guys before I dig in too deep. I am also a seasoned Python developer, but I find a conversation with different points of view to be very helpful when solving difficult problems.
So here we go... I have been building a software to automate the drawing of custom plates and cases. If you have not already seen my thread, you can check it out here:
https://geekhack.org/index.php?topic=65189.0;topicseenHere is a little preview video of where I am currently at so you can get an idea of what I am doing without having to read that whole thread.
OK, so on to my problem and the discussion...
I am using
FreeCAD as my cad engine, because well, I didn't know any better and could not find a better solution in my searches. I am using
cadquery on top of it to simplify some of the concepts. I was pretty comfortable with the scripting for this when I was working on my desktop and testing my drawings through the python interpreter in the FreeCAD UI. It was only after I had basically had everything built and I then moved my code to my server that I started to see the problems...
FreeCAD was not built with a Pythonic standalone library mindset. It was built more as an application runtime environment. Thats all well and good, but that also means that by importing FreeCAD and interacting with it, you are creating global variables and setting module level variables which are global to the entire application. This is fine if you are a single user who is actually interfacing with the UI, but as soon as you remove xorg all together and want to run many different distinct instances of FreeCAD all with their own distinct configurations, things start to get messy. Some stuff just does not work without the UI because the developers were morons and tied the functionality directly to the existence of the UI and the global variables they 'expect' to be present at runtime. These are things I really care about too, like SVG, DXF and DWG exports.
(sad panda)...
Brief interlude: If you have experience with a scriptable CAD software which handles multiple distinct drawings elegantly in different parallel processes, please feel free to suggest it. I am not against rewriting my entire core if there is a better core I could be working with...
Now that you have some backstory, here is what I am trying to do. My UI can have many users each wanting to draw their own cad. When they submit a request to draw a cad to the server, I need to setup different isolated 'environment' to import my FreeCAD instance into for execution so I can have multiple instances of FreeCAD instantiated without them sharing the same global session. I am using
Tornado as my backend server platform, so I have a framework in place to be able to run different coroutines or async callbacks or the like to do some sort of parallelism. I was thinking ahead enough to get that in place right away.
So the question really comes down to, how do I setup distinct and isolated runtimes for my FreeCAD workloads in an elegant way?
Things I am considering or looking into. I am sure some of these will probably work and others probably won't. Instead of me thrashing through all these to find a good solution, I want some advice...
- Use subprocess and run the script as its own OS process. This will probably work, but it really limits my ability to interact with the process. This is not a huge deal, but it is a bit annoying...
- I recently learned about process_isolation which looks like it could be promising. I need to review it more.
- I have direct access to coroutines because I am using Tornado, but I have not worked with coroutines very much, so I don't actually know if they still use the same global namespace. If they do, that screws me a bit. If they don't this may be the most natural way to implement this given my choice of web frameworks. I am not sure how generators handle imports, so I need to do more research on this one.
- I am considering setting myself up for network distribution of the cad drawing workloads right away. It is more work at this point, but is probably what I am going to have to do long term anyway, so I am considering doing the research now. I was considering just using LXC (or LXD) and getting started by installing a container system on my server for now and then spinning up a new container and offloading the workload to that container on each request. If I do something like this I will be able to setup clusters of worker machines to draw the cad files independent of my actual website server. This really only makes sense if this tool takes off and I have a lot of traffic on it (like 5-10 plates being drawn at any given time). I am not sure that the keyboard community can sustain this much demand.
- As the complete opposite of the previous idea, I also considered implementing a queue and only letting one plate draw at a time. Each plate takes about 30-40 seconds to draw, so people would only get queued if they request to draw while another cad is being drawn. This is not usually a realistic solution to this type of problem, but other than the first day or two that I make this public, I don't expect there to be a huge amount of traffic and work being done by the server. I doubt my queue would ever get bigger than 1 or 2 people in it, so this is not entirely unrealistic. It kind of goes against all my principles as a cloud software developer, but I have to be realistic about what my expected use case is.
I think that is enough rambling by me at this point. Please offer other up other solutions I have not mentioned if you have other ideas. Please help me understand the shortcomings or the benefits to any of the solutions I have proposed if you have insight.
Feel free to join the conversation even if you don't have a solution in mind but feel that your commentary might drive more discussion or things to research. My goal for this thread is to try to isolate one or two different potential solutions which we feel is the most realistic and I will then research them and make a decision.
Thanks for your time... Hacker Love...