Python Help

[swill]

I know there are some seasoned Python developers on this board, so I figured I would bounce this off you guys before I dig in too deep.  I am also a seasoned Python developer, but I find a conversation with different points of view to be very helpful when solving difficult problems.

So here we go...  I have been building a software to automate the drawing of custom plates and cases.  If you have not already seen my thread, you can check it out here: https://geekhack.org/index.php?topic=65189.0;topicseen

Here is a little preview video of where I am currently at so you can get an idea of what I am doing without having to read that whole thread.

OK, so on to my problem and the discussion...

I am using FreeCAD as my cad engine, because well, I didn't know any better and could not find a better solution in my searches.  I am using cadquery on top of it to simplify some of the concepts.  I was pretty comfortable with the scripting for this when I was working on my desktop and testing my drawings through the python interpreter in the FreeCAD UI.  It was only after I had basically had everything built and I then moved my code to my server that I started to see the problems...

FreeCAD was not built with a Pythonic standalone library mindset.  It was built more as an application runtime environment.  Thats all well and good, but that also means that by importing FreeCAD and interacting with it, you are creating global variables and setting module level variables which are global to the entire application.  This is fine if you are a single user who is actually interfacing with the UI, but as soon as you remove xorg all together and want to run many different distinct instances of FreeCAD all with their own distinct configurations, things start to get messy.  Some stuff just does not work without the UI because the developers were morons and tied the functionality directly to the existence of the UI and the global variables they 'expect' to be present at runtime.  These are things I really care about too, like SVG, DXF and DWG exports.    (sad panda)...

Brief interlude:  If you have experience with a scriptable CAD software which handles multiple distinct drawings elegantly in different parallel processes, please feel free to suggest it.  I am not against rewriting my entire core if there is a better core I could be working with...

Now that you have some backstory, here is what I am trying to do.  My UI can have many users each wanting to draw their own cad.  When they submit a request to draw a cad to the server, I need to setup different isolated 'environment' to import my FreeCAD instance into for execution so I can have multiple instances of FreeCAD instantiated without them sharing the same global session.  I am using Tornado as my backend server platform, so I have a framework in place to be able to run different coroutines or async callbacks or the like to do some sort of parallelism.  I was thinking ahead enough to get that in place right away. 

So the question really comes down to, how do I setup distinct and isolated runtimes for my FreeCAD workloads in an elegant way?

Things I am considering or looking into.  I am sure some of these will probably work and others probably won't.  Instead of me thrashing through all these to find a good solution, I want some advice...

• Use subprocess and run the script as its own OS process.  This will probably work, but it really limits my ability to interact with the process.  This is not a huge deal, but it is a bit annoying...
• I recently learned about process_isolation which looks like it could be promising.  I need to review it more.
• I have direct access to coroutines because I am using Tornado, but I have not worked with coroutines very much, so I don't actually know if they still use the same global namespace.  If they do, that screws me a bit.  If they don't this may be the most natural way to implement this given my choice of web frameworks.  I am not sure how generators handle imports, so I need to do more research on this one.
• I am considering setting myself up for network distribution of the cad drawing workloads right away.  It is more work at this point, but is probably what I am going to have to do long term anyway, so I am considering doing the research now.  I was considering just using LXC (or LXD) and getting started by installing a container system on my server for now and then spinning up a new container and offloading the workload to that container on each request.  If I do something like this I will be able to setup clusters of worker machines to draw the cad files independent of my actual website server.  This really only makes sense if this tool takes off and I have a lot of traffic on it (like 5-10 plates being drawn at any given time).  I am not sure that the keyboard community can sustain this much demand.
• As the complete opposite of the previous idea, I also considered implementing a queue and only letting one plate draw at a time.  Each plate takes about 30-40 seconds to draw, so people would only get queued if they request to draw while another cad is being drawn.  This is not usually a realistic solution to this type of problem, but other than the first day or two that I make this public, I don't expect there to be a huge amount of traffic and work being done by the server.  I doubt my queue would ever get bigger than 1 or 2 people in it, so this is not entirely unrealistic.  It kind of goes against all my principles as a cloud software developer, but I have to be realistic about what my expected use case is.

I think that is enough rambling by me at this point.  Please offer other up other solutions I have not mentioned if you have other ideas.  Please help me understand the shortcomings or the benefits to any of the solutions I have proposed if you have insight. 

Feel free to join the conversation even if you don't have a solution in mind but feel that your commentary might drive more discussion or things to research.  My goal for this thread is to try to isolate one or two different potential solutions which we feel is the most realistic and I will then research them and make a decision.

Thanks for your time...  Hacker Love... 

[suicidal_orange]

Interesting stuff, nothing I have any experience with so can't answer any of your questions but my first thought was would it be possible to duplicate the binary to enable multiple instances to run?  As you say, you won't have thousands of users so the server should be able to cope.

The other option is that being open source maybe it would be possible to remove all the UI related code and then fix their bad variable design?   It might be a huge program making this a stupid idea but I'm going at it completely blind.

[vvp]

Why do you use FreeCAD?
OpenCASCADE itself can do exports to step, dxf, vrml.
Isn't it enough?
Maybe you can use only OpenCASCADE.

Bothering with scalability looks like a premature optimization. My guess is that after a few days the queue will be mostly empty.

[swill]

Interesting stuff, nothing I have any experience with so can't answer any of your questions but my first thought was would it be possible to duplicate the binary to enable multiple instances to run?  As you say, you won't have thousands of users so the server should be able to cope.

The other option is that being open source maybe it would be possible to remove all the UI related code and then fix their bad variable design?   It might be a huge program making this a stupid idea but I'm going at it completely blind.

Duplicating the binary is an interesting idea.  Unfortunately in Python it is not quite that easy.  Basically, when you install a module like FreeCAD it is installed at a global Python interpreter level, so you can't use the same version of Python with two different clones of the same module.  I say that in general, but that also is not actually true.  There is the concept of a Virtual Environment in Python which are designed pretty much specifically for this sort of thing.  They allow you to create an entire runtime environment that is isolated from any other runtime environment and all of the libraries that you are install in that environment is local to that environment.  The install process for FreeCAD is pretty complex, but this could work.  I have never tried (and I am not sure if it is possible) to have a parent runtime which dispatches jobs to different isolated virtual environments.  If that is possible, that could be an interesting approach.  I think if I was going to go that route I would probably consider actual containers first since I can cluster containers over multiple machines where as virtual environments are only going to be locally available.

Regarding patching FreeCAD.  I think that is more trouble than it is worth.  I don't want to take on more than I have to with this project since it is already bigger than I was expecting and is growing as the community starts to get involved.  I started with a dream and I am realizing it, but the collective community has bigger dreams than I do.    I am trying to focus on an elegant sufficiency of features implemented for the tool to be useful and make sure that I put the groundwork in solid so I can grow it slowly as I have time to work on it...

[swill]

Why do you use FreeCAD?
OpenCASCADE itself can do exports to step, dxf, vrml.
Isn't it enough?
Maybe you can use only OpenCASCADE.

Bothering with scalability looks like a premature optimization. My guess is that after a few days the queue will be mostly empty.

I have been considering it, but there is a lot more documentation on FreeCAD than OpenCASCADE.  To be honest I had never worked with CAD prior to this project and I was trying to get my feet wet with a system that was widely supported by a community of users so I would have resources if/when I got stuck.  On revisiting the problem I have considered this, but I am not convinced that I will not run into the same types of issues with OpenCASCADE.  It also seems to be closely tied with UI elements and requires many UI components from what I can tell.

If I was going to replace the core, I would ideally find a library that does not require any UI elements at all to take a script of input and generate and export file, bypassing the 'drawing' operations entirely.  I have not found any CAD solutions that allow that since pretty much every CAD program seems to assume that you will want to draw the elements in a UI before you would want to export them.

I will only switch cores if I am pretty confident that I will not run into the same issues with the new core.    Thank you for the suggestion.  I think it is totally a valid suggestion which I am entertaining, but I need a better understanding of what the limitations of OpenCASCADE is before I really consider that approach...

[vvp]

OK, I did not know that OpenCASCADE is tied to GUI elements too. I thought it is not since FreeCAD itself does not use it for drawing. Coin3D is used for that. Maybe you can ask on FreeCAD IRC channel whether OpenCASCADE does actually require global variables in its client.

I cannot really help you since my experience with Python is limited. I do mostly C, C++, and Haskell. Anyway I would not bother too much with scalability since I doubt geekhack can proved enough users. If most of the code is already written then I would just separate FreeCAD into a separate OS process (or more of them) and queue requests to that process (or pool of them) through whatever IPC Pyton supports best. Not very efficient but it should do for the limited user base this will have.

[swill]

OK, I did not know that OpenCASCADE is tied to GUI elements too. I thought it is not since FreeCAD itself does not use it for drawing. Coin3D is used for that. Maybe you can ask on FreeCAD IRC channel whether OpenCASCADE does actually require global variables in its client.

I cannot really help you since my experience with Python is limited. I do mostly C, C++, and Haskell. Anyway I would not bother too much with scalability since I doubt geekhack can proved enough users. If most of the code is already written then I would just separate FreeCAD into a separate OS process (or more of them) and queue requests to that process (or pool of them) through whatever IPC Pyton supports best. Not very efficient but it should do for the limited user base this will have.

Ya, this is probably the shortest path to having a working solution that I can make public.  At least this way it will function even if it is a little slower.  I can monitor the queue length and if there is always multiple items in the queue then I can work on a different solution.  At least this method can be implemented relatively easily without having to re-architect everything...    Thanks for the feedback. 

Edit: I may be wrong about OpenCASCADE requiring UI elements.  I went back and looked and it was actually the generation of the documentation for OpenCASCADE which required UI elements.     I will look at that again.  I have actually found better documentation recently on OpenCASCADE than I had previously, so I think it is probably the best option if I do rewrite the core...

[Dihedral]

This is a nasty problem, which I can't help with, but good luck with finding a solution!

[swill]

I just wanted to follow up to let you guys know how I solved the problem.  This is going to be the 30,000ft description and I will be adding a more detailed description in my plate building thread later tonight: https://geekhack.org/index.php?topic=65189.0

I ended up solving this problem by using LXC.  I have a template LXC container setup that has everything needed to build the cad (its basically a clone of my normal server since the same code works on both).  Then when a request to draw a CAD comes in, it will clone that LXC template and start a new container.  It will then pass off the work to that container and when the work is finished, it will then return to the original server thread with the details and the result will get returned to the user.  I am now saving all of the drawings in AWS S3, so it just has to return the metadata back to the client and the client fetches all the extra details it needs directly from S3.

I have tested and I am able to handle two different requests at the same time right now, so I know this solution is working.  Thanks for the suggestions... 

[Dihedral]

I just wanted to follow up to let you guys know how I solved the problem.  This is going to be the 30,000ft description and I will be adding a more detailed description in my plate building thread later tonight: https://geekhack.org/index.php?topic=65189.0

I ended up solving this problem by using LXC.  I have a template LXC container setup that has everything needed to build the cad (its basically a clone of my normal server since the same code works on both).  Then when a request to draw a CAD comes in, it will clone that LXC template and start a new container.  It will then pass off the work to that container and when the work is finished, it will then return to the original server thread with the details and the result will get returned to the user.  I am now saving all of the drawings in AWS S3, so it just has to return the metadata back to the client and the client fetches all the extra details it needs directly from S3.

I have tested and I am able to handle two different requests at the same time right now, so I know this solution is working.  Thanks for the suggestions... 

Doing all that in Python would scare me

[swill]

I just wanted to follow up to let you guys know how I solved the problem.  This is going to be the 30,000ft description and I will be adding a more detailed description in my plate building thread later tonight: https://geekhack.org/index.php?topic=65189.0

I ended up solving this problem by using LXC.  I have a template LXC container setup that has everything needed to build the cad (its basically a clone of my normal server since the same code works on both).  Then when a request to draw a CAD comes in, it will clone that LXC template and start a new container.  It will then pass off the work to that container and when the work is finished, it will then return to the original server thread with the details and the result will get returned to the user.  I am now saving all of the drawings in AWS S3, so it just has to return the metadata back to the client and the client fetches all the extra details it needs directly from S3.

I have tested and I am able to handle two different requests at the same time right now, so I know this solution is working.  Thanks for the suggestions... 

Doing all that in Python would scare me

Haha, the only reason this does not scare me is BECAUSE its in Python.    I love me some Python.  I write code in many languages, but if I am going to just sit down and do something for fun, it will be in Python 85% of the time (the other 15% would probably be GoLang).  Python is just such a beautiful language to me.  It is beautifully simple but can do anything I need...

[Dihedral]

I just wanted to follow up to let you guys know how I solved the problem.  This is going to be the 30,000ft description and I will be adding a more detailed description in my plate building thread later tonight: https://geekhack.org/index.php?topic=65189.0

I ended up solving this problem by using LXC.  I have a template LXC container setup that has everything needed to build the cad (its basically a clone of my normal server since the same code works on both).  Then when a request to draw a CAD comes in, it will clone that LXC template and start a new container.  It will then pass off the work to that container and when the work is finished, it will then return to the original server thread with the details and the result will get returned to the user.  I am now saving all of the drawings in AWS S3, so it just has to return the metadata back to the client and the client fetches all the extra details it needs directly from S3.

I have tested and I am able to handle two different requests at the same time right now, so I know this solution is working.  Thanks for the suggestions... 

Doing all that in Python would scare me

Haha, the only reason this does not scare me is BECAUSE its in Python.    I love me some Python.  I write code in many languages, but if I am going to just sit down and do something for fun, it will be in Python 85% of the time (the other 15% would probably be GoLang).  Python is just such a beautiful language to me.  It is beautifully simple but can do anything I need...

Do you use an IDE or just a text editor / IDLE

[swill]

I just wanted to follow up to let you guys know how I solved the problem.  This is going to be the 30,000ft description and I will be adding a more detailed description in my plate building thread later tonight: https://geekhack.org/index.php?topic=65189.0

I ended up solving this problem by using LXC.  I have a template LXC container setup that has everything needed to build the cad (its basically a clone of my normal server since the same code works on both).  Then when a request to draw a CAD comes in, it will clone that LXC template and start a new container.  It will then pass off the work to that container and when the work is finished, it will then return to the original server thread with the details and the result will get returned to the user.  I am now saving all of the drawings in AWS S3, so it just has to return the metadata back to the client and the client fetches all the extra details it needs directly from S3.

I have tested and I am able to handle two different requests at the same time right now, so I know this solution is working.  Thanks for the suggestions... 

Doing all that in Python would scare me

Haha, the only reason this does not scare me is BECAUSE its in Python.    I love me some Python.  I write code in many languages, but if I am going to just sit down and do something for fun, it will be in Python 85% of the time (the other 15% would probably be GoLang).  Python is just such a beautiful language to me.  It is beautifully simple but can do anything I need...

Do you use an IDE or just a text editor / IDLE

An IDE?  What do you take me for, a n00b? 

I have a bit of a personal vendetta against IDEs. I find that they lead to sloppy coding practices, so I don't really use them for anything (even for Java).

I use sublime text 3 as my editor. It has excellent text manipulation features and that's really all I care about (well other than a solarized theme).

[Dihedral]

I just wanted to follow up to let you guys know how I solved the problem.  This is going to be the 30,000ft description and I will be adding a more detailed description in my plate building thread later tonight: https://geekhack.org/index.php?topic=65189.0

I ended up solving this problem by using LXC.  I have a template LXC container setup that has everything needed to build the cad (its basically a clone of my normal server since the same code works on both).  Then when a request to draw a CAD comes in, it will clone that LXC template and start a new container.  It will then pass off the work to that container and when the work is finished, it will then return to the original server thread with the details and the result will get returned to the user.  I am now saving all of the drawings in AWS S3, so it just has to return the metadata back to the client and the client fetches all the extra details it needs directly from S3.

I have tested and I am able to handle two different requests at the same time right now, so I know this solution is working.  Thanks for the suggestions... 

Doing all that in Python would scare me

Haha, the only reason this does not scare me is BECAUSE its in Python.    I love me some Python.  I write code in many languages, but if I am going to just sit down and do something for fun, it will be in Python 85% of the time (the other 15% would probably be GoLang).  Python is just such a beautiful language to me.  It is beautifully simple but can do anything I need...

Do you use an IDE or just a text editor / IDLE

An IDE?  What do you take me for, a n00b? 

I have a bit of a personal vendetta against IDEs. I find that they lead to sloppy coding practices, so I don't really use them for anything (even for Java).

I use sublime text 3 as my editor. It has excellent text manipulation features and that's really all I care about (well other than a solarized theme).

I never really used an IDE for python or C, but for Java I couldn't live without it. I'm not hardcore and I accept it