As a mechanical engineer, I feel the part of my job is safe from AI for the time being. I don't think quality training data for good mechanical design exists.
3D CAD is only part of good design. To a tinker-er that is 3D printing simple parts, an STL is fine. But most parts that matter require far more design consideration and detail than simply the geometry data that an STL (or other 3D file) provides.
The majority of parts are accompanied with a drawing, and that is where the real design actually is found: Tolerances, GD&T, materials, processing notes...
Even then, most of the calculations and considerations to build the model and drawing are not explicit in the design documents: Nothing about a drawing of a stainless steel part tells you WHY it must be a stainless steel part. I don't think there is a large set of well documented designs out there to act as training data for an AI system to design an assembly beyond basic 3D parts.
The authors identify this gap, but it's a fundamental problem with the wholesale move to AI in mechanical design.
Agreed. At the end of the day, manufactured parts are driven by constraints outside of the CAD environment so analyzing 3D data as the foundation of an AI system strikes me as attacking the problem from the wrong direction. i.e. Simple optimization of a part for injection molding can take it from requiring a bunch of side actions and collapsing cores to a simple 2 sided mold and save hundreds of thousands of dollars in tooling. None of that is obvious from 3D data alone.
That said, I am excited for AI assisted CAD tools. Things like creating and applying global variables to an existing part, complex assembly analysis for part reduction or just making a starting base part can be incredibly tedious and are low hanging fruit for improving CAD workflows with AI imo.
a lot of the why is encoded elsewhere in mechanical engineering at least - the tables, the formulas, textbook problems, engineering reports.
one of the challenges to making a good data set might be around bad designs and why they failed. if we get to a mechanical agent, its going to need to understand that brass was a mistake and redesign a part as steel and change the design for the new contraints
unlike code, that kind of train of experiment i think will be a lot more expensive to make, since you might actually want to create those parts along the way and not just pretend
Amusingly, I tried. Your job is safe. ;-) I'm a scientist. I tried using a couple of free 3d modeling tools that are programmable in Python, using Claude within VS Code to design a part according to my prompts. It was a simple plate with some holes for electrical connectors, switches, etc. Of course it let me make some noobie design mistakes, such as making the hole exactly the same diameter as the thing that went into it, plus or minus of course. And I would have had to really scrutinize the Python code to notice that I got one of the diameters wrong altogether.
Just judging from this experience, the effort would rise exponentially with the complexity of the part, not to mention assemblies. The designers earn their keep. I get really bad eyestrain with any task that requires staring at the screen while doing fine mouse work, so I just can't use CAD. On the other hand, I can code all day because I'm not closely focused on the screen when I'm typing.
I spend a lot of time in eplan, 3d cad programs, and cam programs (when working on machines, furniture, and cnc of furniture or machines respectively :P)
However, the vast majority of time what i'm searching grabcad or friends it's for specific part numbers or parts.
I cannot remember the last time i wanted to try to find some object by generic description. Even as a tool to find possible parts that might fit my task, i don't see how it would make sense, because the stuff that matters is mostly not visual.
IE let's say i'm trying to find ballscrew nuts that might be a replacement for one i can't get anymore.
I'd want to search for ballscrew nuts that have a specific dimension, which maybe it can do (doesn't look like it so far), but the parts that matter are things like "is it preloaded", etc, which wouldn't be part of the description it generates (because it's often not even visualized in the cad model).
This is for mechanical stuff that might be interchangeable. Lots of cad models are of electronics and such that are very much not.
Even when i'm doing 3d printing, most of the time i'd search for the part i'm looking for, not a generic description of what might look like - ie i'm searching for dividers for sidiocrate crates.
Giving me thousands of possible things that might be a divider would be pretty useless.
To the degree i search for generic descriptions, people already provide them, and it's not obvious this is a meaningful friction point for them (IE that being able to generate the labels automatically is really valuable)
So while i think this is overall cool, i struggle to think of a truly practical use.
We rendered the one million part ABC dataset from Deep Geometry, and open-sourced the data. We also built a fun demo with the following pipeline: CAD > render > caption > embed.
Yeah I think the embedding are describing what can be seen from a picture of the model not what it is or what it is used for.
search some things work like "Fan" but others don't so you can search for
"plate with 5 holes" but not for specific engine part cover.
Several searches I tried produced terrible results with no objects matching the description on the first page:
M8x30 SHCS
Lathe chuck
19" rack mount blank plate
The thumbnail descriptions are dominated by boilerplate, such as "The depicted object consists of a wall-mounted shelf unit featuring a rectangular back"
The long descriptions are full of irrelevant fluff and don't answer basic questions like what size it is. For example, on a screw:
> The model features a hexagonal socket head cap screw, characterized by its cylindrical shaft and a larger, flared hexagonal head. The head's interior is designed to accommodate a hex key, allowing for easy installation and removal. The shaft tapers slightly towards the tip, providing an elongated profile. The smooth surface indicates precision engineering, suitable for applications requiring a strong and secure fastening. The overall design emphasizes functionality while ensuring compatibility with various tools and mechanical setups.
A broad search "vise" returns a long list of identical, very eccentric, vises the likes of which I've never seen before.
Neat, but also hilarious! Searching for "mug" gives results where the first item listed (ABC-00008297) is a mug model with a hole not only in the top to pour in your drink, but also in the side and bottom (just in case you wanted more access to your liquid).
Not the first one... but the "pair of interconnected mugs" that is described as "emphasizes connectivity and collaboration, suitable for serving beverages in a shared or communal setting" is pretty amazing too. I never knew I was missing out on communal mug holding.
"box with a hole in it" resulted in a number of boxes and most of these did obviously have holes in them. some did not appear to, but being that they were largely enclosures for some sort of device (as opposed to storage or transport) I'll assume there was some hidden fastener clearance holes or something.
I really only wanted one hole in my box though, so I adjusted the query to "box with a single hole in it". the results looked indentical. except for one that stood out. I would link to the particular model, but there was no way to do that. this model appears to be a rectangular bathroom basin, on its side. I'd describe it as perhaps a ~currently fashionable porcelain design, but it could be a concrete 'getting shit done' sink, or a model from The Sims (the first one). so box-like perhaps, but not many people would describe it as a box. I guess my search continues (elsewhere)...
(actually interesting bit about natural language: I know that a box with two (or more) holes in it has a single hole in it, but most English natural language parsers (humans) will notice that specifying 'single' would be redundant if I wanted any number more than zero, so it's extremely unlikely that I was looking for a multi-hole box.)
where did you steal the models from, by the way? just curious. the original context in which they were found would actually be helpful if someone was for some reason trying to actually use this as a tool. [ed: saw the OP's comment down the page -- you can include a comment with the submission IIRC]
also if you don't have the 3D model spinning incessantly, having the page open won't be obnoxious and it won't (have to) waste power
The approach could be helpful for searching though large 3D model libraries like GrabCAD for some visual placeholder part by just describing it.
The generality of the part descriptions made me chuckle.
> A bevel gear with a circular base and a series of angular, tapered teeth extending radially outward. The teeth are uniformly distributed around the circumference, allowing for meshing at an angle with another gear. The gear's face includes a set of holes, varying in size and symmetrically arranged around the central bore, likely for weight reduction or mounting purposes. The central opening likely acts as an axle or shaft attachment point. The design facilitates the transmission of rotational motion between intersecting shafts, typically at a 90-degree angle.
Door knob seems to return a bunch of chess pieces and analog sticks, I did get a few doors and a few knobs, but nothing I recognized as a door knob. I didn't spend a super long time looking though so that might have been the problem there. Also suggestion, giving the ability to link to specific models might be useful (I tried a few things on mobile, url doesn't change and searching for a specific model ID doesn't return the model).
I tried Google/Claude etc. But none worked. As per Claude, the technical name for that is Pillow Block Bearing/Shaft Coupling Block/Flange Mount Bracket. Funny thing is, your app didn't return any good result when I search with any of those terms.
After reading your blog post, I searched for "block with 2 holes". And lo and behold, it returned ABC-00162357!
Couple of suggestions:
1) Have a permanent link for each model
2) Show related models when a model is clicked
3) and lastly, show models based on an image
edit: Search for "mounting block" returned ABC-00180735 which is exactly what I was looking for. Thank you so much for making this!
Articulated dragon gave me some dragon shaped sillouetes of ninja throwing stars. I'm looking for those 3D interlocking dragons that seem to be all the rage with the kids lately.
This isn't meant to be a commercially useful search engine- just a demonstration. You'll only be able to search for terms that the VLM could directly discern.
From the blog post:
Our search demo proves that it works quite well. As anticipated, text search works well, returning sensible results for even irregular or poorly formed queries. It’s worth mentioning that this is very different from 3D part libraries like Thingiverse or GrabCAD. Search in those repositories requires users to tag or annotate parts with a description, the text of which is used in search. Our system takes only an unnamed part as input, requiring no additional labelling.
I like how the models have varying degrees of accuracy eg. a Raspberry Pi 4B some are simple volumetric, others seem to have every surface mount component which is crazy... wonder if that was 3D scanned.
More likely the detailed ones came out of ECAD systems that often include 3d models in their component libraries so you can automatically visualize/model the finished product while designing a board, and integrate with physical CAD for designing enclosures and other mechanical parts.
I tried "DHO804" (a popular oscilloscope that has printable accessories), got all screws.
Ok too niche, except that's exactly the use-case as I see it so if that's too niche then what good is it? Whatever call it pre-alpha poc and move on...
Tried "grommet", got all finger rings. Closer but not close enough to be useful. It wasn't a mix of ringular-shaped objects including grommets, and grommets aren't only round either. None of the rings were even slightly grommet shaped, purely tori and belts, some with add-ons and cut-outs.
Perhaps it needs a couple orders of magnitude more input samples before it becomes useful. And by "useful" I do mean even just as a proof of concept, because I don't see any concept proven here.
There's a pretty big bias for mechanical engineering components in the dataset- very few organic forms. It's one of the limitations we call out in the dataset card.
There are a few though! Try "dog" or "cookie cutter" for example.
I've just now done exactly this. Model something (not an apple) not because I want to print it, but because I need to import it into CAD to check clearances etc. around it. I can abolsutely imagine modelling different sized apples if I was building an apple sorting machine.
I do this all the time when working with stuff that is either perishable or expensive. I just print a 3D model of it and use that as a stand in for the real thing until I have it working.
Neat.
As a mechanical engineer, I feel the part of my job is safe from AI for the time being. I don't think quality training data for good mechanical design exists.
3D CAD is only part of good design. To a tinker-er that is 3D printing simple parts, an STL is fine. But most parts that matter require far more design consideration and detail than simply the geometry data that an STL (or other 3D file) provides.
The majority of parts are accompanied with a drawing, and that is where the real design actually is found: Tolerances, GD&T, materials, processing notes...
Even then, most of the calculations and considerations to build the model and drawing are not explicit in the design documents: Nothing about a drawing of a stainless steel part tells you WHY it must be a stainless steel part. I don't think there is a large set of well documented designs out there to act as training data for an AI system to design an assembly beyond basic 3D parts.
The authors identify this gap, but it's a fundamental problem with the wholesale move to AI in mechanical design.
Agreed. At the end of the day, manufactured parts are driven by constraints outside of the CAD environment so analyzing 3D data as the foundation of an AI system strikes me as attacking the problem from the wrong direction. i.e. Simple optimization of a part for injection molding can take it from requiring a bunch of side actions and collapsing cores to a simple 2 sided mold and save hundreds of thousands of dollars in tooling. None of that is obvious from 3D data alone.
That said, I am excited for AI assisted CAD tools. Things like creating and applying global variables to an existing part, complex assembly analysis for part reduction or just making a starting base part can be incredibly tedious and are low hanging fruit for improving CAD workflows with AI imo.
a lot of the why is encoded elsewhere in mechanical engineering at least - the tables, the formulas, textbook problems, engineering reports.
one of the challenges to making a good data set might be around bad designs and why they failed. if we get to a mechanical agent, its going to need to understand that brass was a mistake and redesign a part as steel and change the design for the new contraints
unlike code, that kind of train of experiment i think will be a lot more expensive to make, since you might actually want to create those parts along the way and not just pretend
Amusingly, I tried. Your job is safe. ;-) I'm a scientist. I tried using a couple of free 3d modeling tools that are programmable in Python, using Claude within VS Code to design a part according to my prompts. It was a simple plate with some holes for electrical connectors, switches, etc. Of course it let me make some noobie design mistakes, such as making the hole exactly the same diameter as the thing that went into it, plus or minus of course. And I would have had to really scrutinize the Python code to notice that I got one of the diameters wrong altogether.
Just judging from this experience, the effort would rise exponentially with the complexity of the part, not to mention assemblies. The designers earn their keep. I get really bad eyestrain with any task that requires staring at the screen while doing fine mouse work, so I just can't use CAD. On the other hand, I can code all day because I'm not closely focused on the screen when I'm typing.
I spend a lot of time in eplan, 3d cad programs, and cam programs (when working on machines, furniture, and cnc of furniture or machines respectively :P)
However, the vast majority of time what i'm searching grabcad or friends it's for specific part numbers or parts.
I cannot remember the last time i wanted to try to find some object by generic description. Even as a tool to find possible parts that might fit my task, i don't see how it would make sense, because the stuff that matters is mostly not visual.
IE let's say i'm trying to find ballscrew nuts that might be a replacement for one i can't get anymore.
I'd want to search for ballscrew nuts that have a specific dimension, which maybe it can do (doesn't look like it so far), but the parts that matter are things like "is it preloaded", etc, which wouldn't be part of the description it generates (because it's often not even visualized in the cad model).
This is for mechanical stuff that might be interchangeable. Lots of cad models are of electronics and such that are very much not.
Even when i'm doing 3d printing, most of the time i'd search for the part i'm looking for, not a generic description of what might look like - ie i'm searching for dividers for sidiocrate crates. Giving me thousands of possible things that might be a divider would be pretty useless.
To the degree i search for generic descriptions, people already provide them, and it's not obvious this is a meaningful friction point for them (IE that being able to generate the labels automatically is really valuable)
So while i think this is overall cool, i struggle to think of a truly practical use.
We rendered the one million part ABC dataset from Deep Geometry, and open-sourced the data. We also built a fun demo with the following pipeline: CAD > render > caption > embed.
Open-sourced dataset: https://huggingface.co/datasets/daveferbear/3d-model-images-...
Blog writeup: https://www.finalrev.com/blog/embedding-one-million-3d-model...
The search function doesn’t seem to work at all, it provides nonsensical results.
For example if I search “supercolumns” I get regular household furniture.
Yeah I think the embedding are describing what can be seen from a picture of the model not what it is or what it is used for. search some things work like "Fan" but others don't so you can search for "plate with 5 holes" but not for specific engine part cover.
Several searches I tried produced terrible results with no objects matching the description on the first page:
The thumbnail descriptions are dominated by boilerplate, such as "The depicted object consists of a wall-mounted shelf unit featuring a rectangular back"The long descriptions are full of irrelevant fluff and don't answer basic questions like what size it is. For example, on a screw:
> The model features a hexagonal socket head cap screw, characterized by its cylindrical shaft and a larger, flared hexagonal head. The head's interior is designed to accommodate a hex key, allowing for easy installation and removal. The shaft tapers slightly towards the tip, providing an elongated profile. The smooth surface indicates precision engineering, suitable for applications requiring a strong and secure fastening. The overall design emphasizes functionality while ensuring compatibility with various tools and mechanical setups.
A broad search "vise" returns a long list of identical, very eccentric, vises the likes of which I've never seen before.
Neat, but also hilarious! Searching for "mug" gives results where the first item listed (ABC-00008297) is a mug model with a hole not only in the top to pour in your drink, but also in the side and bottom (just in case you wanted more access to your liquid).
Not the first one... but the "pair of interconnected mugs" that is described as "emphasizes connectivity and collaboration, suitable for serving beverages in a shared or communal setting" is pretty amazing too. I never knew I was missing out on communal mug holding.
Yeah, this looks like something you might hang from a wire or string with a slot for a dowel. A+ for effort though, lol!
I think a real value of AI in Engineering will be when it can act as a reviewer.
Reviewing drawings for incompleteness, contradictory informations, deviations from standards, etc and making recommendations would be very valuable.
It doesn’t need to be perfect to be useful.
"box with a hole in it" resulted in a number of boxes and most of these did obviously have holes in them. some did not appear to, but being that they were largely enclosures for some sort of device (as opposed to storage or transport) I'll assume there was some hidden fastener clearance holes or something.
I really only wanted one hole in my box though, so I adjusted the query to "box with a single hole in it". the results looked indentical. except for one that stood out. I would link to the particular model, but there was no way to do that. this model appears to be a rectangular bathroom basin, on its side. I'd describe it as perhaps a ~currently fashionable porcelain design, but it could be a concrete 'getting shit done' sink, or a model from The Sims (the first one). so box-like perhaps, but not many people would describe it as a box. I guess my search continues (elsewhere)...
(actually interesting bit about natural language: I know that a box with two (or more) holes in it has a single hole in it, but most English natural language parsers (humans) will notice that specifying 'single' would be redundant if I wanted any number more than zero, so it's extremely unlikely that I was looking for a multi-hole box.)
where did you steal the models from, by the way? just curious. the original context in which they were found would actually be helpful if someone was for some reason trying to actually use this as a tool. [ed: saw the OP's comment down the page -- you can include a comment with the submission IIRC]
also if you don't have the 3D model spinning incessantly, having the page open won't be obnoxious and it won't (have to) waste power
The approach could be helpful for searching though large 3D model libraries like GrabCAD for some visual placeholder part by just describing it.
The generality of the part descriptions made me chuckle.
> A bevel gear with a circular base and a series of angular, tapered teeth extending radially outward. The teeth are uniformly distributed around the circumference, allowing for meshing at an angle with another gear. The gear's face includes a set of holes, varying in size and symmetrically arranged around the central bore, likely for weight reduction or mounting purposes. The central opening likely acts as an axle or shaft attachment point. The design facilitates the transmission of rotational motion between intersecting shafts, typically at a 90-degree angle.
Door knob seems to return a bunch of chess pieces and analog sticks, I did get a few doors and a few knobs, but nothing I recognized as a door knob. I didn't spend a super long time looking though so that might have been the problem there. Also suggestion, giving the ability to link to specific models might be useful (I tried a few things on mobile, url doesn't change and searching for a specific model ID doesn't return the model).
Thanks! In my search for a good STL for the following, your app gave me the closest model so far!
https://m.media-amazon.com/images/I/41hGjsBlrKL._AC_SL1000_....
I tried Google/Claude etc. But none worked. As per Claude, the technical name for that is Pillow Block Bearing/Shaft Coupling Block/Flange Mount Bracket. Funny thing is, your app didn't return any good result when I search with any of those terms.
After reading your blog post, I searched for "block with 2 holes". And lo and behold, it returned ABC-00162357!
Couple of suggestions: 1) Have a permanent link for each model 2) Show related models when a model is clicked 3) and lastly, show models based on an image
edit: Search for "mounting block" returned ABC-00180735 which is exactly what I was looking for. Thank you so much for making this!
Articulated dragon gave me some dragon shaped sillouetes of ninja throwing stars. I'm looking for those 3D interlocking dragons that seem to be all the rage with the kids lately.
Interesting.
My go-to for CAD files is usually https://grabcad.com/library
I searched this for "WAGO" and "XT90", so I guess not the same use case. Some hits for "Raspberry Pi", though.
This isn't meant to be a commercially useful search engine- just a demonstration. You'll only be able to search for terms that the VLM could directly discern.
From the blog post: Our search demo proves that it works quite well. As anticipated, text search works well, returning sensible results for even irregular or poorly formed queries. It’s worth mentioning that this is very different from 3D part libraries like Thingiverse or GrabCAD. Search in those repositories requires users to tag or annotate parts with a description, the text of which is used in search. Our system takes only an unnamed part as input, requiring no additional labelling.
I see, you did an AI demo of captioning and search over captures specifically for complex geometric objects.
I guess my interest was more piqued by the "CAD" part.
searched for 'bong', would not smoke out of any of the objects returned, 0/10
I like how the models have varying degrees of accuracy eg. a Raspberry Pi 4B some are simple volumetric, others seem to have every surface mount component which is crazy... wonder if that was 3D scanned.
More likely the detailed ones came out of ECAD systems that often include 3d models in their component libraries so you can automatically visualize/model the finished product while designing a board, and integrate with physical CAD for designing enclosures and other mechanical parts.
When I search for "duck shaped trampoline", I see mostly fidget spinners…
i tried “apples” and got lots of nuts-and-bolts models?
edit: looks like the data is trained from machinery parts. impressive regardless, but i’d add that to the lander
I tried "DHO804" (a popular oscilloscope that has printable accessories), got all screws.
Ok too niche, except that's exactly the use-case as I see it so if that's too niche then what good is it? Whatever call it pre-alpha poc and move on...
Tried "grommet", got all finger rings. Closer but not close enough to be useful. It wasn't a mix of ringular-shaped objects including grommets, and grommets aren't only round either. None of the rings were even slightly grommet shaped, purely tori and belts, some with add-ons and cut-outs.
Perhaps it needs a couple orders of magnitude more input samples before it becomes useful. And by "useful" I do mean even just as a proof of concept, because I don't see any concept proven here.
There's a pretty big bias for mechanical engineering components in the dataset- very few organic forms. It's one of the limitations we call out in the dataset card.
There are a few though! Try "dog" or "cookie cutter" for example.
It's CAD. Is there a legitimate reason to use that to engineer a dog? Doesn't make sense to me.
The initial ABC dataset is from public Onshape files -- clearly some people had a reason to design a dog model parametrically!
id guess for 3d printing.
not the best tool for the job of making something organically shaped, but maybe they also wanted to run some aerodynamics tests on it?
It's CAD. Doesn't make sense to engineer an apple...
Fruit picking is a thing. You may want to 3D print some everlasting apples for tests.
I've just now done exactly this. Model something (not an apple) not because I want to print it, but because I need to import it into CAD to check clearances etc. around it. I can abolsutely imagine modelling different sized apples if I was building an apple sorting machine.
I do this all the time when working with stuff that is either perishable or expensive. I just print a 3D model of it and use that as a stand in for the real thing until I have it working.
CAD is a 3D modeling tool.
its not the only one, and other tools are better suited for something like an apple. you can still post-process it to get it right sized for a printer
Oh come on, a nice 3D model of parametric apples in OpenSCAD would be so neat to have.
I think an apple would not be stored in a CAD format like IGES or STEP but rather in OBJ, USD, FBX etc.
I.e. it would not be in dataset because the use cases for 3D apples are outside of typical use cases where people resort to CAD software.
Try searching for "glasses" and you get a page full of blocks?
Sick project. Great work. Thanks for the HF dataset as well.
two searches gave me absolute ridiculous results: chair, laptop. Back to re-learning fusion for me :-)
Amazing!
I hope you make it a real site as I noticed usually these vercel apps go offline after few months.
I put in "dog bone" and it just returned a bunch of random things.