Keyboard Photogrammetry

User avatar
snacksthecat
✶✶✶✶

23 Nov 2019, 05:57

Perhaps this belongs in "offtopic" but ultimately decided to put it here because reasons.

Over the past month or so I've become enthralled with trying to reproduce keyboard in my collection (particularly the interesting looking ones) in 3D using photogrammetry.

What is photogrammetry? In a nutshell, you take a bunch of photos of an object or place, feed them into some software, and out pops a 3D model. Of course this is an over simplification but that's generally the idea.

Now there are a variety of programs that do this task. They range from free to wildly expensive (mainly for industrial applications). I've tried a handful of these programs and have some opinions I can share if anyone's interested. I'll probably save that for a later date.

For now what I wanted to do was go through my journey and the basic things I've leaned. Some of these turned out better than others, but I think I've landed on a workflow that produces good results. Well at least that's the conclusion I've come to with my last scan which turned out very nice. The next keyboard I scan will validate whether that is true or not.

For each of these I'm linking to a short youtube clip which shows the model at a glance, but also a link to Sketchfab where you can actually move it around and examine the model. Really cool site!

The first board I decided to scan was this cute little Micro Switch SD board that I'm trying to convert. It was already in pieces and looked more interesting that way so I decided to go without the case.

https://sketchfab.com/3d-models/micro-s ... d93bfeeb06
Hey, not too shabby for a first shot. Stuff in the back got all jumbled up because I had a bunch of jumper wires hooked up to it. I thought that would look cool but ultimately it just confused the software. But you can really see the potential of capturing small details like electrical components. That really impressed me.

From here I moved on to a Model F AT, thinking I was a pro by now. Boy was I wrong...

https://sketchfab.com/3d-models/ibm-mod ... e220f1034e
I was proud of myself that I was able to get the whole thing to scan, top and bottom. Sure you can tell what it is, but it looks like it was touched by Mr Freeze. The texture is all wrong and the file size is bloated out of this world. At this point I figured there was something wrong with the software I was using (RealityCapture) so I moved on to other packages. I also thought that you needed a crazy number of triangles and vertices to get a good result and that's just simply not the case.

Next I moved on to an IBM M13 keyboard...

https://sketchfab.com/3d-models/ibm-m13 ... 5cfbc07f93
Okay, it looks like the keycaps are made of melted-together marshmallows. WTF happened to the results I got with my first scan? I barely knew what I was doing and I had gotten decent results. Now I'm thinking of myself as a seasoned pro and everything's just getting worse.

But then I took a step back and examined what I could be doing differently. I'll type out my major lessons learned at some point (some I've not had a chance to implement yet so I'd like to prove these things out first). But just know that I made some significant changes when I scanned this last keyboard, Epson QX-10.

https://sketchfab.com/3d-models/epson-q ... 486b1fc12e
Now for this board I screwed up aligning the top and bottom chunks but otherwise, it came out really nice. All of the shapes are (mostly) solid, it's fairly crisp/clear, and even the little itty bitty components are intact.

Anyways, that's what I've got for now. Thanks for checking it out.

User avatar
vvp

23 Nov 2019, 22:16

Interesting.
How many photos were used for one scan?

User avatar
snacksthecat
✶✶✶✶

23 Nov 2019, 22:41

vvp wrote:
23 Nov 2019, 22:16
Interesting.
How many photos were used for one scan?
About 400 on average.

You can get decent results of just the top with about 100. The hard part is getting the software to recognize that the top and bottom belong to the same object. So you end up taking a bunch of incremental pictures to bridge the gap between them. And unfortunately keyboards are generally pretty thin so the geometry changes rapidly as you work your way down the sides which also confuses the program.

Some software packages allow you to manually place control points on the images to tie them together by common features which results in more images matching but it's a very manual process and doesn't always work out how you'd like it to.

User avatar
Muirium
µ

23 Nov 2019, 22:46

400 pics sounds like a lot of work. Have you automated them at all? Rigged a jig? Spun a turntable?

Naturally, I sent a link to my 3D modeling enthusiast friend who once made a nice SSK to plunk into Unreal, based on my own. I wonder which process is more time consuming!

Nice Greenock, Scotland made AT you've got there, Snacks. Yes, we can read its backside just nicely!

The irony is my own, Scotland located AT here is an American model. Guess we're spooky action at a distance, quantum counterweights.

User avatar
snacksthecat
✶✶✶✶

23 Nov 2019, 23:09

In terms of automation, they actually do sell full body scanning rigs with 100 cameras that fire simultaneously!

For my workflow, all I've really been able to automate is the removal of the background. I've been shooting against a solid green background and was previously manually masking it out in photoshop (talk about a lot of work). Then I found out that Agisoft Metashape actually has an intelligent background removal feature where you end up only having to mask out the background of two images and the software picks up on what to ignore from there.

I'm almost certainly overkilling it at this point. Probably taking a lot of redundant pictures of the top and bottom and not enough good ones of the sides. I just struggle a lot trying to get the side photos to register at all. There's unfortunately so many factors to take into account and it's a lot to keep track of. But as frustrating as that is, it's so damn satisfying when it all comes together and you see it pop up on the screen :P
Last edited by snacksthecat on 23 Nov 2019, 23:48, edited 1 time in total.

User avatar
XMIT
[ XMIT ]

23 Nov 2019, 23:17

I'm interested in this for 3D scanning of vintage key caps. It's easy enough to put something on a lazy susan, mark off 1 degree increments, put the camera on a tripod, and take a bunch of pictures.

User avatar
XMIT
[ XMIT ]

23 Nov 2019, 23:19

I've tried a handful of these programs and have some opinions I can share if anyone's interested
I'm interested, can you post your findings to this thread?

User avatar
snacksthecat
✶✶✶✶

23 Nov 2019, 23:23

XMIT wrote:
23 Nov 2019, 23:17
I'm interested in this for 3D scanning of vintage key caps. It's easy enough to put something on a lazy susan, mark off 1 degree increments, put the camera on a tripod, and take a bunch of pictures.
Ya know, it's funny you say that because I was thinking of scanning my red XMIT cap and sending it to you.

Image

User avatar
XMIT
[ XMIT ]

24 Nov 2019, 01:43

Haha! Sure. For me the goal is to use this idea of photogrammetry to give me a 3D scanner workflow, enabling me to take a key cap and clone it with a resin printer.

User avatar
snacksthecat
✶✶✶✶

26 Nov 2019, 03:13

Follows is a rundown of the software packages that I've tried, filtered down to just the ones I think are worth your time. Two are paid software (though different payment models) and one is free.

Before I go into details, I wanted to give a quick rundown of the photogrammetry process, which is basically the same across any tool you use.
  1. First you load in your photos and run photo alignment. The software examines each photo for interesting features and finds matching photos among the inputs. By running the alignment, the program estimates camera positions. This results in a sparse cloud which is basically a rough 3D model consisting of points. If your alignment went well, the sparse cloud will take on the approximate shape of the object/scene. This sounds fairly simple but can become very frustrating if your photos don't all register/align. What looks obvious to the naked human eye isn't always so obvious to the computer. The better your photos are, the more photos you'll get to register (as well as many downstream benefits in the steps below). Some softwares will allow you to refine the sparse cloud to filter out noise, which is a really nice feature. Others, you don't have this type of control.
  2. Once you're happy with the alignment, you move onto the creation of a dense cloud. This looks similar to the sparse cloud but is much more refined and detailed. This will much more closely resemble the object/scene. The dense cloud will, for instance, reflect depth much more accurately.
  3. Next the program creates a mesh which basically means that all that dense cloud data gets translated into a solid object.
  4. Finally, the program creates a texture for the mesh. The texture contains all the pretty details that make the model look photorealitic.
For me, the hardest part is alignment. There are some rules of thumb to follow that will help align more photos and result in a better model. I'll go into detail of what I've learned later but some important ones off the top of my head are:
  • Try to get a lot of the subject in frame. I've read that you should aim to get 80% of the frame filled with the subject.
  • More overlap between photos is best. I believe the recommendation is to try for at least 60% overlap between pictures.
  • Don't zoom (it confuses the program). The recommendation here is to use a prime lens but I use the kit lens that came with my camera, zoomed all the way out. I tape it in place so it doesn't move even a millimeter. But prime lens is best.
  • Don't use auto white balance.
  • Don't use autofocus (I use it because I'm lazy but I always have it set to focus center frame)
RealityCapture (Capturing Reality)
https://www.capturingreality.com/

This used to be a very expensive package until recently when they changed their licensing model. Now the model is basically you buy licenses for your inputs and only pay for what you produce with those inputs. What I mean by this is, if your model consists of 200 photos, you run those photos through the program and if you're happy with the result, you buy licences for those 200 inputs (photos). The benefit of this is there's no upfront cost and you're not stuck with an expensive piece of software that you've invested lots of money into. The micro switch keyboard I did above cost about $17.

The software itself is really nice once you get used to the quirky UI. Photo alignment on this package runs very fast (relative to the others). The output of photo alignment here is one or more "components" or pieces of the puzzle. These can be enriched by adding "control points" which tell the computer that features in the model match up with features in the photos. So you're able to get more pictures to register that way. The fact that control points are available in this package is a really nice feature, since they're only available in the pro versions of other packages. The other thing that you can do with control points is get components to merge together. So if you have the top of your object in one component and the bottom of the component in another, place a couple control points, run alignment, and they'll get merged into one. This is actually kind of fun (until it doesn't work and you want to pull your hair out).

Metashape (Agisoft)
https://www.agisoft.com/

This is what I've been using and I feel there's a lot of potential, if you have good inputs. I won't talk about the pro version since it's prohibitively expensive for hobbiests. But the standard version is $179. I feel that the documentation for this one is much more complete and there's also a very active forum/community that you can dip into as well.

Compared to RealityCapture, this one runs a bit slower but nothing outrageous. With all of these packages it's a trade off between processing time and quality. You can crank up the settings but you're going to pay for it in processing time. I like to run thing through on the lowest settings to produce a rough draft and if it looks good, then I'll go with the uber quality settings. The epson board above took about 20 hours to process on "ultra high". But even one notch down can take a fraction of that time. This is explained well in the user manual so I won't butcher it with my interpretation of how it works.

One thing to note here is that the standard version of this one does not have control points. That may sound like a big downside but the more I'm doing this stuff, the less I'm finding a need for them. In this software you work with "chunks" (similar to components above). So if you have a bunch of common sparse/dense cloud points between two chunks, you can run the "align chunks" action and the program will try to align them for you. This usually gives me crazy results but I manually refine the alignment, using that as a starting point.

Meshroom (Alicevision)
https://github.com/alicevision/meshroom

Hey a free one! This tool is really cool and I feel it has a lot of power in the right hands. Basically you start off with this long, node-based default workflow. For each node, you can tweak the parameters to produce the result you want (again, time/quality trade off). I feel you can get good results by playing a lot and doing lots of refinement to the workflow. The downside is, you can't step in any manually intervene. For me this one is a no-go because I can never get the top and bottom of the keyboard to register. I'm getting closer and closer, but still no luck.


Other honorable mentions (that I've not played with):
Hope this is helpful to someone :)

User avatar
snacksthecat
✶✶✶✶

27 Nov 2019, 15:46

Another challenge that I have is getting enough light in my dingy living room.

Here's my humble little setup.

Image

The reason it's somewhat difficult is because you're supposed to shoot with the f/stop way high in order to get as much depth of field as possible. If you don't, you risk making parts of the photo out of focus. Particularly when taking pictures at more extreme angles. This is something I was unaware of when I started out and making this adjustment has made quite a big difference in my results.

Blurry photos can ruin the end result. So much so that I've found if a part of an image is blurry or has a glare, you're better off masking that part of the image out so it's removed from the equation all together. Either that or retake the photo.

User avatar
Muirium
µ

27 Nov 2019, 18:03

I like your little desk. Of course I would!

What shutter speed are you going with? MORE LIGHT is indeed always the answer for deep depth of field, but it can be emphasised at the cost of time.

User avatar
snacksthecat
✶✶✶✶

28 Nov 2019, 00:16

Oh right! I probably should have mentioned that of course ISO is another factor. I have that set all the way down to 100 in order to reduce noise as much as I can.

So my shutter speed is somewhere around 2 seconds :o

Not a photography expert by any means so it's entirely possible that I'm doing something wrong. But yes, it does take a long time to snap all those photos.

I'm working on a model right now where I tried to be much more disciplined and follow all the camera rules I outlined above. I'm getting pretty good results with 214 photos. Only one photo failed to align and I'm happy with how it's turning out.

User avatar
XMIT
[ XMIT ]

28 Nov 2019, 00:20

Turn those umbrellas around. Shine-through umbrellas are meant to give you light through them, not like bounce-back umbrellas with a darker side and a lighter side. The umbrellas are diffusers meant to reduce specular highlights. (I think I have the same ones.)

Get the brightest bulbs you can. Try to make sure they are all the same color temperature. 5000K is your friend.

Consider light boxes instead. You can make some out of black cardboard and parchment paper if you're on a budget. The trouble with umbrellas is too much light bleed, it's hard to control the light.

User avatar
HAL

28 Nov 2019, 02:20

Are you using a remote-control release?
I found that even when using a tripod the pictures can get a little blurry when you press the release.
I am using something like this:
https://www.amazon.de/LUMOS-Fernauslöse ... B07K8QB97P

User avatar
snacksthecat
✶✶✶✶

28 Nov 2019, 02:34

XMIT wrote:
28 Nov 2019, 00:20
Turn those umbrellas around. Shine-through umbrellas are meant to give you light through them, not like bounce-back umbrellas with a darker side and a lighter side. The umbrellas are diffusers meant to reduce specular highlights. (I think I have the same ones.)

Get the brightest bulbs you can. Try to make sure they are all the same color temperature. 5000K is your friend.

Consider light boxes instead. You can make some out of black cardboard and parchment paper if you're on a budget. The trouble with umbrellas is too much light bleed, it's hard to control the light.
Ooof, what a noobie mistake! Definitely going to switch them around next time. I'll look into what it takes to make a medium sized lightbox as well. That's something I remember exploring a long time back.

I've also heard that halogen is better than these compact florescent lights in terms of CRI. But I don't know what CRI is so I guess that's a good starting place. These were fairly inexpensive off amazon, same with the umbrellas. Although the fixture on the umbrellas is wonky and you have to screw the bulb in just right for them to work.

Always room to improve I suppose!
HAL wrote:
28 Nov 2019, 02:20
Are you using a remote-control release?
I found that even when using a tripod the pictures can get a little blurry when you press the release.
I am using something like this:
https://www.amazon.de/LUMOS-Fernauslöse ... B07K8QB97P
I really should pick up a remote at some point. They're cheap and, like you said, avoid shake issues all together.

Instead I set my camera to a two second delay. I press the button, it beeps down two seconds, and then the shutter goes. But certainly a remote would be more handy, especially considering you need to keep moving back and forth to rotate/move the object.

One guy on /r/photogrammetry has gotten really good results using an automatic turn table. You basically just dial it into what interval you'd like and it rotates on it's own. I think this is the one he was using:
https://www.amazon.com/dp/B07SC67Q93/re ... k_detail_4

I got a kick out of his theory that he could sync up the timing of the camera and the turntable to have a fully automated workflow.

User avatar
snacksthecat
✶✶✶✶

28 Nov 2019, 06:06

This one was kind of like one step forward, two steps back. I discovered that (at least with my data sets) I get much better results meshing from the dense cloud data rather than depth maps (this is all with Agisoft Metashape standard edition).

But there are some obvious problems. Namely the green residue left over from not masking out the green backdrop I shot against. And it has a number of holes, resulting from reprojection errors and/or not enough input data.

Still decided to share it because I think it's such a neat little keyboard. The idea definitely had potential but I'm going to toss this into the failure pile.

Oh, and I also learned one more trick which is: the program automatically categorizes all of the points into classes. And there's an option to select points by class so they can be deleted. I think in the future I can use this technique to remove unwanted elements in the point cloud so that they're removed from the equation when you generate the mesh. So that's a neat little tool to have at our disposal. I've tried doing a similar technique, selecting points by common colors (green in this case) but couldn't exactly get it working right.

If you're handy with tools like Meshmixer, Meshlab, or Blender (all free); you can also fill the holes. But perhaps these holes were too big because I couldn't get that to work either.

In any case, here's the final exported model. Sorry it's a little large, building in such high detail is probably a crutch.

https://skfb.ly/6OWGU

User avatar
XMIT
[ XMIT ]

28 Nov 2019, 16:42

I have a Nikon with infrared release. The remote was under $10. Even less if you have a phone with an IR blaster like my old one.

https://www.amazon.com/AmazonBasics-Wir ... 003L1ZYZ6/

I said "light box" before and realize that was vague. Soft boxes are better for keyboard work. Example link:

https://www.amazon.com/HPUSN-Profession ... 07NBP6D98/

You can also use a "portable photo studio" box but since you're chroma-keying (the name for the technique of removing a green background) it won't work unless you paint it green.

User avatar
snacksthecat
✶✶✶✶

01 Dec 2019, 01:35

Thanks for the links, XMIT. I decided to pick up the soft boxes you shared. My umbrellas are on their last leg (as I mentioned before, the socket is very finicky with how the bulb is screwed in). The boxes were also on sale which is nice as well. Excited to give them a whirl.

This next model I actually shot a while back, so hopefully I've learned some things since then. Nevertheless, I wanted to share it and talk about what turned out good vs. bad.

The Good
Keycaps here actually look really nice. Almost photorealistic in most spots. Compared to other models I've done, I think the trick here is exposure. If the pictures are overexposed, You don't get the definition you'd like to see between the caps. Texturing also came out really nice, without too much weirdness.

Image

The Decent
Switches turned out okay. The color isn't quite right, which is probably on me for how I took the pictures. The general shape is on-point but a little bit too rounded off for my taste. Bare in mind that the meshes for these models have been decimated down to comply with the sketfab upload limits. I have a pro account so that allows me 200mb per upload. Even after decimation these are still pretty bloated. If you look at the untextured meshes, you'll see a whole bunch of extra bumps and stuff that shouldn't be there on a smooth surface. This again can be fixed in post, but I'm no good with 3D modeling yet so I just stick with what comes out of Metashape.

Image

The Bad
The backside of the keyboard is totally smooth, which is difficult for the program to figure out. It needs features or texture in order to reconstruct properly. That is why the stickers look good but the plastic is all wacky. Another factor is that it's shiny, which also confuses things. A tip I've been given to combat this is to dust the shiny/unfeatured surface with powder (e.g. baby powder). Well forget that, I'm not getting this fine keyboard all dirty just to make a model :lol: I have plenty of dirty boards anyways. Plus, the dirty ones turn out a bit more interesting, IMO.

Image

The Ugly
The sides. Here there are two factors at play. Again, these are sparsely featured which we already know is a problem. In addition to that, it's very difficult to get pictures of the sides to match up during photo alignment step because the geometry changes so quickly as you move down to more extreme angles. I'm still trying to figure out a good method for capturing the sides. If anyone has any ideas I'm willing to try anything.

Image

Thanks again for reading. Getting this stuff down on paper helps me digest things and reflect on what's making a difference vs. not.

https://sketchfab.com/3d-models/leading ... 68e915315e

User avatar
snacksthecat
✶✶✶✶

05 Dec 2019, 20:48

Another day another crappy 3D model that nobody asked for :lol:

This time I tried scanning an old school Cherry keyboard that I have. This was probably the most difficult scan I've attempted so far. Reason being; it has a bunch of detail (e.g. risers and switch bottoms) between the plate and PCB. I took a lot of close-up photos of these parts to capture everything and of course, hardly any of it registered. You can see some of the switch housings if you examine it from the sides, but the front view is completely empty.

https://skfb.ly/6PpGV
Image

Here's another breakdown of what turned out good/bad.

The Good
Not much to classify as good. The whole thing turned out pretty mediocre. Shame because it's a really neat looking keyboard and looking at it in good quality 3D would be very interesting.

The Decent
This time I was actually able to capture (most of) the space between the keycaps. In some of my past scans, the keycaps melt together into a big blob. The fact that I was able to capture much of the spacing can probably be attributed to two things: (1) I took close ups of each key (it didn't take that long) (2) I ran the "mesh refinement" procedure a few times. This helps to refine the generated mesh further by re-examining the photo set.
Image

The Bad
The texture is again all wonky. I really feel that it's likely due to how I'm taking the pictures and not "what" I'm taking pictures of. Reason I feel this way is because I took a ton of close up pictures, most of which did register. Perhaps it's the fact that I was still using auto focus at this point. The best practice is to manually focus on whatever is in the center of the frame. I've since started to do this. It doesn't add too much time because most new cameras have some manual focus assist features to aid you. I've just started another keyboard using only manual focus and all of the pictures so far have registered, which is a good sign for all subsequent steps in the process.
Image

The Ugly
Hardly any of the detail between the plate and PCB layers shows up in the final model. I also had to run this in two chunks and manually align them, making things even more disjointed. I have a new turntable technique I'm playing with which should help me capture everything in one chunk.
Image

User avatar
adamcobabe

05 Dec 2019, 21:23

These are fun. I would second the suggestion to use a lazy susan type device. It doesn't need to be automatic, though that's nice. Just add some clock ticks on the edge and rotate it by hand. I work in visual effects and I've seen the capture people use that method.

I'd also suggest getting rid of the green screen. It's just going to cast a green tint on everything and you don't need to remove the background in the images. Just delete any background geometry after meshing.

Post Reply

Return to “Workshop”