KBDB?

[Muirium]

However, it seems apparent to me that people are trying to find ways to replace a system they won't use and don't understand, and therefore cannot reliably comment on its strengths and weaknesses. Likewise, they cannot conceive exactly how much work is involved in solving the stated problems.

Take-home questions:

1) How does creating a separate database stimulate wiki growth?

2) Are we satisified that the database and wiki will remain in sync?

All true. And now my suitably conditioned answers:

1. It does not. Directly, at any rate.
2. Nope. The database might be a dud. In which case sync doesn't matter.

The best case scenario is the db gets a bunch of wiki wary rubes (hi) to contribute… to the db. That's really the purpose of the project, as I can see it: a neat oracle for arbitrary queries (the perfect "this as been asked before" link for many newcomers' questions), including our own collections (which ought to really juice updates), with a drop dead easy form for punching in info on every keyboard the simple folk among us (inc. yours truly) come across.

If that structured data turns out to be useful, sync becomes an issue worth addressing. But if it's a dud, let it be.

My gut feeling, for what it's worth, is that keyboards are a sufficiently closed set to be compatible with a crafty database. So many subjects are not: those require the free form a wiki provides. A pertinent example is all the knowledge around keyboards that our own wiki addresses: like switch histories and terminology definitions. KBDB can't address that, nor is it necessary for it to try. The two technologies difference is their strength. Including the different people to which they appeal.

[Halvar]

OK, I think what I learned from the last few statements about the state of things after this discussion is:

1) the maintainers and main authors of the wiki here (webwit, Daniel) very strongly oppose the thought of a separate structured/query-able database somewhere else and clearly view it as a threat to the future of the DT wiki. They want a solution that is an extension to the wiki.

2) matt3o doesn't seem to view wiki and database as complements to each other any more either, but now calls his database an attempt to find a "better way".

So, it seems like these are not the conditions for this to become a peaceful/fruitful loose cooperation of two complementing systems with different strengths. Which is kind of sad, because I think it could have been.

While I still think the wiki-based database will be a kludge (and a tables-based universal knowledge collection would be a stretch too), all in all the wiki is much more important to me, more fit to the task of user-generated knowledge collecting about the whole topic IMO, and last not least all of this work has been put into it already.

I guess my question is: is Deskthority going to work on a structured wiki? if so, I'm not going to waste time on a new project.

I'm afraid this question has now been very clearly answered by Daniel, and it has kind of been answered by webwit before.

[webwit]

1) the maintainers and main authors of the wiki here (webwit, Daniel) very strongly oppose the thought of a separate structured/query-able database somewhere else and clearly view it as a threat to the future of the DT wiki. They want a solution that is an extension to the wiki.

Not necessarily, and you had some good arguments earlier. Personally if I would make a db, I'd say fuck it to the wiki, so it doesn't hold me back. Then I might use the wiki info later to input new keyboards in the db. But I wouldn't import/export. Because if you design a proper relational database, it will not be the same as a few columns in an info box. I still think if one scrapes the wiki to make it searchable, it must either be read-only or give back, it's the proper thing to do.

[Daniel Beardsmore]

I do not feel that the database threatens the wiki, only that the people who "value" the wiki still treat it like it's run by leprechauns.

[Halvar]

@webwit: would exposing the database for automated exports for a DT programmer to convert and feed it back into the wiki in the way he sees fit be considered "giving back", or would you consider that "leaving it up to the leprechauns"?

@Daniel: If these people who "value" the wiki treated it like it was run by real people and open to their contributions, they would ... ? I don't see what you're saying in the context of this thread. Contribute more? Value it even more? Not hurt those people's feelings?

[webwit]

We don't have a team of DT programmers, it doesn't work that way. This is a hobby, so members work on phpBB or mediaWiki when they have an itch and when they have the time. I currently don't have the time and a big todo list waiting for me. So if the DB developer wants someone else to do the link with mediaWIki, it all depends if there is a volunteer who likes to do this.

[Halvar]

I hope I don't surprise you too much: I actually knew or strongly suspected that DT doesn't have it own team of programmers! I also strongly suspect that you don't just let some arbitrary registered members of DT feed whatever they like into the wiki in an automated import.

So it is probably somewhere in between ...

So what I understand is: for you this would only be a fair deal if matt3o, if he uses data from the wiki to initially fill the database, not only exposed the database back to deskthority but also managed how "his" data are fed back into the wiki in a sensible way?

[webwit]

That would be nice, because otherwise it's probably not happening unless someone volunteers. But hey, it's open source info and his itch, so matt3o should do whatever he likes.

[Daniel Beardsmore]

@Daniel: If these people who "value" the wiki treated it like it was run by real people and open to their contributions, they would ... ? I don't see what you're saying in the context of this thread. Contribute more? Value it even more? Not hurt those people's feelings?

Let's put it another way: if a group of regular contributors to the wiki decided that we need to improve the efficiency of maintaining and accessing the data, I would take them far more seriously than people who don't even bother with it. It's a surprisingly complicated problem to solve, and few if any in this discussion have the will or the subject knowledge to actually consider it deeply and understand what's actually required.

Even a read-only scraper will take a lot of work to corral all the data.

No-one has actually put their hand up and affirmed that they are happy to manually sync data in either direction, which means that the wiki will never benefit from any updated data. No-one is prepared to think this through properly and they're simply ignoring the facts that don't suit their dream bubble.

Several people have noted that they value the wiki, but they're letting someone else do all the work. In what way do you think this inspires me to believe that these same people have the willpower to design, build, maintain, support and oversee a new project and keep this up for years to come?

[Halvar]

My experience is with databases and programming much more than with keyboards. I don't have a lot to contribute to this already very comprehensive wiki, but I'm a software developer and was technical admin on a hobbyist forum with now > 1.3 million posts and experimenting with different forms of user-generated knowledge collection (we decided against a wiki in 2004) for almost 10 years. That's why I'm interested in this.

This thread started with matt3o proposing an independent database, without talking about syncing with the wiki or scraping stuff from it at all. Programming this database would be a lot of work that matt3o wanted to do, with the goal to have a database that can be queried mainly by people interested in buying a keyboard. This would most probably lead to a database design that's more different from the wiki's infoboxes than you'd think. (webwit said this already.)

So that was the offer that was on the table, and his (then still implicit) question was: "is Deskthority going to work on a structured wiki?"

Then the first posts from you and webwit started to talk about how it needs to be tied back to the wiki and how all the data should be in "Boyce–Codd normal form" (including the wiki), und ultimately how the wiki must always have all the information that the database has. That is actually the point where it started to become "surprisingly complicated".
Or, as I would put it, "almost impossible without constraining the database far too much".

Is that goal really so important that it's worth the effort? To whom? Certainly not to most users. They don't care where they find the information. Hell, many of them will think they "just googled it".

The alternative, the database within the wiki, is only happening if DT users make it happen as we see now, both with programming and contributions. I have been very hesitant so far in promising to help because I still don't see how this could be a good solution. I only see Semantic MediaWiki or scraping tables from infoboxes as possible solutions as of now, and as far as I see both can only be made a so-so solution with an immense programming effort that nobody's going to put into it because it's frankly not worth it. If there's a better solution you know of, let us know.

I would rather help with a solution that regularly tries to deep-link database entries to wiki articles and vice versa and maybe later tries strictly limited syncs back into the wiki where possible because that seems easier and makes more sense to me.

Whatever you do, please make a decision on matt3o's question and live with it. Don't impede his offer to the community with something that you would merely like to happen if it probably won't.

[matt3o]

Merge db and wiki (for keyboards of course).

Rules:
- A wiki page is only created by the DB. If you want to add a keyboard to the wiki you have to add it from the DB front-end first
- Structural data is only handled by the DB. Every time a KB is updated in the DB the wiki is updated. You cannot update structural data from the wiki.
- The wiki is only for historical/anecdotal kind of information or anyway all that info that doesn't match the DB.

How to proceed:
- step 1, scrape the current wiki and fill the DB
- step 2, create the new wiki pages with just the DB info
- step 3, take the additional current wiki info that doesn't fit the DB and fill the new wiki
- step 4, live happily ever after

A hell of a job!

Edit: I'm not saying that I'm going to do that. It's just a crazy proposal.

[002]

I would like to see matt3o's idea come to fruition but I think it should just start with a clean slate and not even bother trying to scrape anything from the DT wiki.

Let's face it, the wiki is like 90% DanielB and 10% the rest of us. Even though it might be easy (and really, it is) some people are probably too intimidated to make edits for whatever reason -- KBDB could potentially be much more noob/sloth-friendly. I think that it has to have some control over who can edit it as well as a way for mitigating the risk of dirty data for (e.g. limiting free-text fields?). There are a number of ideas running through my head now about how the data entry would occur. For any of you who are what.cd members, you might be familiar with the upload form which has some cool ideas that might work well for KBDB too. You could make submissions to the database (either as new entries or as values for the various fields) subject to review and all that stuff. You could affix multiple values to a various keyboard model as was mentioned earlier. Free-typing into any of the fields during data entry would auto-complete for stuff that was already there.

Maybe I've got it all wrong in my head but either way, I'd really just like to see something tangible (yet safe) to muck around with to begin with

[matt3o]

current status... so beta that's not even alpha... but you get some spotlight on the available features.

[Muirium]

Neat!

Of course, we're going to want ways to query for "all TKL and smaller currently available in mx clear" and stuff. I prefer to construct filters in a graphical form than as a text search query. But any way is better than none.

Quite impressed it's already looking good.

[matt3o]

I was thinking to let people search by: free text (the one pictured), switch type, form factor, brand. All these for the "basic search", you have then the "advanced search" where you can filter by any field in the DB (with select boxes or whatever).

Also you can browse the whole database (in table view). Initially you get the whole DB and then you can visually filter (and reorder) the results.

The freetext search might be augmented with some arcane syntax such as: "switch:mxblue", "layout:tkl", "color:black" etc... As of now the standard search searches in: brand, model name, model number.

[Muirium]

Searching across many fields is appealing, too. A dead simple example would be "TKL" to show up every single Tenkeyless board. But the trouble starts when model names or manufacturers also match. Like every Cherry keyboard showing up if you enter "cherry mx white".

[HaaTa]

How is this coming along?

[matt3o]

I'm slowly working on it. I suggest you to visit http://kbdb.io once in a while. Anyway I'll post the updates here as well.