A better Rubygems search Tweet

Some time ago, Kaspar mentioned to me that it would be nice to have a gem dependency search, i.e. where you could search in which gems a gem is used.

I thought so too, so I wrote one :) (and added some more features in the process)

Take a look: http://gemsearch.heroku.com/

(Note, it might take Heroku some time to ramp up the server)

Current state

While the current search isn’t bad, it is missing the possibility of searching for an author, where a gem is used, or which version it has. Or any combination thereof, for that matter.

Building the search

I happened to have a fast & clever search engine lying around ;) so this is what I used.

How do you go about building or configuring a search engine?

1. Look at what your goals are.

My goals seemed simple enough.

Each gem should be findable under:

• Its name (Try it).
• Its version(s), entered like x.y.z, or part thereof, x, x., x.y, x.y. (try it).
• Its author’s names, or first/last names. Or parts thereof, like “flo” for florian (try it).
• The gems it is dependent upon. universe-parsing depends on parslet, for example (try it).
• The names, gem name and dependent gem name should be phonetically findable (try it).
• The authors too should be phonetically findable – since who knows how to write “Heinemeier” (try it)?
• All should be findable without entering the whole thing, like “1.0”, or “activesupp” (try it).
• One should be able to specify what he is looking for by prefixing e.g. “uses:” in front of the search term (Try it). Or others, like “dependency:”, “dependencies:”, “depends:”, “using:”, “uses:”, “use:”, “needs:” (all possible).

I leave out the description for now, as it requires quite a bit of thinking and tinkering.

With the goals defined…

2. Look at the data.

I downloaded the Marshal file, extracted the relevant data and restructured it into a readable CSV.

Two potential problems I noticed:

• Gem names are spaced using either an underscore _ or a hyphen -.
• For the same name, there are sometimes up to three different encodings. Take the gems of Nicolás Sanguinetti for example. Try it and look at the author names.

Those were problematic. What does one do? Try to find an optimal solution.

3. Marry the goals and the data.

I decided not to tackle the display issues of the second point, encodings, but just the indexing issues. What I do is use character substitution, to make “Nicolás” findable under “nicolas”. This I do by saving the name as “nicolas” in the index, and also perform this character substitution on each search. Try it with án áccent.

Deciding on what to with the gem names was a little harder. What is the problem?

The problem is manyfold. For one, searchers should not need to know whether a gem was spaced with an underscore or a hyphen. Actually, I thought it best they be able to find it using a space. So the picky-live gem should be findable by typing “picky live” (Try it).

However, if you then look for “sinatra”, the actual sinatra gem is not the first in the list. This is because I opted to go for an alphabetical ordering.

However, if I need the user to enter the full name (like “anthonymoralez-apn_on_rails”), they might not find it at all.

So, the way I did it now is have the user be able to use spaces when searching and trust people to depend on Picky’s combinatorial nature. For example, if you look for sinatra and know that one of the owners is called Tomayko, you’ll get to your answer directly: Search for ‘sinatra tomayko’.

Generally, the more you can help Picky, the more it will help you right back.

4. Have users try it and get feedback.

This is where you come in :)

Check it out, if you haven’t yet and tell me what you think @hanke! Do you have ideas for improvement? (If yes, tell me which so I can improve it)

How about we use this search on rubygems.org? :)

A few technical Picky specifics.

A few Picky specifics for insiders:

• We have 4 data categories: name, version, author, dependencies.
• The partial search “rail*” is done using Partial::Substring.new(from: 1).
• The similarity “hallou~” is done using: Similarity::Phonetic.new(2).
• A singly occurring name will be weighted up a little: :weights => { [:name] => +1 } }.
• The author for example can be prefixed with: qualifiers: [:author, :authors, :written, :writer, :by].

Yes, currently I break the web with hashtags – I’m rewriting it to use pushState.

Thanks

Many thanks to Heroku for providing the infrastructure!

Conclusions

So we’ve seen

1. that there’s a better way to search Rubygems
2. where you can go to try it
3. how you could go about creating a search

Hope you learnt something new :)

Next Searching with Picky: Live reloading indexes

Share

Tweet

Previous Running Sinatra inside a Ruby Gem

Comments?