rake environment tire:import CLASS=User take hours to index records. is this normal?

[naveenagarwal]

Hi,

I have around 10 million records and and using mongodb as primary storage engine. When I run rake environment tire:imort CLASS=User, it takes more 2 hours to index. Is this normal? Is there any faster way to do it? Also, if I add more records to the User model will it recreate the full index and will take again hours to re-index?

[dzhlobo]

I guess you have identity map enabled. Tire iterates through all users and mongoid put them all in memory if identity map is enabled. I made pull request in which I disable identity map during process. See #884. It would be nice if you try and tell us if it is help you. (It helps in my projects.)

Add in your Gemfile:

gem 'tire', github: 'Proghat/tire', branch: 'mongoid-identity-map'

Also, if I add more records to the User model will it recreate the full index and will take again hours to re-index?

It depends on you. You can simple use tire's callbacks which add new documents into index after creation or program another logic yourself. See https://github.com/karmi/retire#activemodel-integration.

[naveenagarwal]

Using your tire version with mongoid throws the error when I ran the rake tire:imort:all

rake aborted!
undefined method `unit_of_work' for Mongoid:Module

I tried to search this method in the mongoid gem but it seems has been used in benchmarks but method is not defined anywhere.

[dzhlobo]

See it in docs: http://mongoid.org/en/mongoid/docs/identity_map.html#unit_of_work