Tag Archives: Gitorious

Gitorious & Resque

As I’ve mentioned in previous posts, I’ve had problems with resque, as it’s currently set up, running as its supposed to on Centos 6.5. I can see the application active if I check for it:

[root@srv-git-01 ~]# ps -ef | grep "resque"                                     
root     27821     1  0 13:14 ?        00:00:00 /bin/bash -i -l -c chruby ruby- && /var/www/gitorious/app/bin/rake resque:work QUEUE=*
git      27850 27821 77 13:14 ?        00:00:02 ruby /var/www/gitorious/app/bin/rake resque:work QUEUE=*
root     27853  7112  0 13:14 pts/0    00:00:00 grep resque

But it isn’t doing the work it’s supposed to: I manually have to run the command for new repositories to be created, and updates from users don’t show up on the website until I run resque manually.

I now have a way to make it work. I kill the above processes that are running. I su to the Gitorious user, git, in our case, and run the command from the Upstart file from the command line:

-bash-4.1$ chruby-exec ruby- — bin/rake resque:work QUEUE=*

I again ‘ps’ to find the running process:

root@srv-git-01 ~]# ps -ef | grep "resque"                                     
git      33861 33601  0 13:22 pts/0    00:00:00 /bin/bash -i -l -c chruby ruby- && bin/rake resque:work QUEUE=*
git      33889 33861 56 13:22 pts/0    00:00:13 ruby bin/rake resque:work QUEUE=*
root     33931 33903  0 13:22 pts/1    00:00:00 grep resque

I kill the parent process, in this case 33861:

[root@srv-git-01 ~]# kill -9 33861
[root@srv-git-01 ~]# ps -ef | grep "resque"
git      33889     1 62 13:22 pts/0    00:00:26 resque-1.24.1: Waiting for *     
root     34055 33903  0 13:23 pts/1    00:00:00 grep resque

Now the child process is running and, as you can see from the second check, it’s in a ‘waiting’ state. And, in this state, it is working as I had expected it to.

I still haven’t figured out how to do this to run correctly at start. That’s the next step.

As an aside, when doing my latest search on this topic, I discovered that Upstart is going away:

http://unix.stackexchange.com/questions/118454/upstart-documentation-for-centos-6

I haven’t done more research to find out why. But maybe the alternative will solve my problem.

Cleaning Up Danglers on Gitorious

We’ve had problems lately with the server running out of space. It seems to hearken back to two items:

  1. We didn’t allocate enough space on the VM in the first place
  2. The external drive to which the backups are supposed to be stored seems to go down alot. So the backup is staying on the local box. Since it’s about 5 GB, the space fills up fast.

We finally allocated more space for the hard drive. But, in the interim, some of the gitorious files seemed to have been corrupted on creation. Not everyone was having problems, which made it difficult to diagnose. The first item that showed up was this:

ActionView::Template::Error (/var/www/gitorious/repositories/raisr/raisr_detailselectronics.git):
    24: 
    25:
<div class="gts-events">

26: <% prev_date = nil %> 
27: <% events.map { |event| EventPresenter.build(event, self) }.each do |event| %> 
28: <% if prev_date != event.created_at.strftime("%A, %B %e %Y") %> 
29:
<p class="gts-date-heading">30: <span class="gts-date-heading-inner"> app/models/repository.rb:250:in `new' 
  app/models/repository.rb:250:in `git' 
  app/presenters/event_presenter/push_summary_event.rb:165:in `fetch_first_commit' 
  app/presenters/event_presenter/push_summary_event.rb:132:in `initialize_commits' 
  app/presenters/event_presenter/push_summary_event.rb:31:in `initialize' 
  app/presenters/event_presenter.rb:28:in `new' 
  app/presenters/event_presenter.rb:28:in `build' 
  app/views/events/_events.html.erb:27:in `block in _app_views_events__events_html_erb___4043706570647562128_49011240' 
  app/views/events/_events.html.erb:27:in `map' app/views/events/_events.html.erb:27:in `_app_views_events__events_html_erb___4043706570647562128_49011240' 
  app/views/site/dashboard/_activities.html.erb:23:in `block in _app_views_site_dashboard__activities_html_erb__1312100781819782992_61802240' 
  app/helpers/tabs_helper.rb:89:in `block in tab_content' 
  app/helpers/tabs_helper.rb:88:in `tab_content' 
  app/helpers/tabs_helper.rb:55:in `block in tabbable' app/helpers/tabs_helper.rb:44:in `tabbable' app/helpers/tabs_helper.rb:61:in `pjax_tabbable' 
  app/helpers/tabs_helper.rb:35:in `activity_tabbable' 
  app/views/site/dashboard/_activities.html.erb:22:in `_app_views_site_dashboard__activities_html_erb__1312100781819782992_61802240' app/views/site/dashboard.html.erb:24:in `_app_views_site_dashboard_html_erb__754282507050535463_62777760' 
  app/controllers/site_controller.rb:123:in `block (3 levels) in render_dashboard' 
  app/controllers/site_controller.rb:111:in `block in render_dashboard' 
  app/controllers/application_controller.rb:339:in `paginate' 
  app/controllers/site_controller.rb:110:in `render_dashboard' 
  app/controllers/site_controller.rb:146:in `render_global_index' 
  app/controllers/site_controller.rb:39:in `index'

I ended up fixing that one by deleting the entry for that particular repository from the repositories table in the database, and by removing the one related comment that appeared in the committership table.

Before I did this, no one could get onto the Gitorious web interface. Afterwards, a few people were having problems, which made it a bit more difficult to debug. But then I saw this in the production log:

NoMethodError (undefined method `list_as_favorite?' for nil:NilClass):
 app/models/dashboard.rb:39:in `block in favorites'
 app/models/dashboard.rb:39:in `select'
 app/models/dashboard.rb:39:in `favorites'
 app/presenters/dashboard_presenter.rb:51:in `favorites'
 app/controllers/site_controller.rb:121:in `block (3 levels) in render_dashboard'
 app/controllers/site_controller.rb:111:in `block in render_dashboard'
 app/controllers/application_controller.rb:339:in `paginate'
 app/controllers/site_controller.rb:110:in `render_dashboard'
 app/controllers/site_controller.rb:146:in `render_global_index'
 app/controllers/site_controller.rb:39:in `index'

A search of the web finally brought me to the following solution in the Gitorious forum that worked. Thank you to Marcin Kulik for the solution:

This looks to me like an issue with old data in the db.
You may try running the following command in the app directory, as "git"
user:
RAILS_ENV=production bundle exec rake fix_dangling_comments
fix_dangling_committerships fix_dangling_events fix_dangling_favorites
fix_dangling_memberships fix_dangling_projects fix_dangling_repositories
fix_missing_repos fix_missing_wiki_repos fix_system_comments

(this is a single line in case email formatting breaks it)

Basically these tasks will find and fix inconsistent data in the
database and hopefully solve your issue with dashboard.

Marcin

The solution produced the following output:

[fix_dangling_comments] removing 0 orphaned comments
[fix_dangling_comments] removing 0 orphaned comments
[fix_dangling_committerships] removing 0 orphaned committerships
[fix_dangling_committerships] removing 0 orphaned committerships
[fix_dangling_events] removing 0 events with missing target Project
[fix_dangling_events] removing 0 events with missing target User
[fix_dangling_events] removing 0 events with missing target Group
[fix_dangling_events] removing 0 events with missing target MergeRequest
[fix_dangling_events] removing 2 events with missing target Repository
Removing 1 dangling favorites for Repository
Removing 0 dangling favorites for MergeRequest
Removing 0 dangling favorites for Project
Removing 0 invalid memberships
[fix_dangling_projects] removing 0 orphaned projects
[fix_dangling_projects] removing 0 orphaned projects
[fix_dangling_repositories] removing 0 orphaned repositories
[fix_dangling_repositories] removing 0 orphaned repositories
[fix_dangling_repositories] removing 0 orphaned repositories
[fix_dangling_repositories] removing 0 orphaned repositories
Attempting to fix 0 projects without wiki repo
Disabling edition of 0/0 merge request version comments

Migrating Gitorious to Another Server, Part III

If you’ve read my earlier posts, you know that I’ve migrated our Gitorious instance to a new server. In the project, I had to update it from 2.3 to 3.0.

All of the projects and data appears, so far, to have migrated successfully. The latest issue, however, that I’ve run into is that not all the projects show up on the Projects index page. Since all the projects are supposed to be visible, this didn’t make sense.

The fix ended up being time consuming but fairly straightforward. I did a search in the database to find all of the projects, their slugs, and their owners:

SELECT id, title, slug, owner_id FROM projects;

For each project not appearing on the index page, I:

  • opened the project page:
    • [main site url]/[slug]
  • opened the Admin menu by clicking the Admin button
  • clicked Edit
  • saved the edit page (no changes necessary)

Voila! The project appeared on the index page.

One small gotcha: if the admin (the user I was logged in as) wasn’t the owner of the project, the Admin button wouldn’t appear. So I’d temporarily change the owner of the site to admin:

update projects set owner_id = 1 where id = [project id from table];

Once I’d saved the Edit page, I simply changed the owner back to the original, as seen in the original table.