Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker2 #2756

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Docker2 #2756

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ postgres-data
redis-data
# Required for puma to have a place to put the pid file
!tmp/pids/.keep
public/assets
8 changes: 2 additions & 6 deletions Capfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,9 @@ require 'capistrano/deploy'
require 'capistrano/scm/git'
install_plugin Capistrano::SCM::Git

require 'capistrano/bundler'
require 'capistrano/rails'
require 'capistrano/honeybadger'
require 'capistrano/passenger'
require 'capistrano/maintenance'
require 'whenever/capistrano'
require 'dlss/capistrano'

require 'dlss/docker/capistrano'

# Load custom tasks from `lib/capistrano/tasks` if you have any defined
Dir.glob('lib/capistrano/tasks/*.rake').each { |r| import r }
7 changes: 3 additions & 4 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,8 @@ group :development, :test do
end

group :development do
gem 'faker'
gem 'listen', '~> 3.2'
gem 'multi_json', require: false # needed to update RBIs after adding reform-rails
gem 'puma', '~> 5.6', '>= 5.6.4'
gem 'spring'
gem 'spring-watcher-listen', '~> 2.0.0'
gem 'state_machines-graphviz'
Expand All @@ -41,9 +39,8 @@ end

group :deployment do
gem 'capistrano-maintenance', '~> 1.2', require: false
gem 'capistrano-passenger', require: false
gem 'capistrano-rails', require: false
gem 'dlss-capistrano', require: false
gem 'dlss-capistrano-docker', github: 'sul-dlss/dlss-capistrano-docker', branch: 't2-rollback', require: false
end

gem 'action_policy', '~> 0.5.3'
Expand All @@ -57,6 +54,7 @@ gem 'devise-remote-user', '~> 1.0'
gem 'druid-tools'
gem 'dry-types'
gem 'edtf'
gem 'faker'
gem 'faraday', '~> 2.0'
gem 'honeybadger', '~> 4.0'
gem 'jbuilder'
Expand All @@ -67,6 +65,7 @@ gem 'pg'
gem 'preservation-client', '~> 5.0'
gem 'propshaft'
gem 'pry'
gem 'puma', '~> 5.6', '>= 5.6.4'
gem 'redis', '~> 4.0'
# TODO: Deal with this
# pinned because 2.6.0 broke the build: [Reform] Your :populator did not return a Reform::Form instance for `authors`.
Expand Down
27 changes: 11 additions & 16 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
GIT
remote: https://github.com/sul-dlss/dlss-capistrano-docker.git
revision: 6db0ad1335507359f32b6a40ce8cd56e8be42be9
branch: t2-rollback
specs:
dlss-capistrano-docker (1.0.0)
capistrano (~> 3.0)
capistrano-one_time_key
capistrano-shared_configs

GEM
remote: https://rubygems.org/
specs:
Expand Down Expand Up @@ -80,9 +90,6 @@ GEM
bootsnap (1.13.0)
msgpack (~> 1.2)
builder (3.2.4)
bundler-audit (0.9.1)
bundler (>= 1.2.0, < 3)
thor (~> 1.0)
bunny (2.19.0)
amq-protocol (~> 2.3, >= 2.3.1)
sorted_set (~> 1, >= 1.0.2)
Expand All @@ -92,18 +99,12 @@ GEM
i18n
rake (>= 10.0.0)
sshkit (>= 1.9.0)
capistrano-bundle_audit (0.4.0)
bundler-audit (~> 0.5)
capistrano (~> 3.0)
capistrano-bundler (>= 1.4)
capistrano-bundler (2.1.0)
capistrano (~> 3.1)
capistrano-maintenance (1.2.1)
capistrano (>= 3.0)
capistrano-one_time_key (0.1.0)
capistrano (~> 3.0)
capistrano-passenger (0.2.1)
capistrano (~> 3.0)
capistrano-rails (1.6.2)
capistrano (~> 3.1)
capistrano-bundler (>= 1.1, < 3)
Expand Down Expand Up @@ -179,11 +180,6 @@ GEM
declarative-option (< 0.2.0)
representable (>= 2.4.0, <= 3.1.0)
uber (< 0.2.0)
dlss-capistrano (4.3.1)
capistrano (~> 3.0)
capistrano-bundle_audit (>= 0.3.0)
capistrano-one_time_key
capistrano-shared_configs
docile (1.4.0)
druid-tools (3.0.0)
dry-configurable (0.16.1)
Expand Down Expand Up @@ -548,7 +544,6 @@ DEPENDENCIES
bunny (~> 2.17)
byebug
capistrano-maintenance (~> 1.2)
capistrano-passenger
capistrano-rails
capybara (~> 3.34)
capybara-screenshot
Expand All @@ -559,7 +554,7 @@ DEPENDENCIES
cypress-rails
devise (~> 4.7)
devise-remote-user (~> 1.0)
dlss-capistrano
dlss-capistrano-docker!
druid-tools
dry-types
edtf
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,13 @@ Then run tests with `bundle exec rspec`. If you also want to do style checks & l

### Integration

Spin up all docker compose services for local development and in-browser testing:
Spin up all docker compose services for local testing:

```shell
$ docker compose up # use -d to run in background
```

This will spin up the H2 web application, its background workers, and all service dependencies declared in docker-compose.yml.
This will spin up the H2 web application, its background workers, and all service dependencies declared in docker-compose.yml. H2 will be running in production mode.

### Cypress
Cypress is primarily used to test features implemented with JS/Stimulus. Cypress tests are located in `cypress/spec`.
Expand Down
13 changes: 10 additions & 3 deletions app/controllers/application_controller.rb
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,21 @@ def groups_from_request_env
session['groups'] = begin
raw_header = request.env[Settings.authorization_group_header]
roles = ENV.fetch('ROLES', nil) # rubocop:disable Rails/EnvironmentVariableAccess
raw_header = roles if Rails.env.development?
# This test setting is for cypress.
raw_header = roles if Rails.env.test? && roles
raw_header = roles if use_roles?(roles)
logger.debug("Roles are #{raw_header}")
raw_header&.split(';') || []
end
end

def use_roles?(roles)
return true if Rails.env.development?
# This test setting is for cypress.
return true if Rails.env.test? && roles
return true if ENV.fetch('LOCAL_DOCKER', nil) && roles # rubocop:disable Rails/EnvironmentVariableAccess

false
end

def deny_access
flash[:warning] = 'You are not authorized to perform the requested action'
redirect_to :root
Expand Down
2 changes: 1 addition & 1 deletion app/jobs/assign_pid_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ class AssignPidJob
# This worker will connect to "h2.druid_assigned" queue
# env is set to nil since by default the actual queue name would be
# "h2.druid_assigned_development"
from_queue 'h2.druid_assigned', env: nil
from_queue Settings.rabbitmq.queues.druid_assigned, env: nil
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of these queue changes are so that we can run multiple instances of H2 in a single environment and each has its own queue. This will allow us to run a dockerized H2 and a plain-old H2 together.


def work(msg)
model = build_cocina_model_from_json_str(msg)
Expand Down
2 changes: 1 addition & 1 deletion app/jobs/deposit_complete_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ class DepositCompleteJob
# example, if the embargo was lifted, DSA would open and close a version. The
# workflow message "end-accession" would end up here. We must be able to handle
# these messages in addition to those that result from depositing in h2.
from_queue 'h2.deposit_complete', env: nil
from_queue Settings.rabbitmq.queues.deposit_complete, env: nil

def work(msg)
druid = parse_message(msg)
Expand Down
2 changes: 1 addition & 1 deletion app/jobs/record_embargo_release_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ class RecordEmbargoReleaseJob
# This worker will connect to "h2.embargo_lifted" queue
# env is set to nil since by default the actual queue name would be
# "h2.embargo_lifted_development"
from_queue 'h2.embargo_lifted', env: nil
from_queue Settings.rabbitmq.queues.embargo_lifted, env: nil

def work(msg)
model = build_cocina_model_from_json_str(msg)
Expand Down
29 changes: 7 additions & 22 deletions config/deploy.rb
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,11 @@
# Default value for :pty is false
# set :pty, true

# Default value for :linked_files is []
append :linked_files,
'config/database.yml', # in Puppet
'config/secrets.yml', # in shared_configs
'config/honeybadger.yml' # in shared_configs
set :linked_files, %w[config/honeybadger.yml]

# Default value for linked_dirs is []
append :linked_dirs, 'log', 'config/settings', 'tmp/pids', 'tmp/cache', 'tmp/sockets', 'public/system'
set :linked_dirs, %w[log config/settings public/system]

set :dereference_dirs, %w[config/settings]

# Default value for default_env is {}
# set :default_env, { path: "/opt/ruby/bin:$PATH" }
Expand All @@ -46,21 +43,9 @@
# honeybadger_env otherwise defaults to rails_env
set :honeybadger_env, fetch(:stage)

# Manage sidekiq via systemd (from dlss-capistrano gem)
set :sidekiq_systemd_use_hooks, true

# Manage sneakers via systemd (from dlss-capistrano gem)
set :sneakers_systemd_use_hooks, true

# Set Rails env to production in all Cap environments
set :rails_env, 'production'

# Deploy passenger-standalone via systemd service
set :passenger_restart_command, 'sudo systemctl restart passenger'
set :passenger_restart_options, -> { '' }

set :whenever_environment, fetch(:rails_env)
set :whenever_roles, [:cron]

# update shared_configs before restarting app (from dlss-capistrano gem)
before 'deploy:restart', 'shared_configs:update'
set :docker_compose_file, 'docker-compose.prod.yml'
set :docker_compose_seed_use_hooks, true
set :docker_compose_rabbitmq_use_hooks, true
3 changes: 2 additions & 1 deletion config/deploy/qa.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# frozen_string_literal: true

server 'sul-h2-qa.stanford.edu', user: 'h2', roles: %w[web app db]
# Roles are passed to docker-compose as profiles.
server 'h2-docker-qa.stanford.edu', user: 'h2', roles: %w[web app db cron worker]

Capistrano::OneTimeKey.generate_one_time_key!
1 change: 1 addition & 0 deletions config/environments/production.rb
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@

config.action_mailer.perform_caching = false
config.action_mailer.asset_host = "https://#{Settings.host}"
config.action_mailer.smtp_settings = { address: ENV.fetch('SMTP_HOST', 'localhost') }

# Ignore bad email addresses and do not raise email delivery errors.
# Set this to true and configure the email server for immediate delivery to raise delivery errors.
Expand Down
2 changes: 1 addition & 1 deletion config/initializers/devise_remote_user.rb
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
DeviseRemoteUser.configure do |config|
config.env_key = lambda do |env|
remote_user = ENV.fetch('REMOTE_USER', nil)
if (Rails.env.development? || Rails.env.test?) && remote_user
if (Rails.env.development? || Rails.env.test? || ENV.fetch('LOCAL_DOCKER', false)) && remote_user
remote_user
else
# Return the first non-blank value of a remote user header, or return nil (unauthenticated)
Expand Down
8 changes: 4 additions & 4 deletions config/puma.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum; this matches the default thread size of Active Record.
#
max_threads_count = ENV.fetch('RAILS_MAX_THREADS', 5)
min_threads_count = ENV.fetch('RAILS_MIN_THREADS') { max_threads_count }
max_threads_count = ENV.fetch('PUMA_MAX_THREADS', 5)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing these variables so that their naming is clearer elsewhere (e.g., in puppet files).

min_threads_count = ENV.fetch('PUMA_MIN_THREADS') { max_threads_count }
threads min_threads_count, max_threads_count

# Specifies the `port` that Puma will listen on to receive requests; default is 3000.
Expand All @@ -27,14 +27,14 @@
# Workers do not work on JRuby or Windows (both of which do not support
# processes).
#
# workers ENV.fetch("WEB_CONCURRENCY") { 2 }
workers ENV.fetch('PUMA_WORKERS', 2)

# Use the `preload_app!` method when specifying a `workers` number.
# This directive tells Puma to first boot the application and load code
# before forking the application. This takes advantage of Copy On Write
# process behavior so workers use less memory.
#
# preload_app!
preload_app!

# Allow puma to be restarted by `rails restart` command.
plugin :tmp_restart
3 changes: 3 additions & 0 deletions config/schedule.rb
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
# Use this file to easily define all of your cron jobs.
# Learn more: http://github.com/javan/whenever

# Execute without bash.
set :job_template, nil

every :day, at: '1:00am' do
runner 'WorkReminderGenerator.send_draft_reminders'
end
Expand Down
6 changes: 5 additions & 1 deletion config/settings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,14 +58,18 @@ purl_url: https://purl.stanford.edu
terms_url: https://library.stanford.edu/research/stanford-digital-repository/documentation/sdr-terms-deposit
sdr_url: https://library.stanford.edu/research/stanford-digital-repository

host: <%= Socket.gethostname %>
host: <%= ENV.fetch('HOST', Socket.gethostname) %>

rabbitmq:
enabled: false
hostname: localhost
vhost: /
username: guest
password: guest
queues:
deposit_complete: 'h2.deposit_complete'
druid_assigned: 'h2.druid_assigned'
embargo_lifted: 'h2.embargo_lifted'

notifications:
first_draft_reminder:
Expand Down
2 changes: 1 addition & 1 deletion db/seeds.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@
# Character.create(name: 'Luke', movie: movies.first)

# The SDR user is used in Events performed by SDR.
User.create!(name: 'SDR', email: 'sdr@stanford.edu')
User.find_or_create_by!(name: 'SDR', email: 'sdr@stanford.edu')
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allows seeding as part of deploy. Currently, it was performed as a manual operation.

Loading