Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: automate resetting dev deploy (resolves #1902, #1903) #1935

Merged
merged 23 commits into from
Nov 14, 2023
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
91583ba
feat(command): add schedule to run devSeeder daily at midnight on dev
peterhebert Oct 4, 2023
4eea0be
feat(command): updated seeds disk to be s3
peterhebert Oct 4, 2023
3f51a13
fix(seeder): fixed s3 config for seeder storage
marvinroman Oct 5, 2023
a4977d9
fix(command): :bug: changed from --env because it changes APP_ENV res…
marvinroman Oct 5, 2023
3ebebc7
feat(command): :sparkles: updated crons for backing up and restoring …
marvinroman Oct 5, 2023
2bc4a2e
docs: :memo: updated docs for db:seed:backup command
marvinroman Oct 5, 2023
1c4cf2c
feat(command): run dev seeder restore at 12:30am
peterhebert Oct 12, 2023
42f0750
chore(command): changed time that dev seeder will run 15m after backup
marvinroman Oct 13, 2023
1d558e9
chore: updated comments for where the migrate:fresh runs
marvinroman Oct 16, 2023
aeb969e
chore(command): explicity set timezone to PST for scheduled tasks
marvinroman Oct 17, 2023
2ae2b08
chore(command): explicity set timezone to PST for scheduled tasks
marvinroman Oct 17, 2023
39fe449
chore: merge 1903 into 1902
marvinroman Oct 26, 2023
3b95fa8
refactor: migrated production to dev seeder to the DevRefresh command
marvinroman Oct 26, 2023
a535091
Merge branch 'dev' into 1902-automate-resetting-dev-deploy
marvinroman Oct 26, 2023
2cd6aa0
chore(command): limit app:refresh-dev to run only on dev
marvinroman Oct 26, 2023
37bf11b
chore: mock s3 disk if it fails to connect
marvinroman Oct 26, 2023
b89ab47
chore(command): ensure the command cannot be run on production or tes…
marvinroman Oct 26, 2023
184cea1
Merge branch 'dev' into 1902-automate-resetting-dev-deploy
marvinroman Oct 26, 2023
e80659e
refactor: testing changing seeds to local when there aren't s3
marvinroman Oct 31, 2023
d8fb37c
refactor: finalized swap to seeds being local driver when MINIO_DRIVE…
marvinroman Oct 31, 2023
5d4a011
Merge branch 'dev' into 1902-automate-resetting-dev-deploy
jobara Nov 14, 2023
6ef1407
fix: typo
jobara Nov 14, 2023
1aee83c
chore: clean up comments
jobara Nov 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@ Takes filament tables and backs them up to JSON files so that they can be used b

| option | Description |
| --- | ---- |
| `--env` | Override the environment tag that is being handled. |
| `--from` | Override the environment tag that is being handled. |

* Available environments are found in config **backup.filament_seeders.environments**.
* When used by default backup it will tag the json files with environment tag.
Expand Down
35 changes: 35 additions & 0 deletions app/Console/Commands/RefreshDev.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;

class RefreshDev extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'app:refresh-dev';

/**
* The console command description.
*
* @var string
*/
protected $description = 'Runs a development database refresh.';

/**
* Execute the console command.
*/
public function handle()
{
$this->call('down');
if (in_array(config('app.env'), ['dev']) !== true) {
$this->call('migrate:fresh --seeder=DevSeeder');
}
marvinroman marked this conversation as resolved.
Show resolved Hide resolved
$this->call('db:seed:backup --all --restore --from=production');
$this->call('up');
}
}
10 changes: 5 additions & 5 deletions app/Console/Commands/SeederBackupData.php
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ class SeederBackupData extends Command
*/
protected $signature = 'db:seed:backup
{--a|all : Whether to run through all available backups/restores in config}?
{--env= : Specify an environment configured in backup.filament_seeders.environments}?
{--from= : Specify an environment configured in backup.filament_seeders.environments}?
{--remove : Remove backed up files}?
{--restore : Restore the filament table}?
{--t|table=* : Create/remove specific table file}?
Expand Down Expand Up @@ -54,7 +54,7 @@ public function handle()
foreach ($options['table'] as $table) {
if (in_array($table, $available_tables)) {
printf("Deleting table: %s, for environment: %s\r\n", $table, $environment);
Storage::disk('seeds')->delete(sprintf('%s.%s.json', $table, $environment));
Storage::disk('seeds')->delete(sprintf('%s/%s.%s.json', config('filesystems.disks.seeds.path'), $table, $environment));
} else {
printf("You have might have misspelled the tablename.\r\nTable %s not found.\r\nAvailable tables: %s\r\n", $table, implode(', ', $available_tables));

Expand All @@ -66,7 +66,7 @@ public function handle()
} else {
foreach ($available_tables as $table) {
printf("Deleting table: %s, for environment: %s\r\n", $table, $environment);
Storage::disk('seeds')->delete(sprintf('%s.%s.json', $table, $environment));
Storage::disk('seeds')->delete(sprintf('%s/%s.%s.json', config('filesystems.disks.seeds.path'), $table, $environment));
}
}

Expand Down Expand Up @@ -115,7 +115,7 @@ public function handle()
} elseif ($options['all']) {
foreach ($available_tables as $table) {
printf("Backing up table seeder: %s for environment: %s\r\n", $table, $environment);
Storage::disk('seeds')->put(sprintf('%s.%s.json', $table, $environment), DB::table($table)->get()->toJson());
Storage::disk('seeds')->put(sprintf('%s/%s.%s.json', config('filesystems.disks.seeds.path'), $table, $environment), DB::table($table)->get()->toJson());
}

return 0;
Expand All @@ -125,7 +125,7 @@ public function handle()
foreach ($options['table'] as $table) {
if (in_array($table, $available_tables)) {
printf("Backing up table seeder %s\r\n", $table);
Storage::disk('seeds')->put(sprintf('%s.%s.json', $table, $environment), DB::table($table)->get()->toJson());
Storage::disk('seeds')->put(sprintf('%s/%s.%s.json', config('filesystems.disks.seeds.path'), $table, $environment), DB::table($table)->get()->toJson());
} else {
printf("You have might have misspelled the tablename.\r\nTable %s not found.\r\nAvailable tables: %s\r\n", $table, implode(', ', $available_tables));

Expand Down
8 changes: 8 additions & 0 deletions app/Console/Kernel.php
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,19 @@ protected function schedule(Schedule $schedule)
{
$schedule->command('db:seed:backup --all') // use custom command to make sure that the commands are chained
->daily() // Run daily at midnight
->environments(['staging', 'dev', 'local', 'production']) // only run for APP_ENV tagged staging, dev, local, or production
->timezone('America/Los_Angeles') // Run as PST timezone
->onOneServer(); // run only on a single server at once

$schedule->command('app:refresh-dev') // use custom hcommand to make sure that te commands are chained
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo "hcommand"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marvinroman this typo is still present.

->dailyAt('00:15') // Run daily at 12:15 am
marvinroman marked this conversation as resolved.
Show resolved Hide resolved
->environments(['staging', 'dev', 'local']) // only run for APP_ENV tagged staging, dev, or local
->timezone('America/Los_Angeles') // Run as PST timezone
->onOneServer(); // run only on a single server at once
marvinroman marked this conversation as resolved.
Show resolved Hide resolved

$schedule->command('notifications:remove:old --days=30') // remove notifications older than 30 days old and read
->daily() // Run daily at midnight
->timezone('America/Los_Angeles') // Run as PST timezone
->onOneServer(); // run only on a single server at once
}

Expand Down
10 changes: 8 additions & 2 deletions config/filesystems.php
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,14 @@
],

'seeds' => [
'driver' => 'local',
'root' => storage_path('app/seeds'),
'driver' => 's3',
'path' => 'seeds',
'key' => env('MINIO_ACCESS_KEY'),
'secret' => env('MINIO_SECRET_KEY'),
'region' => env('MINIO_REGION'),
'bucket' => env('MINIO_PROJECT_BUCKET'),
'use_path_style_endpoint' => true,
'endpoint' => 'https://'.env('MINIO_ENDPOINT'),
],

'public' => [
Expand Down
4 changes: 2 additions & 2 deletions database/seeders/IdentitySeeder.php
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@ public function run()
$environment = config('app.env');
}

if (Storage::disk('seeds')->exists(sprintf('identities.%s.json', $environment))) {
if (Storage::disk('seeds')->has(sprintf('%s/identities.%s.json', config('filesystems.disks.seeds.path'), $environment))) {
// if trucate was set via seeder restore command then truncate the table prior to seeding data
if (config('seeder.truncate')) {
DB::statement('SET foreign_key_checks=0');
Identity::truncate();
DB::statement('SET foreign_key_checks=1');
}

$identities = json_decode(Storage::disk('seeds')->get(sprintf('identities.%s.json', $environment)), true);
$identities = json_decode(Storage::disk('seeds')->get(sprintf('%s/identities.%s.json', config('filesystems.disks.seeds.path'), $environment)), true);

foreach ($identities as $identity) {
Identity::firstOrCreate([
Expand Down
3 changes: 1 addition & 2 deletions database/seeders/InterpretationSeeder.php
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,7 @@ public function run()
}

if (Storage::disk('seeds')->exists(sprintf('interpretations.%s.json', $environment))) {
$interpretations = json_decode(Storage::disk('seeds')->get(sprintf('interpretations.%s.json', $environment)), true);

$interpretations = json_decode(Storage::disk('seeds')->get(sprintf('%s/interpretations.%s.json', config('filesystems.disks.seeds.path'), $environment)), true);
foreach ($interpretations as $interpretation) {
Interpretation::firstOrCreate([
'name' => $interpretation['name'],
Expand Down
4 changes: 2 additions & 2 deletions database/seeders/ResourceCollectionSeeder.php
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,15 @@ public function run()
}

// TODO need to write handling of attachments
if (false && Storage::disk('seeds')->exists(sprintf('resource_collections.%s.json', $environment))) {
if (false && Storage::disk('seeds')->exists(sprintf('%s/resource_collections.%s.json', config('filesystems.disks.seeds.path'), $environment))) {
// if trucate was set via seeder restore command then truncate the table prior to seeding data
if (config('seeder.truncate')) {
DB::statement('SET foreign_key_checks=0');
ResourceCollection::truncate();
DB::statement('SET foreign_key_checks=1');
}

$resourceCollections = json_decode(Storage::disk('seeds')->get(sprintf('resource_collections.%s.json', $environment)), true);
$resourceCollections = json_decode(Storage::disk('seeds')->get(sprintf('%s/resource_collections.%s.json', config('filesystems.disks.seeds.path'), $environment)), true);

foreach ($resourceCollections as $resourceCollection) {
ResourceCollection::firstOrCreate([
Expand Down
4 changes: 2 additions & 2 deletions database/seeders/ResourceSeeder.php
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,15 @@ public function run()
}

// TODO need to write handling of attachments
if (false && Storage::disk('seeds')->exists(sprintf('resources.%s.json', $environment))) {
if (false && Storage::disk('seeds')->exists(sprintf('%s/resources.%s.json', config('filesystems.disks.seeds.path'), $environment))) {
// if trucate was set via seeder restore command then truncate the table prior to seeding data
marvinroman marked this conversation as resolved.
Show resolved Hide resolved
if (config('seeder.truncate')) {
DB::statement('SET foreign_key_checks=0');
Resource::truncate();
DB::statement('SET foreign_key_checks=1');
}

$resources = json_decode(Storage::disk('seeds')->get(sprintf('resources.%s.json', $environment)), true);
$resources = json_decode(Storage::disk('seeds')->get(sprintf('%s/resources.%s.json', config('filesystems.disks.seeds.path'), $environment)), true);

foreach ($resources as $resource) {
if ($resource['content_type_id']) {
Expand Down
4 changes: 2 additions & 2 deletions database/seeders/TopicSeeder.php
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,15 @@ public function run()
$environment = config('app.env');
}

if (Storage::disk('seeds')->exists(sprintf('topics.%s.json', $environment))) {
if (Storage::disk('seeds')->exists(sprintf('%s/topics.%s.json', config('filesystems.disks.seeds.path'), $environment))) {
// if trucate was set via seeder restore command then truncate the table prior to seeding data
if (config('seeder.truncate')) {
DB::statement('SET foreign_key_checks=0');
Topic::truncate();
DB::statement('SET foreign_key_checks=1');
}

$topics = json_decode(Storage::disk('seeds')->get(sprintf('topics.%s.json', $environment)), true);
$topics = json_decode(Storage::disk('seeds')->get(sprintf('%s/topics.%s.json', config('filesystems.disks.seeds.path'), $environment)), true);

foreach ($topics as $topic) {
Topic::firstOrCreate([
Expand Down
Loading