Skip to content

Commit

Permalink
Suggestions for increased transparency
Browse files Browse the repository at this point in the history
This PR suggests some text changes to the copy, and adds back a chronology.

Because our site is used by:

1. Auditors and auditees submitting audits
2. State auditors (and other entities) responsible for passthrough monitoring
3. Federal audit resolution officials
4. IG offices
5. Grants managers/grants issuing offices
6. GAO
8. OMB
9. The general public
10. ...

I think credibility is enhanced by transparency, and because we have a
public record of our efforts in this space (our Github repository), I
think we should use it to document the actions we've taken so far.

For consideration.
  • Loading branch information
jadudm committed Mar 19, 2024
1 parent f78c551 commit 8005b1b
Showing 1 changed file with 17 additions and 3 deletions.
20 changes: 17 additions & 3 deletions src/status.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,21 @@ meta:
**Audit search is currently limited to UEI-only.** You can search your UEI to confirm your audit submission is complete. The PDF report will still be available for download but summary report workbooks won't.

## Issue background
In mid-February, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system exponentially and slowed down search performance.
Our team has released multiple patches to stabilize search performance since then but have been unable to do so consistently. For this reason, we've decided to reduce search to UEI-only while we focus on building new search tables instead of one-off bug fixes.

We're working as quickly and efficiently as possible to return full search functionality to our users as soon as we can. Thank you for your patience.
On February 7th, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system more than twenty-fold, drastically impairing search performance.

Our team has taken incremental, iterative steps to improve search performance since then but have been unable to do so in a manner that addresses the multiple challenges at hand. For this reason, we've decided to reduce search to UEI-only while we focus on building the infrastructure necessary to support a more sustainable search experience.

We're working as quickly as possible to return full search functionality to our users as soon as we can. We recognize the significant disruption this represents to users with oversight responsibilities, and will provide daily updates here until this issue is resolved.

A brief history of our efforts:

1. February 7th, data from Census is completely loaded, increasing the number of audits from 10K to more than 250K, and the number of awards tracked from 100K to nearly 4.5M.
2. February 8th, [resources increased in production](https://github.com/GSA-TTS/FAC/pull/3376) to address load.
3. February 9th, [My ALN](https://github.com/GSA-TTS/FAC/pull/3378) functionality is removed in an attempt to address crashing behaviors stemming from this feature.
4. February 14th, worked with more than a dozen NSACs to [prioritize search needs](https://github.com/GSA-TTS/FAC/issues/3388).
5. February 16th, [increased search capacity in production](https://github.com/GSA-TTS/FAC/pull/3424).
6. February 28th, [added prioritized search needs](https://github.com/GSA-TTS/FAC/pull/3440), re-introducing performance challenges.
7. March 11th, heavy system load prevents system updates for nearly a week. [Post-workday update to increase resources in production](https://github.com/GSA-TTS/FAC/pull/3496) carried out, partially addressing new search performance challenges.
8. March 14th, performance issues and crashing persist in search. [New search tables and queries introduced](https://github.com/GSA-TTS/FAC/pull/3511). Changes require significant engineering and testing work.
9. March 18th, decision to take down search made by team.

0 comments on commit 8005b1b

Please sign in to comment.