Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(api_service): exclude health checks from throttling with ConcurrencyLimitLayer #2320

Merged
merged 7 commits into from
Oct 10, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/).

## [Unreleased]

## Fixed
- [2320](https://github.com/FuelLabs/fuel-core/issues/2320): Prevent `/health` from being throttled by the concurrency limiter.
rymnc marked this conversation as resolved.
Show resolved Hide resolved

## [Version 0.38.0]

### Added
Expand Down
14 changes: 10 additions & 4 deletions crates/fuel-core/src/graphql_api/api_service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -221,17 +221,23 @@ where
.extension(ViewExtension::new())
.finish();

let router = Router::new()
let unthrottled_routes = Router::new()
.route("/v1/health", get(health))
.route("/health", get(health));

let throttled_routes = Router::new()
.route("/v1/playground", get(graphql_playground))
.route("/v1/graphql", post(graphql_handler).options(ok))
.route(
"/v1/graphql-sub",
post(graphql_subscription_handler).options(ok),
)
.route("/v1/metrics", get(metrics))
.route("/v1/health", get(health))
.route("/health", get(health))
.layer(ConcurrencyLimitLayer::new(concurrency_limit))
.layer(ConcurrencyLimitLayer::new(concurrency_limit));
Copy link
Collaborator

@xgreenx xgreenx Oct 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want something like. Because graph-sub also is limited by number of subscriptions, and playground is one time query that returns pre rendered html. Throteling metrics doesn't allow request data=)

image

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Throteling metrics doesn't allow request data=)
are you saying that because metrics are throttled, it affects how graphql and graphql-sub are throttled as well?

why shouldn't we throttle the metrics?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because if we have a lot of queries, it becomes impossible for us to track the state of the node from the Grafana=)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, making the change

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 1a66422


let router = Router::new()
.merge(unthrottled_routes)
.merge(throttled_routes)
.layer(Extension(schema))
.layer(TraceLayer::new_for_http())
.layer(TimeoutLayer::new(request_timeout))
Expand Down
24 changes: 20 additions & 4 deletions tests/tests/dos.rs
Original file line number Diff line number Diff line change
Expand Up @@ -385,15 +385,12 @@ async fn complex_queries__2_state_transition_bytecode__query_to_complex() {
}

#[tokio::test]
async fn concurrency_limit_0_prevents_any_queries() {
async fn concurrency_limit_0_prevents_any_throttled_queries() {
// Given
let mut config = Config::local_node();
config.graphql_config.max_concurrent_queries = 0;
config.graphql_config.api_request_timeout = std::time::Duration::from_millis(100);

let query = FULL_BLOCK_QUERY.to_string();
let query = query.replace("$NUMBER_OF_BLOCKS", "40");

let node = FuelService::new_node(config).await.unwrap();
let url = format!("http://{}/v1/graphql", node.bound_address);
let client = reqwest::Client::new();
Expand All @@ -412,6 +409,25 @@ async fn concurrency_limit_0_prevents_any_queries() {
assert_eq!(result.status(), 408);
}

#[tokio::test]
async fn concurrency_limit_0_allows_unthrottled_queries() {
// Given
let mut config = Config::local_node();
// perhaps this shouldn't be called `graphql_config` but `api_config` or something
config.graphql_config.max_concurrent_queries = 0;

let node = FuelService::new_node(config).await.unwrap();
let url = format!("http://{}/v1/health", node.bound_address);
let client = reqwest::Client::new();

// When
let response = client.get(url).send().await;

// Then
let result = response.unwrap();
assert_eq!(result.status(), 200);
}

#[tokio::test(flavor = "multi_thread", worker_threads = 8)]
async fn concurrency_limit_1_prevents_concurrent_queries() {
// Given
Expand Down
Loading