Memory leak

[mohammadjavad948]

Hello i am currently facing memory leak in my axum app. when i start the app my memory usage is ~600KB and with every request it goes up by ~200KB and it never stops.

here is my handler function:

use std::{env, time::SystemTime};

use axum::{extract::Multipart, http::StatusCode, Extension};
use shared_utils::auth::UserInfo;
use tokio::{fs::File, io::AsyncWriteExt};

pub async fn accept_form(
    mut multipart: Multipart,
    Extension(user): Extension<UserInfo>,
) -> Result<String, StatusCode> {
    let mut files = vec![];
    let mut paths = vec![];
    let destination = env::var("UPLOAD_DEST").expect("expected `UPLOAD_DEST` env variable");

    while let Some(field) = multipart
        .next_field()
        .await
        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?
    {
        let name = field
            .name()
            .ok_or(StatusCode::INTERNAL_SERVER_ERROR)?
            .to_string();

        let content_type = field
            .content_type()
            .ok_or(StatusCode::INTERNAL_SERVER_ERROR)?
            .to_string();

        if name == "file" && content_type.starts_with("image/") {
            let data = field
                .bytes()
                .await
                .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;

            files.push(data);
        }
    }

    if files.len() < 4 || files.len() > 8 {
        return Err(StatusCode::NOT_ACCEPTABLE);
    }

    for (index, file) in files.iter().enumerate() {
        let time = SystemTime::now()
            .duration_since(SystemTime::UNIX_EPOCH)
            .unwrap()
            .as_millis();

        let location = format!("{}-{}-{}.jpeg", time, index, user.user_id);

        let path = std::path::Path::new(&destination).join(&location);

        let mut f = File::create(&path).await.unwrap();

        f.write_all(file).await.unwrap();
        paths.push(location);
    }

    Ok(paths.join("|"))
}

also i have a middleware for user auth:

pub async fn auth_middleware<B>(
    mut req: Request<B>,
    next: Next<B>,
) -> Result<Response, StatusCode> {
    // get header
    let auth_header = req
        .headers()
        .get("auth")
        .and_then(|header| header.to_str().ok())
        .ok_or(StatusCode::UNAUTHORIZED)?;

    let secret = std::env::var("SECRET").map_err(|_| StatusCode::UNAUTHORIZED)?;

    let mut validation = Validation::new(Algorithm::HS256);
    validation.validate_exp = false;
    validation.validate_nbf = false;
    validation.required_spec_claims = HashSet::new();

    // decode token
    let token_message = decode::<AuthHeader>(
        auth_header,
        &DecodingKey::from_secret(secret.as_ref()),
        &validation,
    )
    .map_err(|_| StatusCode::UNAUTHORIZED)?;

    let mut hash = Sha256::new();
    hash.update(auth_header.as_bytes());

    let hash_result = hash.finalize();

    let db: &Arc<DatabaseConnection> = req
        .extensions()
        .get()
        .ok_or(StatusCode::INTERNAL_SERVER_ERROR)?;

    let session = get_user_session(
        db.clone(),
        token_message.claims.id,
        format!("{:x}", hash_result),
    )
    .await?
    .ok_or(StatusCode::UNAUTHORIZED)?;

    let user = user::Entity::find_by_id(session.user_id)
        .one(db.as_ref())
        .await
        .map_err(|_| StatusCode::UNAUTHORIZED)?
        .ok_or(StatusCode::UNAUTHORIZED)?;

    let user = UserInfo {
        user_id: session.user_id,
        session_id: session.id,
        role: UserRole::get_enum_eqivalent(&user.role).unwrap(),
    };

    req.extensions_mut().insert(user);

    Ok(next.run(req).await)
}

i am confused and i don`t know what to do to fix it :( .

[davidpdrsn]

Have you tested it with something like valgrind to know that it is actually a leak? Could also just be allocator not giving memory back to the OS immediately? For how long did you run your app?

[fggarcia]

@mohammadjavad948 can you try to use jemalloc-tv . I had the same problem with actix-web. After i change to this, i never saw the problem again. As @davidpdrsn said maybe the default allocator not give memory back to the os.

[mohammadjavad948]

@fggarcia i tried using it but it didn't change anything

[mohammadjavad948]

@fggarcia can you show the code you used for it?

[fggarcia]

@mohammadjavad948

crates dependencies:

use main.rs:

[mohammadjavad948]

@fggarcia nope it didn't help.

[lab360-ch]

Hello

I have the same experience. The memory in increasing by every request.
I also see the same behavior within the hello world example.

When I start the https://github.com/tokio-rs/axum/tree/main/examples/hello-world example and hit the endpoint with ab -n 20000 -c 30 http://localhost:3000/ I see the memory increasing but never decreasing.

Any idea how to address this problem?

[davidpdrsn]

See this comment #1546 (comment)

[lab360-ch]

I did a lot of testing and found out that's not direct a memory leak. When I use valgrind no related memory leak was detected. But still with every request the memory is increasing.

It looks more likely that there are futures that don't get resolved correctly.

I used this hello world code:

use std::convert::Infallible;

use hyper::service::{make_service_fn, service_fn};
use hyper::{Body, Request, Response, Server};

async fn hello(_: Request<Body>) -> Result<Response<Body>, Infallible> {
    Ok(Response::new(Body::from("Hello World!")))
}

#[tokio::main]
pub async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    // For every connection, we must make a `Service` to handle all
    // incoming HTTP requests on said connection.
    let make_svc = make_service_fn(|_conn| {
        // This is the `Service` that will handle the connection.
        // `service_fn` is a helper to convert a function that
        // returns a Response into a `Service`.
        async { Ok::<_, Infallible>(service_fn(hello)) }
    });

    console_subscriber::init();

    let addr = ([127, 0, 0, 1], 3000).into();

    let server = Server::bind(&addr).serve(make_svc);

    println!("Listening on http://{}", addr);

    server.await?;

    Ok(())
}

with the tokio-console and get this result:

After start without any request:

after 15 requests with curl http://localhost:3000/ the console looks like this:

And this stays like this until the program gets restarted.

Is this a correct behavior, or should the task disappear after the request?

For me, It looks like this tasks holding the memory and never release them.

[davidpdrsn]

I don't know. Try asking in the hyper repo.

[mohammadjavad948]

i deployed my service to production server and it was fine. after we decreased our ram from 4GB to 1GB, its ram usage was not growing. so i think its not a problem with axum or hyper but it is because the memory that is deallocated does not go to OS instantly but it goes back if OS needs it.

[fundon]

    let mut files = vec![];
    let mut paths = vec![];

I think you should not go to hold data. If you want to check how many files are uploaded, you can count them on the front-end and add a field to put it in the first position.

[mohammadjavad948]

I changed the code so It received data from a stream and counted files at the end but the problem persist

[fundon]

Try to flush or shutdown the file stream. https://docs.rs/tokio/latest/tokio/io/trait.AsyncWriteExt.html#method.flush

[gitmalong]

I'm facing a memory leak as well. Measured by the memory stats of Kubernetes dashboard which shows the available memory.

[g-mero]

Same issue on actix-web and salvo, according to the post https://rustcc.cn/article?id=42f6ad67-55d6-4377-a6f6-f97e6dcc0d0e