JSON Deserialization Causes Memory Overflow and Service Restart

[zhuquanzhen]

Search before asking

• I searched in the issues and found nothing similar.

Describe the bug

When a JSON character string with many empty nodes is deserialized to a list, (com.fasterxml.jackson.databind.ObjectMapper#readValue(java.lang.String, java.lang.Class)) generates a large number of empty objects, occupying a large amount of memory. Attackers can easily construct attack scenarios and restart services. Does jackson-databind provide a good protection method for this scenario?

Version Information

2.13.5

Reproduction

<-- Any of the following

1. Brief code sample/snippet: include here in preformatted/code section
2. Longer example stored somewhere else (diff repo, snippet), add a link
3. Textual explanation: include here
   -->

// Your code here

Expected behavior

No response

Additional context

No response

[pjfanning]

250Mb isn't very much memory.
If you allow anyone to send you malicious content and if you don't make any attempt to protect your services by checking the input before parsing it, then you have a problem. You should be using sandboxed environments and have an architecture that can recover from failures.

Newer versions of Jackson have configurable limits - called StreamReadConstraints and these have defaults that should help - but you are advised to tune these constraint values for your own use cases.

So try out jackson 2.16.1. Try playing around with lower limits than the defaults.

https://www.javadoc.io/static/com.fasterxml.jackson.core/jackson-core/2.16.0/com/fasterxml/jackson/core/StreamReadConstraints.html

You might also want to write a solution that streams through the JSON instead of binding the whole document. Search around the web for examples but this has a brief write up - https://nurkiewicz.com/2017/09/streaming-large-json-file-with-jackson.html

[zhuquanzhen]

Thank you so much for getting back to me. But I still have a problem. When the JSON string is deserialized, the same string references are different, and a new object is created each time. This will take up a lot of memory. Why do you need to create new objects for Strings? Why not use the same object if they are the same string? This will save a lot of memory.

[WDXDenys]

I also encountered the same problem, but it is much larger than 250M, as shown in the figure, with 5.7G. How should I limit the modification

[cowtowncoder]

 Why do you need to create new objects for Strings? Why not use the same object if they are the same string? This will save a lot of memory.

Finding out whether Strings are the same requires keeping track of already seen Strings, looking up matches: technique called as "de-duplication", "canonicalization" etc. This is possible to do but adds significant processing overhead, both CPU and memory usage. This is not generally beneficial for most use cases so it is not done by default by Jackson, for String values.
De-duplication is done for Object property names however.

It would be possible to offer possibility of also de-duping String values, but the challenge is that of how to allow enabling it only for some properties and not just "all or nothing" (either do for everything, or do for nothing).
Such functionality does not exist yet.

But it is already possible to make JDK handle de-duping by adding:

 -XX:+UseStringDeduplication

when starting JVM (see https://medium.com/codex/string-deduplication-in-java-2dc1e1893a74 f.ex)

[cowtowncoder]

It'd be good to have actual code, class(es) involved.

But in general, if you read the contents and bind to a class, there is some memory usage involved.
Amount depends on classes used and so forth.

If you want to limit sizes, you will probably need to use combination of input limits that @pjfanning alluded to, as well as design your target object class defitions to limit sizes accepted (like max array or List lengths on setters).

[cowtowncoder]

Currently there is no de-duping of String values since for most usage there is little benefit in trying to de-duplicate values and trying to do so adds overhead. Adding such a feature would be a possibility, the main challenge being that of how to enable it only for specific properties and not all String values.

But JDK can handle de-duplication for you too:

https://medium.com/codex/string-deduplication-in-java-2dc1e1893a74

and this might be something for you to check.

This would only help with String duplicates: if you wanted to avoid duplicate List<String>s, you would need to do something else: you could -- for example -- add handling in setNameList(): even if Jackson creates new instances, you could implement simple LRU lookup cache to remove duplicates. That is some work of course, but it is also something that I don't think Jackson can add support for, in general case.

[zhuquanzhen]

Thank you very much, you are very reasonable, my idea is also to limit the size of the list, prevent memory overflow scenarios. I think jackson could have some annotations to limit the size of lists, maps, etc. sets so that developers can control their size with simple annotations when deserializing. This also improves jackson's ease of use. Of course, it's just my own advice.

[cowtowncoder]

There are multiple levels at which limits could apply, with or without annotations.
The problem I see wrt limitations aimed at Maps and Collections is that there are many potential types to consider, and it'd be game of whack-a-mole to try to address all possible types, adding complexity in implementation.

So it might be better to consider limitations at streaming processing, to limit size of JSON Object and Array values (wrt properties/elements). This would be limiting all value types which can be both good and bad, depending on requirements.

[pjfanning]

I think a new max array size in StreamReadConstraints might be useful. Supporting interning of String values could also be an option to avoid the need for a JVM setting. Having lots of the same String value in a doc looks like a sneaky exploit. Or even lots of short Strings.

[cowtowncoder]

I think that StreamReadConstraints for max Array and Object size (latter wrt huge Maps or ObjectNodes) would be make sense; it would be conceptually similar to max String and Number value length constraints.

OTOH I don't think use of lots of same Strings is much of an attack; one may just use slight variations to have lots of slightly different Strings.
But perhaps there is some value in allowing a setting for auto-canonicalizing JSON String values, similar to how keys are ... then again, it probably should not use same symbol tables as those are optimized for shorter Strings.
Regardless, that'd be different new feature.

[zaiji]

I ran into a similar problem where I couldn't limit the number of nodes for map objects when deserializing them to map objects. Is there any good solution?

Such as:
(Map)objectMapper.readValue(jsonStr, new TypeReference<Map<String, Object>>(){});

[cowtowncoder]

There are no settings to specify maximum sizes for Java Objects.

I would probably implement my own Map type (sub-class of HashMap, for example), where put() method could impose maximum limit and throw exception if limit is exceeded.
As long as your custom Map class has 0-argument constructor, Jackson should be able to use it as target.

Furthermore, it is possible to also specify "abstract type mapping" to register your own Map and List implementations.
To do that, you need to implement AbstractTypeResolver (or use default SimpleAbstractTypeResolver).
Unit test TypeResolverTest has following:

    static class MyMap<K,V> extends HashMap<K,V> { }

    public static void testSubtypeResolution() throws Exception
    {
        ObjectMapper mapper = new ObjectMapper();
        SimpleAbstractTypeResolver resolver = new SimpleAbstractTypeResolver();
        resolver.addMapping(Map.class, MyMap.class);

        SimpleModule basicModule = new SimpleModule();
        basicModule.setAbstractTypes(resolver);
        mapper.registerModule(basicModule);

       // Now anything declared as `Map` will be deserialized as `MyMap` instead
    }

I hope this help.