Intermintent failures due to Solr timeouts

[amercader]

Looks like when running the full test suite linearly there's a high chance that the Solr container will timeout, making one or two tests fail:

ckan/plugins/core.py:267: in unload_non_system_plugins
    unload(*plugins_to_unload)
ckan/plugins/core.py:224: in unload
    plugins_update()
ckan/plugins/core.py:136: in plugins_update
    environment.update_config()
ckan/config/environment.py:178: in update_config
    search.check_solr_schema_version()
ckan/lib/search/__init__.py:363: in check_solr_schema_version
    res = _get_schema_from_solr(SOLR_SCHEMA_FILE_OFFSET_MANAGED)
ckan/lib/search/__init__.py:324: in _get_schema_from_solr
    response = requests.get(url, timeout=timeout)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/requests/api.py:73: in get
    return request("get", url, params=params, **kwargs)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/requests/api.py:59: in request
    return session.request(method=method, url=url, **kwargs)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)

[...]



                 # This branch is for urllib3 versions earlier than v1.22
                raise SSLError(e, request=request)
            elif isinstance(e, ReadTimeoutError):
>               raise ReadTimeout(e, request=request)
E               requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8983): Read timed out. (read timeout=5)

[...]

 =========================== short test summary info ============================
ERROR ckan/tests/controllers/test_package.py::TestPackageNew::test_after_create_plugin_hook - requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8983): Read timed out. (read timeout=5)
ERROR ckanext/example_theme_docs/custom_emails/test_custom_emails.py::TestExampleCustomEmailsPlugin::test_invite_user_custom_body - requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8983): Read timed out. (read timeout=5)

Options:

• Investigate if we can make the solr container more reliable (more memory?)
• Investigate if we can split tests across jobs like we do in circleci