Redirect support

[java4devgmail]

Hello

After working several years on java based CMS systems I have an idea to build a CMS system and wanted to use cowboy for this. I am new to both erlang and cowboy.
Here is my problem
I want to be able to support redirects. Normally the sites I have seen have several thousand redirect URLs that are used mostly by marketing and some for legacy pages.
I was thinking to create a rule in cowboy to check first if the request URL is matching any of the thousands URLs
{"some.site", [
{'', redirect_handler, []},
{"/", welcome_handler, []},
{'', cms_handler, []}
]}.

The problem is though that once you match the first rule, I couldnt find a way to tell the router to continue with the next rule.
In java based server usually there is a FROWARD feature that you can forward the request to another servlet. IS there something similar to forward the request to another handler?
I though to use Constraints for that but besides a few line on the documentation I couldnt find any example.
Has anybody any idea on how to solve this problem?

Normally is java based apps this kind of problems is solved by the use of filters. Is there something similar here?

Kind Regards
Nikolas

[essen]

You probably want a stream handler. It has been suggested before to add such a feature. I have put some thoughts about it here: #1346 (comment)

The proposed "rewrite" can either result in a 3xx status code or an internal redirect.

The router would only list valid routes, and the rewriter would simply modify the incoming URL to point to one of these routes.

[java4devgmail]

Hi

I read the relevant chapter and looked at the code of cowboy_stream_h.erl, but I am not wiser than before.
It is not clear what is the call flow and what needs to be implemented.
I also looked at the diagram https://ninenines.eu/docs/en/cowboy/2.9/guide/flow_diagram/ and to me it makes more sense if the handler returned {next. Req, State} instead of {ok, Req, State}.

In both cases though if we want to enhance either cowboy_stream_h.erl or cowboy_router.erl shouldn't we also get back the Tail of of the Dispatcher rules so we can continue working on the remaining ones?
{next. Req, State, DispatcherTail}

By rereading the doc for https://ninenines.eu/docs/en/cowboy/2.9/guide/handlers/ I found a pragraph about other handlers where it says
<<Instead of returning ok, you simply return the name of the handler type you want to use. >>
The problem though here is that the handlers are behaviour(cowboy_sub_protocol) instead of behavior(cowboy_handler)

Best
Nikolas

[essen]

You're mixing things up. cowboy_sub_protocol is for implementing things like Websocket. cowboy_handler is for high-level request handlers.

Stream handlers are low level and receive in/out events as they come in. Any function might be called at any time, other than init/terminate/early_error.

You do not want to change cowboy_stream_h, you want to create a NEW stream handler that will modify the Req by doing URL rewriting. You do not need to touch cowboy_router or the dispatch list, you just need to modify the incoming Req when it points to a path that you want to rewrite.

[java4devgmail]

I probably missunderstood your previous message or havent explained properly what I want to do.
If I undersood correctly the concept of cowboy is to define the different paths and the handler that will process it (route list).
Redirect cases need to be handled first. e.g having following two redirect rules

• host: www.example.com, source: /se/en/products.html, target: https://www.products.com/example.html, permannent
• host: www.example.com, source: /se/en/careers.html, target: https://www.jobfinder.com/example.html, temporary

so the only way to do this is to define first a generic routing rule
{'_', redirect_handler, []},
if the URL matches one in my redirect list I want to return a 301 or 302. I dont see how a rewrite of the url can fix the issue and also if the URL does not match any of the redirect rules then the handler should pass the request back to the route engine so that it tries to match it with one of the next route rules

[essen]

I understood what you are trying to do. You cannot do this with the router. You can do this with a stream handler. The stream handler can either rewrite or end the request with a 3xx response.

You could probably do this with a middleware running before the router as well.

[java4devgmail]

Hi

sorry to persist on this one, but I beleive I have a solution that conforms with cowboy's concept.
route constraint.
Instead of having the rule {'', redirect_handler, []}
I can add a constraint {'_', [{cowboy_req:path(Req0), F}], redirect_handler, []}
I will try to build this and see if it works.
If I undrstood the docs and the code then if the constraint fails the routing will try to match the next rule.
My only problem is that I am trying to do everything dynamically, routes are loaded from configuration files and since my erlang experience is below novice level every line of code takes a few hours to google for relevant documentation.
it might take a few days/weeks but I will reply back if I manage to do this.

Kind Regards
Nikolas

[java4devgmail]

Hi
I managed to create a prototype using the constraints in routes but my problem now is that the constraints never match and it just continues to the next route rule.

The routes for compilation are

 ConstraintRoutes [{"www.abc.local",
                   [{'_',[{path,#Fun<cms_routes.0.118504675>}],
                         redirect_handler,[]},
                    {"/",welcome_handler,[]},
                    {"/example",testcowboyapp_handler,[]},
                    {'_',cms_handler,[]}]},
                  {"mob.abc.local",
                   [{'_',[{path,false}],redirect_handler,[]},
                    {"/",welcome_handler,[]},
                    {'_',cms_handler,[]}]},
                  {'_',[{"/",testcowboyapp_handler,[]},
                        {"/explore/:site/[...]",explorer_handler,[]}]}]

What is the value the constraint function must return if it is a match and what if it is not?

Kind Regards
Nikolas

[essen]

An ok or error tuple.

[java4devgmail]

Thank you for the info.
I tried that but it seems that the constraint Fun is not called at all.
This is strange because it is the first rule and the path is '_'

I will keep looking to see if I missed something.

Best
Nikolas

[java4devgmail]

Hi again

I still cannot make this work, I spend hours of trial and error with countless combinations
but after reading the source code again I found this in the source code of cowboy_router.erl
If the path is '_' then there can be no constraints.

match_path([], _, _, _) ->
	{error, notfound, path};
%% If the path is '_' then there can be no constraints.
match_path([{'_', [], Handler, Opts}|_Tail], HostInfo, _, Bindings) ->
	{ok, Handler, Opts, Bindings, HostInfo, undefined};

Could that be the reason that my constraints are not even considered?
as said earlier my path match line looks like this

 [{'_', [{path,#Fun<cms_routes.0.118504675>}], redirect_handler,[]},

Kind Regards
Nikolas

[essen]

Right it makes sense because constraints apply to a binding (a name) found in the path and '_' leaves room for no names. Same goes for [...]. I don't think you can do this in the router. The right way to do this is to write a stream handler to do URL rewriting. It's just a matter of writing code in init and deferring to the next in every other callback I'm not sure what's the problem.

[java4devgmail]

Hi Loïc

I found another problem while trying to debug the cowboy_router.erl,
the Bindingsis is always an empty map

 check_constraints Field: {path,#Fun<cms_routes.0.75820517>} Tail:[] Bindings: #{} 

Now going back to your proposal I have two questions:

1. Why do you keep calling the redirect as rewrite? Redirect is a responce of type 301, 302, 303, 307 308
   A rewrite is usually an internal server feature, where you rewrite the original request and restart the request internally.

2. Could you please refer me to some example or documentation or source code to get a head start?

Kind Regards
Nikolas

[essen]

Bindings being empty is normal because you didn't include any of them in your route.

Take a look at cowboy_stream_h, cowboy_compress_h, cowboy_metrics_h, cowboy_tracer_h and the documentation on cowboy_stream and stream handlers.

[java4devgmail]

Hi Loïc

after a lot of search and effort to reverse engineer cowboy in order to understand what is expected, I managed to built something by copying some other projects but I cannot find any information on how a stream should be built and as a result, normal requests fail.
I attach a test project cowboy_redirect.zip (run with $rebar3 shell)
http://localhost:8080/redirect seems to work, I get what I expect, but
http://localhost:8080/welcome fails.

Do you have any reference on how the redirect_stream.erl should be built or can you tell me what is wrong with redirect_stream.erl

Kind Regards
Nikolas

[essen]

If you don't need to keep state then get rid of #state{} and just pass and return the state to/from cowboy_stream. Your code has a few functions that already do that but for some reason others use a state record.

[java4devgmail]

For the standard url /welcome that fails, I dont return any #state{}
I just call the init from cowboy_strean like in cowboy_tracer_h

init(StreamID, Req, Opts) ->
...
{_, Reason} ->
			cowboy_stream:init(StreamID, Req, Opts)
end.

[essen]

You do in other callbacks.