Link verification

[AnthonyMuscio]

Folks,

We can freely quote external urls in tiddlywiki. People are making large bookmark libraries and more.

The problem with external links is they can die, and when they do we may want to identify this and find its new address if possible.

I am wondering if there is a way in tiddlywiki already or by plugin or core change where we could have external links tested and indicated if an error is returned without needing to click and follow that link? Its like how a visited link changes color, could wee make its so dead links are marked as such?

Regards
Tony

[pmario]

I can think of this function as a plugin. ..

The problem is, it will "eat" a lot of bandwidth, just to give a different colour to an external link. ... For some users this may be a no-go. ..

From my point of view, this is a "waste of resources", if it is done automatically and always.

I could think about a function like this, that is manually activated eg: every 4 to 6 months.

So we would need a function in the core, that will "scan all tiddlers for external links" and a widget eg: <$scan-links> with the same parameters as <$links> ...

There is 1 more technical problem. ... If you load several 100 links from the same provider or URL in a very fast pace, it may be interpreted as an attack. ... So the logic needs to take care of this behaviour.

Last but not least a bit of maths.

• According to this post: https://speedcurve.com/blog/web-performance-page-bloat/ the average size of a page in 2019 was 4MByte.
• So let's say a "Link Bookmarks Wiki" contains 1000 links.

To check if the links exist we will download about 4GigaByte of content. ... Sure that is done only the first time, if browser caching is allowed. ... BUT with my mobile account this will need my data volume for 2 months. ...

Some users may also throw away the browser cache when the browser is closed. ... So this may be an even bigger problem, we need to think about.

[AnthonyMuscio]

Mario

Thanks for your response.

I thought there was a way to test DNS resolution of an address without loading the resource at the end of the link.

I agree either a throttled batch process may be needed but I could see a macro or widget given a url could test it on render. Thus when a tiddler is opened with such a link it would take place. You could force an interactive click to check.

Regards
Tony

[Jermolene]

Just to clarify that it would only be possible to verify links under Node.js. Browser security restrictions make it impossible for us to check URLs when running standalone in the browser.

[AnthonyMuscio]

There looks like there may be some standard ways to active this such as here https://stackoverflow.com/questions/102605/can-i-perform-a-dns-lookup-hostname-to-ip-address-using-client-side-javascript/102681#102681

Regards
Tony

[saqimtiaz]

Quoting from your link:

Pure JS can't. If you have a server script under the same domain that prints it out you could send a XMLHttpRequest to read it.

[AnthonyMuscio]

Saq,

There is more further down about DNS over http and others but it seemed possible by using open services.

Tony