Custom RDAP Servers

[jobr97]

Hello,

I noticed that domain-watchdog uses the IANA dns.json to get the information about the RDAP Servers. This list does somehow not contain most (or all?) ccTLDs.

I found this page which seems to contain RDAP urls for all TLDs. Would it be possible to add them?
Or maybe add a page on the UI to see and modify RDAP servers?

[vinceh121]

Hi,

Both lists contain the same number of RDAP servers (currently 1181), and are identical. I'd assume the rdap.org JSON is based on the IANA bootstrap data.
Moreover, due to stability concerns, only official IANA data as specified in the RDAP specifications will be used.

I agree that the lack of support of certain TLDs is a source of frustration due to late RDAP adoption, however trying to jump the gun and use unofficial RDAP servers will lead to issues.

[maelgangloff]

Hello @jobr97,

Thanks for your question !
The list you mention comes from rdap.org which is not an official source.

At the end of the file we can read: «Please note that this RDAP service is NOT provided by the IANA.»

In this project, I was careful to use the most official sources possible, especially in obtaining the list of RDAP servers.

Here is how the update of this list works every month:

1. Domain Watchdog makes a first request to IANA. This request allows to obtain the entire list of TLDs (without the classification by category).
   Link: https://data.iana.org/TLD/tlds-alpha-by-domain.txt

2. Domain Watchdog makes a second request to ICANN. It obtains the list of gTLDs (therefore does not contain ccTLDs).
   Link: https://www.icann.org/resources/registries/gtlds/v2/gtlds.json

3. Domain Watchdog makes a last request to IANA to obtain the list of "official" RDAP servers.
   Link: https://data.iana.org/rdap/dns.json

After a bit of sorting, the TLDs are sorted by category and stored in DB.

Only RDAP servers officially published by IANA are integrated into Domain Watchdog for the moment.

Have you been confronted with the case of an existing RDAP server but not known by Domain Watchdog?

[jobr97]

@vinceh121 @maelgangloff Thank you both for the quick responses!

I do appreciate the thought for security and therefore not using third-party information.

I did find some RDAP servers that are not on the list but do exist:

TLD	URL	Description
de	https://rdap.denic.de/	ccTLD for Germany
ch	https://rdap.nic.ch/	ccTLD for Switzerland
li	https://rdap.nic.li/	ccTLD for Liechtenstein

Once I added them to the rdap_server table in prostgres I was able to add domains with the .de tld.

[maelgangloff]

This is a complex question that will require careful thought. I have mixed feelings about making it possible to add new RDAP servers.

The information returned by these servers is very limited (legal considerations I suppose).
I hope that the RDAP servers you mention will soon be listed by IANA if they are compliant.

Adding a configuration file with the same format as dns.json with a list of predefined RDAP servers would be a possible solution I guess. I will see about adding this option ;)

[maelgangloff]

Hello @jobr97 ,

I just added this feature. Here's how it works:
To use this feature, a YAML file must be present on config/app/custom_rdap_servers.yaml

Every month, the list of RDAP servers is updated from the IANA bootstrap file. Following this, if this optional config file is present, its data will be added.

Since YAML is a superset of JSON, you can use pure JSON in this file if you prefer.

For convenience, I used the same parser as the one for the dns.json file retrieved from IANA.
So you can use it as inspiration to create your own (or read RFC 9224 which describes the format of this file ;))

In your case, this would look like:

{
    "services": [
        [
            [
                "de"
            ],
            [
                "https://rdap.denic.de/"
            ]
        ],
        [
            [
                "ch"
            ],
            [
                "https://rdap.nic.ch/"
            ]
        ],
        [
            [
                "li"
            ],
            [
                "https://rdap.nic.li/"
            ]
        ]
    ]
}

This feature will require documentation (I really need to take care of the documentation of this project...)

[jobr97]

You're a legend 😄
I am looking forward to trying it out.

[maelgangloff]

This message is for future users of this feature.

A new file has been added to the repository : config/app/custom_rdap_servers.example.yaml.
This is a list of unofficial RDAP servers that I obtained by testing on common domains (like rdap.nic.TLD).

To use it, simply copy it to config/app/custom_rdap_servers.yaml (or create a symbolic link which avoids making a copy each time you update the project).

ln -s custom_rdap_servers.example.yaml custom_rdap_servers.yaml

I have constructed the file so that RDAP servers published by IANA will have a preponderant value over those in this list (typically when IANA publishes an RDAP server for a TLD present in this file).
This behavior can be changed by removing the publication parameter on this file.

Warning

Please be careful. This list is in no way published by IANA. We know nothing about the stability of these servers or the veracity of the information they return.
If I add this file to this repository, it is only to facilitate the identification of RDAP servers not listed by IANA.

[NoverNobida]

This message is for future users of this feature.

A new file has been added to the repository : config/app/custom_rdap_servers.example.yaml. This is a list of unofficial RDAP servers that I obtained by testing on common domains (like rdap.nic.TLD).

To use it, simply copy it to config/app/custom_rdap_servers.yaml (or create a symbolic link which avoids making a copy each time you update the project).

ln -s custom_rdap_servers.example.yaml custom_rdap_servers.yaml

I have constructed the file so that RDAP servers published by IANA will have a preponderant value over those in this list (typically when IANA publishes an RDAP server for a TLD present in this file). This behavior can be changed by removing the publication parameter on this file.

Warning

Please be careful. This list is in no way published by IANA. We know nothing about the stability of these servers or the veracity of the information they return. If I add this file to this repository, it is only to facilitate the identification of RDAP servers not listed by IANA.

How is this configuration going to work? For example, I can't query example.us at all without this configuration, and after I copied custom_rdap_servers.example.yaml to custom_rdap_servers.yaml and recreated the containers, I still can't query example.us. I also got the error "Unable to determine which RDAP server to contact".

domain-watchdog/config/app/custom_rdap_servers.example.yaml

Line 58 in 534a8db

[ [ "us" ], [ "https://rdap.nic.us/" ] ],

[maelgangloff]

Hello @NoverNobida,

The list of RDAP servers is updated every month (from the bootstrap file provided by IANA and your custom configuration file).
To force the update of this list, you can run:

php bin/console app:update-rdap-servers

This will update the list of RDAP servers and update the database accordingly.

On my side, I can retrieve the information for this domain name:

Normally, if this is the first start of the container, this list is updated. It's weird that this is not the case in your situation.
I suggest you check the logs to see if Domain Watchdog has detected your custom configuration file or not.

[NoverNobida]

@maelgangloff Thanks for the answer, that was a quick reply.
I solved it. I mounted to local. But it should be

······
    php-worker:
        volumes:
            - ./custom_rdap_servers.yaml:/app/config/app/custom_rdap_servers.yaml
······

not

······
    domainwatchdog:
        volumes:
            - ./custom_rdap_servers.yaml:/app/config/app/custom_rdap_servers.yaml
······

[maelgangloff]

Yes, it is the worker that updates the list of RDAP servers every month. So it is the worker that needs to access this file.
I will add a comment in the docker-compose to make this feature easier to use.
As said before, this project really needs documentation to clarify the installation I think ;)

If everyone has managed to get RDAP servers working that are not listed in the bootstrap file, I will close this issue. (I hope it is also good for you @jobr97).

Feel free to reopen it if necessary.