Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenRewrite's Maven resolution does not use the proxy configuration from the settings.xml #543

Open
s-kocher opened this issue Apr 17, 2023 · 9 comments
Labels
enhancement New feature or request

Comments

@s-kocher
Copy link

s-kocher commented Apr 17, 2023

I recreate an issue for the use of proxy settings in maven settins as mentionned here : #449 (comment)

Thanks for the extra information. OpenRewrite has its own maven resolution logic and it is aware of the settings.xml file (it correctly handles mirrors and servers, and profiles). However, it does not yet have the logic to use the proxies defined in that file. We will need to add support for this.

I'm using a company repository mirror hosted on AWS, so proxy is needed to reach it :
settings.xml

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/xsd/settings-1.0.0.xsd">

	<proxies>
		<proxy>
		  <id>optional</id>
		  <active>true</active>
		  <protocol>http</protocol>
		  <host>proxy.corporate.com</host>
		  <port>8080</port>
                  <username>login</username>
                  <password>{maven_ciphered_password=}</password>
		</proxy>	
	</proxies>

	<mirrors>
		<mirror>
			 <id>mycompany-repo</id>
			<mirrorOf>*</mirrorOf>
			<url>https://my.aws.company.com/nexus/repository/mycompany-repo/
			</url>
		</mirror>
	</mirrors>
</settings>

When executing this :
org.openrewrite.maven:rewrite-maven-plugin:4.43.0:run

      <plugin>
        <groupId>org.openrewrite.maven</groupId>
        <artifactId>rewrite-maven-plugin</artifactId>
        <version>4.43.0</version>
        <configuration>
          <activeRecipes>
            <recipe>org.openrewrite.java.testing.assertj.Assertj</recipe>
          </activeRecipes>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>org.openrewrite.recipe</groupId>
            <artifactId>rewrite-testing-frameworks</artifactId>
            <version>1.36.0</version>
            <scope>runtime</scope>
          </dependency>
        </dependencies>
      </plugin>

I get this error

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  49.965 s
[INFO] Finished at: 2023-04-17T15:48:43+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.openrewrite.maven:rewrite-maven-plugin:4.43.0:run (default-cli) on project my-project: Execution default-cli of goal org.openrewrite.maven:rewrite-maven-plugin:4.43.0:run failed: Failed to download dependencies for pom.xml:
[ERROR] <?xml version="1.0" encoding="UTF-8"?>
[ERROR] <project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
[ERROR]   xmlns="http://maven.apache.org/POM/4.0.0"
[ERROR]   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
...
[ERROR]   <dependencies>
[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-actuator:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR] https://my.aws.company.com/nexus/repository/mycompany-repo/: java.net.UnknownHostException: my.aws.company.com)~~>--><dependency>
[ERROR]       <artifactId>spring-boot-starter-actuator</artifactId>
[ERROR]       <groupId>org.springframework.boot</groupId>
[ERROR]     </dependency>
...
[ERROR] </project>: MavenDownloadingExceptions
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

Process finished with exit code 1

I tried to add the JVM argument -Dorg.openrewrite.test.readMavenSettingsFromDisk=true without success

@timtebeek timtebeek added the enhancement New feature or request label Apr 17, 2023
@timtebeek timtebeek moved this to Backlog in OpenRewrite Apr 17, 2023
@timtebeek
Copy link
Contributor

Thanks for taking the time to provide that context from an older issue comment! As you've found we don't have support yet, but we'll track support in this issue.

Are you able to bypass the proxy locally for dependencies also hosted at Maven Central, such that you can still run recipes? Since the error is on spring-boot-starter-actuator. Or is this blocking you from running any recipe at all?

@s-kocher
Copy link
Author

s-kocher commented Apr 17, 2023

I shrink the error log but many pom's dependencies failed to be downloaded.

Many Spring dependencies are KO and some non spring too :

[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-actuator:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-web:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-aop:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-data-jdbc:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-data-jpa:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(io.netty:netty-tcnative-boringssl-static:2.0.51.Final failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(net.bytebuddy:byte-buddy:1.12.22 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-autoconfigure:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-security:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-test:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-web:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-webflux:2.7.8 failed. Unable to download POM. Tried repositories:

Sadly I have corporate internal dependencies too, they will not be retrieved through Maven Central.

But I don't understand why OpenRewrite Maven plugin did not use the pom.xml already available in local repository ?

@timtebeek
Copy link
Contributor

We need to check with Maven Central (or another reachable repository) to resolve our dependency version selectors. That's how we're able to upgrade you to the latest version within a range without needing to publish a new release each time Spring, JUnit, Mockito, ... release a new version.

In your case your settings.xml configuration is tripping you up because you've set mycompany-repo to be a mirror of *, rather than only used for internal dependencies. I think if you adjust your mirror settings you might be able to run recipes still, depending on what your network allows.

@s-kocher
Copy link
Author

s-kocher commented Apr 17, 2023

I tested with that :

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <proxies>
    <proxy>
      <id>optional</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>proxy.corporate.com</host>
      <port>8080</port>
      <username>login</username>
      <password>{maven_ciphered_password=}</password>
    </proxy>
  </proxies>

<!-- Remove mirror of central -->

  <servers>
    <server>
      <id>company-group</id>
      <username>read-only-login</username>
      <password>PASSWORD</password>
    </server>
  </servers>

  <profiles>
    <profile>
      <id>my-company</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>

      <repositories>
        <repository>
          <id>company-group</id>
          <name>My Company repository</name>
          <url>https://my.aws.company.com/nexus/repository/mycompany-repo</url>
          <releases>
            <enabled>true</enabled>
            <updatePolicy>never</updatePolicy>
          </releases>
          <snapshots>
            <enabled>true</enabled>
            <updatePolicy>always</updatePolicy>
          </snapshots>
        </repository>
      </repositories>
    </profile>
  </profiles>

</settings>

But now, error moves on maven central host's resolution :

[ERROR]     <!--~~(org.springframework.boot:spring-boot-starter-web:2.7.8 failed. Unable to download POM. Tried repositories:
[ERROR] https://repo.maven.apache.org/maven2: java.net.UnknownHostException: repo.maven.apache.org)~~>--><dependency>

I guess it's normal if maven proxy is not used.

@timtebeek
Copy link
Contributor

That indeed looks like a limitation on your network; guess you really do need proxy support for anything to work then. :/

@s-kocher
Copy link
Author

yes that how it works, I guess, in almost all big companies : internet can only be reached through a proxy, often requiring authentication (and often with a lot of complications like NTLM authentication, rarely supported / well implemented in many softwares able to use an internet proxy)

@timtebeek
Copy link
Contributor

Understandable; we'll track any progress towards proxy support here, although it's competing for attention with other issues, and only happens on restrictive corporate networks using the OSS plugins to run OpenRewrite.

For our corporate customers it's not an issue as the Moderne.io platform runs the recipes on separate workers that are configured differently to cope with such network limitations. I understand that might not yet be an option for you, but thought to provide that context such that you know there is an alternative, which also influences the priority on this ticket. If you have a strong need for this feature, and the platform is not an option, then we're open to contributions to see this resolved more quickly.

@zacthompson
Copy link

Note that -DskipMavenParsing=true is an option, depending on what recipes you're running. Worked in my case

@torsten-liermann
Copy link

torsten-liermann commented Dec 17, 2024

I am now running into this problem. "In restrictive networks" sounds a bit biased because it is actually the norm that artifacts are not loaded from any public sources in CI/CD processes but through internal repository managers. However, in the case of my evaluation, certain repositories are fetched from the internet and only added to the repository manager after successful evaluation.

I am somewhat surprised that the OpenRewrite Maven plugin deliberately does not use the Maven mechanisms for artifact verification or, put differently, apparently works without a proxy.

[INFO] --- rewrite:5.46.3:run (default-cli) @ rewrite-eap10-migration-test ---
[INFO] Using active recipe(s) [MigrateEAPDependencies]
[INFO] Using active styles(s) []
[INFO] Validating active recipes...
[INFO] Project [rewrite-eap10-migration-test] Resolving Poms...
[WARNING] Failed to access maven repository https://maven.repository.redhat.com/ga/ due to: Connect timed out

Disabling the parsing of the POM -DskipMavenParsing=true is not an option for me because I want to migrate pom.xml files – see my posts on Slack.

As a workaround, configuring the proxy for the Java VM in the following form (Windows CLI) helps here:

set MAVEN_OPTS=-Dhttp.proxyHost=proxy.my-domain.de ^
-Dhttp.proxyPort=3124 ^
-Dhttp.proxyUser=me ^
-Dhttp.nonProxyHosts="localhost|127.0.0.1|172.18.*|172.16.*|*.my-domain.de|10.*" ^
-Dhttp.proxyPassword=my-password ^
-Dhttps.proxyHost=proxy.my-domain.de ^
-Dhttps.proxyPort=3128 ^
-Dhttps.proxyUser=me ^ 
-Dhttps.proxyPassword=my-password

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Backlog
Development

No branches or pull requests

4 participants