Replies: 5 comments 4 replies
-
You could use multiple threads to compile and gather the results into the builder thread. The java builder works like that: only one thread writes into workspace, but multiple threads read. |
Beta Was this translation helpful? Give feedback.
-
How exactly are you writing your results to the workspace? I would expect that you can get an ISchedulingRule for the complete workspace, and write your results from within that scheduling rule, thereby effectively blocking further jobs running on the same projects. But I haven't actually tested this, I just remember we had the opposite of your issue (not enough stuff being triggered in parallel because of too large scheduling rule). |
Beta Was this translation helpful? Give feedback.
-
Thank you both for your responses. After digging through both our and the auto-build implementation, I don't think it's possible to prevent the auto-build job from rescheduling itself over and over again. Given the size of our builders, I also don't see an easy way for splitting the read and write without a major time investment and without re-creating this problem, whenever we do changes there. My question: Would there be an interest in me creating a pull request to address those issues? Possible solutions I came up with are: (I) Introduce a new property which can be assigned to jobs. The interrupt flag is currently set on all workspace modifications, unless those were caused auto-build job itself. External jobs could then opt-in (at their own risk, of course) by setting this property, so that their modifications are also ignored. (II) Introduce a new job state. Our worker jobs are inside a job group. From within the AutoBuild job, we then call join(), to wait for those worker jobs to complete. While waiting, the auto-build job would then switch to an "idle" state, during which interrupts are ignored. I'm leaning towards the first option, as we can reuse the setProperty() method of the Job API and because it shouldn't cause any unintended side-effects for code that doesn't use this functionality. Thoughts? |
Beta Was this translation helpful? Give feedback.
-
Can you please provide a stack showing how exactly this happens?
If the problem is clearly stated & understood. I'm not sure this is the case. My understanding is that you want to generate something in parallel during the autobuild and write it via resources API synchronously to the build invocation that locked same project. I wonder if you can postpone the write after the build is done or use Java NIO API or write from builder thread (put lambdas that write on the queue of your builder)? |
Beta Was this translation helpful? Give feedback.
-
I don't see any reason to extend eclipse for parallel builder support, as (at least on windows) the main performance bootleneck is always writing the result, and currently the resource API just can't write concurrently. You are better off fixing your builder. |
Beta Was this translation helpful? Give feedback.
-
Currently the auto-builder is able to work on multiple (independent) projects in parallel, but each project itself is still compiled sequentially, regardless how big the project is.
I've come accross some rather old bug reports regarding this issue, like Bug 126121 or Bug 227025. Have any changes been made since then?
Context: We use the incremental builder in our product to generate byte code from custom entities. Those entities are stored in separate projects, with each project containing several hundred files at a time. With a purely sequential build, this takes quite a lot of time.
In order to increase the performance, our builder spawns multiple "worker" jobs, which perform the actual compilation. The auto-builder remains inactive during that time and only continues after those jobs have finished. This itself works for the most part.
The issue: Because the worker jobs run in separate threads, they raise the "interrupted" flag of the auto-builder, whenever the compilation result is stored. This makes it difficult to cancel the job, as the auto-builder schedules itself for execution again, whenever this flag is set. This then triggers the "worker" jobs, they then raise the "interrupt" flag, the user cancels the new job and the cycle repeats itself over and over again...
Reason being that the auto-builder reserves the entire workspace. meaning that any modifications to it are causing a conflict. Which is hardcoded and not something I can influence in a meaningful way. Even if those "worker" jobs are technically a part of it, it is not something that can be expressed by the current Job API.
As a note, this rescheduling is only done whenever an OperationCanceledException is thrown by the builder. Meaning a "workaround" is to simply convert this into a plain CoreException. However, I wonder if there is cleaner way to do it or whether I've simply missed a feature for exactly this use case.
Beta Was this translation helpful? Give feedback.
All reactions